Lightweight real-time target detection method and device, server and storage medium

Abstract

The invention discloses a lightweight real-time target detection method and device, a server and a storage medium. The method comprises the following steps: extracting shallow image features by using a basic convolution module of an RT-DETR model backbone network; a lightweight sampling sub-module of the multi-scale cavity convolution module is used for extracting context features and fusing the context features with local features; extracting and fusing multi-scale features by using a multi-scale dilatation attention sub-module of the multi-scale cavity convolution module to obtain deep image features; and obtaining a target detection result of the image data by using the hybrid encoder and the decoder. A BasicBlock module is improved through a lightweight sampling sub-module and a multi-scale dilatation attention sub-module, surrounding context semantic information can be captured without changing the size of a convolution kernel by utilizing cavity convolution, an original mode is replaced, sparsity of multi-scale dilatation attention at different scales is utilized, the parameter quantity and the calculation quantity of the model are reduced, and the model performance is improved. And the model can be deployed on equipment with limited computing power.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a lightweight real-time target detection method, device, server and storage medium. Background Art

[0002] In the fields of intelligent security and autonomous driving, it is often necessary to detect targets in images in real time. Target detection models such as the YOLO series and the DETR series can detect specific targets in images by extracting features from images. RT-DETR (Real-Time DEtection TRansformer) is a target detection model based on the Transformer structure. It uses an encoder-decoder architecture and uses a self-attention mechanism to extract and associate target features. RT-DETR can better capture long-distance dependencies in images. In images with dense targets or complex backgrounds, it can show stronger detection effects than the YOLO series through adaptive feature capture.

[0003] The RT-DETR model has a large computational overhead and high demand for hardware resources. To achieve real-time target detection, it is often necessary to deploy it in a data center with sufficient computing power. The data center uses the RT-DETR model to detect targets on the received images and transmits the detection results back to the terminal device. However, the deployment of smart security systems often requires low costs. Transmitting image data to the data center consumes a lot of transmission resources, and the cost of establishing a data center with high computing power is high. As the terminal of the security system, the computing power of smart security cameras is limited and cannot meet the real-time target detection computing power requirements of the RT-DETR model. Summary of the invention

[0004] The embodiments of the present invention provide a lightweight real-time target detection method, device, server and storage medium to solve the technical problem that the RT-DETR model cannot run in an environment with limited computing power.

[0005] In a first aspect, an embodiment of the present invention provides a lightweight real-time target detection method, comprising: Input the collected image data into the RT-DETR model, the backbone network of which includes a basic convolution module and a multi-scale dilated convolution module, and the multi-scale dilated convolution module includes a lightweight downsampling submodule and a multi-scale dilated attention submodule; The basic convolution module is used to extract basic features from the collected image data to obtain shallow image features; Utilize the lightweight downsampling submodule to perform hole convolution on shallow image features to extract context features, fuse the context features with local features extracted by standard convolution on the shallow image by the lightweight downsampling submodule, and obtain joint image features containing context; The multi-scale expansion attention submodule is used to extract features of different scales from the joint image features at different expansion rates, and features of different scales are fused to obtain deep image features; The hybrid encoder of the RT-DETR model is used to fuse the deep image features, and the decoder of the RT-DETR model is used to identify the target features to obtain the target detection results of the image data.

[0006] Furthermore, the light downsampling submodule is used to perform hole convolution on shallow image features to extract context features, and the context features are fused with local features extracted by standard convolution on the shallow image by the light downsampling submodule to obtain joint image features containing context, including: The lightweight downsampling submodule performs standard convolution processing on shallow image features to obtain refined local features, and performs hole convolution processing on shallow image features to obtain surrounding context features; The lightweight downsampling submodule performs feature fusion on the refined local features and surrounding context features to obtain fused features; The lightweight downsampling submodule enhances the key information of the fused features to obtain joint image features containing context.

[0007] Furthermore, the lightweight downsampling submodule performs feature fusion on the refined local features and the surrounding context features to obtain fused features, including: Perform feature splicing on the refined local features and surrounding context features to obtain spliced ​​features; The concatenated features are batch normalized and the fused features are generated through the activation function.

[0008] Furthermore, the lightweight downsampling submodule enhances key information of the fused features to obtain a joint image feature containing context, including: Perform global average pooling on the fused features to obtain global features; A multi-layer perceptron is used to enhance the key information of global features and obtain joint image features containing context.

[0009] Furthermore, the multi-scale expansion attention submodule is used to extract features of different scales from the joint image features at different expansion rates, and features of different scales are fused to obtain deep image features, including: The multi-scale dilation attention submodule uses multiple heads to perform multi-scale SWDA operations on the joint image features with different dilation rates to obtain the head features of each head; The multi-scale dilated attention submodule uses a linear layer to aggregate all head features to obtain deep image features.

[0010] Furthermore, the hybrid encoder of the RT-DETR model is used to perform feature fusion on deep image features, and the decoder of the RT-DETR model is used to perform target feature recognition to obtain target detection results of image data, including: The hybrid encoder uses the deep image features generated by the last three multi-scale dilated convolution modules in the backbone network to perform multi-scale feature fusion to obtain encoder features; The decoder iterates and predicts the features of the encoder to obtain the target detection results of the image data.

[0011] Furthermore, the multi-scale dilated convolution module also includes a basic convolution submodule and an activation function submodule.

[0012] In a second aspect, an embodiment of the present invention provides a lightweight real-time target detection device, including: An image acquisition unit, used to acquire image data and transmit it to the RT-DETR model; The basic convolution unit is used to perform standard convolution on the collected image data; Multi-scale dilated convolution unit, used to perform multi-scale dilated convolution on shallow image features generated after standard convolution; A hybrid encoder unit for feature fusion of deep image features generated after multi-scale dilated convolution; The decoder unit is used for object detection after feature fusion from the hybrid encoder.

[0013] In a third aspect, an embodiment of the present invention provides a server, including: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned lightweight real-time target detection method.

[0014] In a fourth aspect, an embodiment of the present invention provides a storage medium comprising computer executable instructions, which are used to execute the above-mentioned lightweight real-time target detection method when executed by a computer processor.

[0015] The embodiment of the present invention provides a lightweight real-time target detection method, device, server and storage medium. The method improves the BasicBlock module in the backbone network of the RT-DETR model, and replaces the original basic convolution submodule with a lightweight downsampling submodule and a multi-scale dilated attention submodule. The lightweight downsampling submodule extracts features by combining standard convolution with dilated convolution to obtain joint image features containing context, thereby reducing the number of parameters and the amount of calculation of the model, and improving the accuracy of target detection by capturing contextual semantic information. The multi-scale dilated attention submodule uses different dilation rates to extract features of different scales to capture multi-scale semantic information, effectively making up for the detection performance of the lightweight downsampling submodule in complex scenes, small targets and high-density targets, and uses the sparsity of the self-attention mechanism at different scales to further reduce computational redundancy while ensuring the detection performance of the model. The deep image features extracted by the lightweight downsampling submodule and the multi-scale dilated attention submodule contain the details of local features, the surrounding context semantic information, and the multi-scale semantic information. While improving the accuracy of target detection, it can also reduce the number of model parameters and the amount of calculation. The RT-DETR model can be deployed on terminal devices or edge devices with limited computing power such as smart security cameras to achieve real-time target detection. It only needs to transmit the detection results to the data center without transmitting image or video data, which greatly reduces the transmission resource occupancy and computing energy consumption, and extends the working time of the equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings: Figure 1 This is a flow chart of a lightweight real-time target detection method according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the architecture of the RT-DETR model described in Example 1 of the present invention; Figure 3 This is a flow chart of a lightweight real-time target detection method according to Embodiment 2 of the present invention; Figure 4 Schematic diagram of the architecture of the lightweight downsampling sub-module according to the second embodiment of the present invention; Figure 5 This is a flow chart of a lightweight real-time target detection method according to Embodiment 3 of the present invention; Figure 6 This is a schematic diagram of the architecture of the multi-scale dilated attention submodule described in Embodiment 3 of the present invention; Figure 7 This is a structural schematic diagram of a lightweight real-time target detection device according to Embodiment 4 of the present invention; Figure 8 This is a structural diagram of the server described in Example 5 of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0018] The RT-DETR (Real-Time Detection Transformer) model is a real-time target detection model based on the Transformer structure. By adopting an encoder-decoder structure and using the self-attention mechanism to extract and associate target features, it can better capture long-distance dependencies in images and improve the detection accuracy of the model in complex scenes. In images with dense targets or complex backgrounds, it can achieve higher target detection accuracy through adaptive feature capture. It also has an end-to-end detection structure, which simplifies the detection process and reduces error accumulation to a certain extent. However, the RT-DETR model has a large number of parameters and a large amount of calculation. In order to meet the real-time performance of target detection, it has high requirements for hardware resources and computing resources. On some devices or edge devices with limited computing resources, it is difficult to meet the required computing resource requirements, making it difficult to achieve real-time target detection. Especially in smart security systems, in order to facilitate system deployment and reduce costs, real-time target detection is required on the smart security cameras at the system terminals, and only the detection results are transmitted back to the data center, replacing the large amount of transmission resources occupied by traditional cameras to transmit image / video data back to the data center. However, the computing power of smart security cameras is limited, and it is difficult to meet the computing requirements of the RT-DETR model to achieve real-time target detection.

[0019] Embodiment 1

[0020] Figure 1 This is a flow chart of a lightweight real-time target detection method according to Embodiment 1 of the present invention. This embodiment improves the module for extracting deep image features in the backbone network, and specifically includes the following steps: S101, input the collected image data into the RT-DETR model, the backbone network of the RT-DETR model includes a basic convolution module and a multi-scale dilated convolution module, and the multi-scale dilated convolution module includes a lightweight downsampling submodule and a multi-scale dilated attention submodule.

[0021] The RT-DETR model mainly extracts features from the input image through the backbone network. Figure 2As shown in the figure, the backbone network includes multi-layer basic convolution modules ConvBN and multi-layer multi-scale dilated convolution modules CCM. The basic convolution module ConvBN is used to extract features from the image of the input model using the standard convolution kernel. Each basic convolution module contains a standard convolution submodule Conv, a batch normalization submodule BN and an activation function submodule Relu. The multi-scale dilated convolution module CCM is used to extract features from the image using standard convolution and multi-scale dilated convolution. Each multi-scale dilated convolution module contains a basic convolution submodule ConvBN, a lightweight downsampling submodule CG, a multi-scale dilated attention submodule MSDA and an activation function submodule Relu. By improving the BasicBlock module in the backbone network of the model to a multi-scale dilated convolution module CCM, the number of parameters and the amount of calculation of the improved model have been significantly reduced, and the context features are paid attention to through multi-scale dilated convolution, which ensures the accuracy of target detection.

[0022] S102, using the basic convolution module to extract basic features from the collected image data to obtain shallow image features.

[0023] The multi-layer basic convolution module ConvBN in the backbone network uses convolution kernels of different sizes to perform multi-layer feature extraction on the image of the input model in sequence, gradually extracting features from low-level to high-level, and finally obtaining shallow image features. When extracting shallow image features through multi-layer basic convolution modules, more attention is paid to local features in the image.

[0024] S103, using the lightweight downsampling submodule to perform hole convolution on shallow image features to extract context features, and fuse the context features with local features extracted by the lightweight downsampling submodule through standard convolution on the shallow image to obtain joint image features containing context.

[0025] In the traditional RT-DETR model, the BasicBlock module in the deep layer of the backbone network extracts deep image features based on shallow image features and serves as the input of the hybrid encoder for feature fusion. The BasicBlock module contains the basic convolution submodule ConvBN and the activation function submodule Relu. The ConvBN submodule is replaced by the lightweight downsampling submodule CG (CG Block, Context Guided Block). The lightweight downsampling submodule extracts the context information of local features by performing dilated convolution on shallow image features, improves scene understanding, and can enhance the semantic segmentation accuracy when extracting deep features of the backbone network. The context features extracted by dilated convolution are fused with the local features extracted by standard convolution at the same time. The fused joint image features contain the semantic information of the context. By replacing the standard convolution with the dilated convolution to extract the surrounding context of the local features, the number of parameters during feature extraction is reduced, thereby reducing the amount of calculation of the model.

[0026] S104, using the multi-scale dilation attention submodule to extract features of different scales from the joint image features at different dilation rates, and fusing the features of different scales to obtain deep image features.

[0027] The multi-scale dilated attention submodule MSDA is used as the subsequent processing of the lightweight downsampling submodule because the lightweight downsampling module reduces the computational complexity and parameter quantity of the model by introducing dilated convolution, but it is easy to produce insufficient detection accuracy of complex targets or small targets in high-density target scenes. The self-attention mechanism of the multi-scale dilated attention submodule is used to perform multi-scale SWDA with different dilation rates on the joint image features in different heads, and the features with different dilation rates extracted from different heads are aggregated through feature concatenation and linear layers to obtain deep image features containing multi-scale features. The features extracted by MSDA contain multi-scale semantic information. Especially in complex scenes, the semantic information of different scales can capture the information in the image more comprehensively. At the same time, the sparsity of the self-attention mechanism at different scales is used to reduce computational redundancy while ensuring detection performance.

[0028] S105, using the hybrid encoder of the RT-DETR model to perform feature fusion on the deep image features, and using the decoder of the RT-DETR model to perform target feature recognition, to obtain the target detection result of the image data.

[0029] The Efficient Hybrid Encoder receives the deep image features extracted from the deep CCM module of the backbone network, performs feature alignment and feature fusion, and the fused features contain shallow local details and deep semantic information, and are sent to the decoder. The decoder is composed of multiple layers of Transformer decoding layers. It performs multi-layer iterative decoding on the features input to the decoder and classifies the detected targets to obtain the final target detection result.

[0030] This embodiment improves the BasicBlock module in the backbone network of the RT-DETR model, replaces the original basic convolution submodule with a lightweight downsampling submodule and a multi-scale dilated attention submodule, and performs feature extraction by combining the standard convolution with the hole convolution in the lightweight downsampling submodule. The hole convolution can capture the surrounding contextual semantic information of the local features without changing the size of the convolution kernel, replacing the original method of expanding the size of the standard convolution kernel. The refined local features and surrounding contextual features extracted by the standard convolution and the hole convolution are then fused, and finally a joint image feature containing the context is obtained, which reduces the number of parameters and calculations of the model, and improves the accuracy of target detection by capturing contextual semantic information. Then, the multi-scale dilated attention submodule uses different dilation rates to extract features of different scales to capture multi-scale semantic information, and uses the sparsity of the self-attention mechanism at different scales to further reduce computational redundancy while ensuring the model detection performance. The deep image features extracted by the lightweight downsampling submodule and the multi-scale dilated attention submodule contain the details of local features, the surrounding context semantic information, and multi-scale semantic information. It can not only improve the accuracy of target detection through a variety of semantic information, but also reduce the number of parameters and calculations of the model, so that the RT-DETR model can be deployed on terminal devices or edge devices with limited computing power such as smart security cameras. Smart security cameras can perform real-time target detection and only transmit the detection results to the data center without transmitting image or video data, which greatly reduces resource usage.

[0031] An optional implementation of this embodiment is that the multi-scale dilated convolution module also includes a basic convolution submodule and an activation function submodule.

[0032] When the CCM module is used to replace the BasicBlock module to improve the RT-DETR model, the original BasicBlock module includes two basic convolution ConvBN submodules and one activation function Relu submodule. CG-MSDA is used to replace the second ConvBN submodule in the original BasicBlock module to form a ConvBN-CG-MSDA-Relu structure. After the data is input into the CCM module, it is processed and transmitted in the ConvBN submodule, CG submodule, MSDA submodule and Relu submodule in turn. Finally, the CCM module outputs deep image features. The CG submodule uses dilated convolution instead of standard convolution when extracting the surrounding context features of local features. The dilated convolution can expand the receptive field of the convolution kernel through the dilation rate, instead of extracting the surrounding context features of local features by increasing the size of the convolution kernel, which significantly reduces the number of parameters and calculations of the model. Then, the MSDA submodule is used for multi-scale feature extraction. MSDA can adaptively extract features of different targets from a multi-scale perspective, improving the feature extraction accuracy in complex environments. The sparse performance of the MSDA self-attention mechanism at different scales further reduces computational redundancy and reduces the amount of computation of the model while maintaining model performance. By combining the two, the richness of contextual information can be guaranteed in complex environments, and the detection accuracy of small targets, high-density targets, and targets of different scales in complex backgrounds can be improved.

[0033] Embodiment 2

[0034] Figure 3 This is a flow chart of a lightweight real-time target detection method described in Embodiment 2 of the present invention. This embodiment is optimized based on the above embodiment. In this embodiment, the lightweight downsampling submodule is used to perform hole convolution on shallow image features to extract context features, and the context features are fused with the local features extracted by the lightweight downsampling submodule through standard convolution on the shallow image to obtain joint image features containing context. The specific optimization is as follows: The lightweight downsampling submodule performs standard convolution processing on shallow image features to obtain refined local features, and performs hole convolution processing on shallow image features to obtain surrounding context features; The lightweight downsampling submodule performs feature fusion on the refined local features and surrounding context features to obtain fused features; The lightweight downsampling submodule enhances the key information of the fused features to obtain joint image features containing context.

[0035] Accordingly, the lightweight real-time target detection method provided in this embodiment specifically includes: S201, input the collected image data into the RT-DETR model, the backbone network of the RT-DETR model includes a basic convolution module and a multi-scale dilated convolution module, and the multi-scale dilated convolution module includes a lightweight downsampling submodule and a multi-scale dilated attention submodule.

[0036] S202, using the basic convolution module to extract basic features from the collected image data to obtain shallow image features.

[0037] S203, the lightweight downsampling submodule performs standard convolution processing on the shallow image features to obtain refined local features, and at the same time performs hole convolution processing on the shallow image features to obtain surrounding context features.

[0038] The lightweight downsampling submodule CG Block includes a local feature extractor ( ), surrounding context extractor ( ), Joint Feature Extractor ( ) and the global context extractor ( ),like Figure 4 As shown in the figure, firstly, the local feature extractor performs standard convolution on the input shallow image features to extract detailed local features, and at the same time, the surrounding context extractor performs dilated convolution on the input shallow image features in parallel to extract surrounding context features with a wider receptive field. The dilated convolution can expand the receptive field of the convolution kernel through the dilation rate, and no longer extracts surrounding context information by expanding the size of the standard convolution kernel, thus reducing the number of parameters and the amount of calculation.

[0039] S204, the lightweight downsampling submodule performs feature fusion on the refined local features and the surrounding context features to obtain fused features; A joint feature extractor is used to receive the refined local features and surrounding context features extracted in parallel by the local feature extractor and the surrounding context extractor, and the refined local features and surrounding context features are processed through feature concatenation, batch normalization and activation function. The generated fusion features contain local detail information and context semantic information.

[0040] S205, the lightweight downsampling submodule enhances the key information of the fused features to obtain a joint image feature containing the context.

[0041] The global context extractor is used to receive the fused features fused by the joint feature extractor, and the global context features are aggregated through global average pooling. Then, the key information is enhanced and refined through a multi-layer perceptron. The focus on key features is increased through weighted processing, and the accuracy of feature recognition is improved by using rich contextual semantic information.

[0042] S206, using the multi-scale expansion attention submodule to extract features of different scales from the joint image features at different expansion rates, and fusing the features of different scales to obtain deep image features.

[0043] S207, using the hybrid encoder of the RT-DETR model to perform feature fusion on the deep image features, and using the decoder of the RT-DETR model to perform target feature recognition, to obtain the target detection result of the image data.

[0044] This embodiment extracts detailed local features through standard convolution of the local feature extractor, and extracts surrounding context features through hole convolution of the surrounding context extractor in parallel, then performs feature fusion through the joint feature extractor, and then enhances key information through weighted processing through the global context extractor. The receptive field is expanded by the hole rate of the hole convolution, instead of the original method of expanding the standard convolution kernel to expand the receptive field, which reduces the number of parameters and calculations of the model, and improves the attention of key information through feature fusion and weighted processing, aggregates rich contextual semantic information, and improves the performance of the model.

[0045] Specifically, the lightweight downsampling submodule performs feature fusion on the refined local features and the surrounding context features to obtain fused features, including: The refined local features and surrounding context features are spliced ​​to obtain spliced ​​features.

[0046] After the joint feature extractor receives the refined local features and surrounding context features extracted in parallel by the local feature extractor and the surrounding context extractor, it first uses the Concat layer to concatenate the refined local features and the surrounding context features. The concatenated features generated after concatenation include the local features and the context features around the local features. The context features are extracted through hollow convolution, and the size of the convolution kernel remains unchanged. Instead of obtaining the context features by expanding the convolution kernel, the number of model parameters and the amount of calculation are reduced.

[0047] The concatenated features are batch normalized and the fused features are generated through the activation function.

[0048] Batch normalization (BN) is used to process the concatenated features to improve the stability and generalization ability of the model. Then, the parameterized ReLU (PReLU) activation function is used to improve the expressiveness of the model and enhance the nonlinear expression ability of the model. Finally, a fusion feature containing local details and contextual semantics is generated, ensuring the robustness and training efficiency of the model.

[0049] Specifically, the lightweight downsampling submodule enhances key information of the fused features to obtain a joint image feature containing context, including: Perform global average pooling on the fused features to obtain global features.

[0050] The fusion features of all channels are aggregated through average pooling, which retains the features between channels and reduces the number of model parameters and model complexity. The global features are smoothed through mean calculation to improve the robustness of the model.

[0051] A multi-layer perceptron is used to enhance the key information of global features and obtain joint image features containing context.

[0052] The multilayer perceptron consists of two fully connected layers FC, which use two fully connected layers to weight the global features. By increasing the weight of key information, the key information is strengthened and irrelevant information is suppressed. The fully connected layer can adaptively strengthen the key information of different input images through weight learning, thereby improving the accuracy of the model.

[0053] Embodiment 3

[0054] Figure 5 This is a flow chart of a lightweight real-time target detection method according to Embodiment 3 of the present invention. This embodiment is optimized based on the above embodiment. In this embodiment, the multi-scale expansion attention submodule is used to extract features of different scales from the joint image features at different expansion rates, and features of different scales are fused to obtain deep image features. The specific optimization is as follows: The multi-scale dilated attention submodule uses multiple heads to perform multi-scale SWDA operations on the joint image features with different dilation rates to obtain the head features of each head.

[0055] The multi-scale dilated attention submodule uses a linear layer to aggregate all head features to obtain deep image features.

[0056] Accordingly, the lightweight real-time target detection method provided in this embodiment specifically includes: S301, input the collected image data into the RT-DETR model, the backbone network of the RT-DETR model includes a basic convolution module and a multi-scale dilated convolution module, and the multi-scale dilated convolution module includes a lightweight downsampling submodule and a multi-scale dilated attention submodule.

[0057] S302: Utilize the basic convolution module to extract basic features from the collected image data to obtain shallow image features.

[0058] S303, using the lightweight downsampling submodule to perform hole convolution on shallow image features to extract context features, and fuse the context features with local features extracted by the lightweight downsampling submodule through standard convolution on the shallow image to obtain joint image features containing context.

[0059] S304, the multi-scale dilation attention submodule uses multiple heads to perform multi-scale SWDA operations on the joint image features with different dilation rates to obtain head features of each head.

[0060] The multi-scale dilated attention submodule first performs a linear projection on the input feature map, such as Figure 6 As shown in the figure, the query, key and value are obtained, and then different heads are used to perform multi-scale SWDA (Sliding Window Dilated Attention) operations with different expansion rates to finely capture target features and semantic information at different scales. It can adapt to factors such as different sizes, shapes and backgrounds of the targets, and adaptively process different targets through the self-attention mechanism.

[0061] S305, the multi-scale dilated attention submodule uses a linear layer to perform feature aggregation on all head features to obtain deep image features.

[0062] The features output by all heads are concatenated and then aggregated by the linear layer. The final output deep image features contain multi-scale semantic information, which can effectively improve the detection performance of the model.

[0063] S306, the hybrid encoder uses the deep image features generated by the last three multi-scale dilated convolution modules in the backbone network to perform multi-scale feature fusion to obtain encoder features.

[0064] This embodiment uses a multi-scale dilated attention submodule to perform SWDA of different scales on different heads with different dilation rates, and then performs feature aggregation on the outputs of all heads through a linear layer, and uses a multi-scale self-attention mechanism to capture semantic information and target features of different scales, thereby improving the detection accuracy of the model for small targets, high-density targets and targets of different sizes in complex environments, and can effectively make up for the deficiencies of the lightweight downsampling submodule in detecting complex targets, small targets and high-density scenes. At the same time, the computational overhead is further reduced through attention sparsification, while still ensuring the detection performance of the model.

[0065] Optionally, the hybrid encoder of the RT-DETR model is used to perform feature fusion on deep image features, and the decoder of the RT-DETR model is used to perform target feature recognition to obtain target detection results of the image data, including: The hybrid encoder uses the deep image features generated by the last three multi-scale dilated convolution modules in the backbone network to perform multi-scale feature fusion to obtain encoder features.

[0066] In the backbone network of the RT-DETR model, the deep multi-scale dilated convolution module is mainly used to extract deep semantic information and improve the model's ability to understand features and detection accuracy. The multi-scale dilated convolution modules at the p3, p4, and p5 levels output feature maps of different scales to the hybrid encoder of the RT-DETR model, such as Figure 2 As shown in the figure, the hybrid encoder first uses the self-attention mechanism to independently process the feature maps of each scale through the attention-based intra-scale feature interaction module (AIFI) to enhance the expressiveness of the features. Then, the cross-scale feature fusion module (CCFM) is used to fuse the features of different scales to generate encoder features and transmit them to the decoder.

[0067] The decoder iterates and predicts the features of the encoder to obtain the target detection results of the image data.

[0068] After receiving the encoder features output by the hybrid encoder, the decoder performs initial query selection (Uncertainty-Minimal Query Selection), then adds position embedding (position embedding) to each query, and then performs decoding iterations by multi-layer Transformer modules. Each layer of Transformer modules contains self-attention mechanism and cross-attention mechanism, and uses the prediction head to classify the target and determine the bounding box, and finally outputs the target detection result.

[0069] Embodiment 4

[0070] Figure 7 This is a schematic diagram of the structure of a lightweight real-time target detection device according to Embodiment 4 of the present invention. In this embodiment, the lightweight real-time target detection device includes: An image acquisition unit 810, for acquiring image data and transmitting it to the RT-DETR model; A basic convolution unit 820, for performing standard convolution on the collected image data; A multi-scale dilated convolution unit 830, used to perform multi-scale dilated convolution on shallow image features generated after standard convolution; A hybrid encoder unit 840, for performing feature fusion on deep image features generated after multi-scale dilated convolution; The decoder unit 850 is used to perform object detection after feature fusion of the hybrid encoder.

[0071] This embodiment performs multi-scale dilated convolution on shallow image features through the dilated convolution unit, and can use dilated convolution to expand the receptive field and extract surrounding contextual signs without increasing the size of the standard convolution kernel, thereby reducing the number of model parameters and the amount of calculation. Multi-scale feature extraction is then performed through dilated convolutions with different expansion rates, which better aggregates the semantic information of the context, making target detection more accurate, and enabling real-time target detection under limited computing power, reducing resource usage.

[0072] On the basis of the above embodiments, the multi-scale dilated convolution unit further includes: A lightweight downsampling subunit is used to perform standard convolution and dilated convolution on shallow feature image features to jointly extract joint image features containing context; Multi-scale dilated attention submodule for multi-scale feature extraction of joint image features.

[0073] Based on the above embodiments, the lightweight downsampling subunit further includes: Local feature extractor, used to perform standard convolution extraction on shallow image features to refine local features; A surrounding context extractor is used to extract surrounding context features by performing dilated convolution on shallow image features; Joint feature extractor for feature fusion of refined local features and surrounding context features; The global context extractor is used to enhance the key information of the fused features.

[0074] The lightweight real-time target detection device provided in the embodiment of the present invention can execute the lightweight real-time target detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0075] Embodiment 5

[0076] Figure 8 This is a structural diagram of a server according to Embodiment 5 of the present invention. Figure 8 A block diagram of an exemplary server 12 suitable for use in implementing embodiments of the present invention is shown. Figure 8 The server 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0077] like Figure 8 As shown, the server 12 is in the form of a general purpose computing device. The components of the server 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0078] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0079] The server 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the server 12, including volatile and non-volatile media, removable and non-removable media.

[0080] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 8 not shown, usually called a "hard drive"). Although Figure 8 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0081] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0082] The server 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the device / server / server 12, and / or may communicate with any device that enables the server 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. In addition, the server 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the server 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the server 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0083] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the lightweight real-time target detection method provided by the embodiment of the present invention.

[0084] Embodiment 6

[0085] Embodiment 6 of the present invention further provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to execute the lightweight real-time target detection method provided in the above embodiments.

[0086] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device.

[0087] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0088] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0089] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0090] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A lightweight real-time target detection method, characterized in that: include: Input the collected image data into the RT-DETR model, the backbone network of which includes a basic convolution module and a multi-scale dilated convolution module, and the multi-scale dilated convolution module includes a lightweight downsampling submodule and a multi-scale dilated attention submodule; Using the basic convolution module to extract basic features from the collected image data to obtain shallow image features; Utilize the lightweight downsampling submodule to perform hole convolution on shallow image features to extract context features, fuse the context features with local features extracted by standard convolution on the shallow image by the lightweight downsampling submodule, and obtain joint image features containing context; The multi-scale expansion attention submodule is used to extract features of different scales from the joint image features at different expansion rates, and features of different scales are fused to obtain deep image features; The hybrid encoder of the RT-DETR model is used to fuse the deep image features, and the decoder of the RT-DETR model is used to identify the target features to obtain the target detection results of the image data; The method of using the light downsampling submodule to perform hole convolution on shallow image features to extract context features, and fusing the context features with local features extracted by the light downsampling submodule through standard convolution on the shallow image to obtain joint image features containing context, includes: The lightweight downsampling submodule performs standard convolution processing on shallow image features to obtain refined local features, and performs hole convolution processing on shallow image features to obtain surrounding context features; The lightweight downsampling submodule performs feature fusion on the refined local features and surrounding context features to obtain fused features; The lightweight downsampling submodule enhances the key information of the fused features to obtain joint image features containing context.

2. The method according to claim 1, characterized in that The lightweight downsampling submodule performs feature fusion on the refined local features and the surrounding context features to obtain fused features, including: Perform feature splicing on the refined local features and surrounding context features to obtain spliced ​​features; The concatenated features are batch normalized and the fused features are generated through the activation function.

3. The method according to claim 1, characterized in that The lightweight downsampling submodule enhances the key information of the fused features to obtain a joint image feature containing context, including: Perform global average pooling on the fused features to obtain global features; A multi-layer perceptron is used to enhance the key information of global features and obtain joint image features containing context.

4. The method according to claim 1, characterized in that The multi-scale expansion attention submodule is used to extract features of different scales from the joint image features at different expansion rates, and features of different scales are fused to obtain deep image features, including: The multi-scale dilation attention submodule uses multiple heads to perform multi-scale SWDA operations on the joint image features with different dilation rates to obtain the head features of each head; The multi-scale dilated attention submodule uses a linear layer to aggregate all head features to obtain deep image features.

5. The method according to claim 1, characterized in that The hybrid encoder of the RT-DETR model is used to perform feature fusion on deep image features, and the decoder of the RT-DETR model is used to perform target feature recognition to obtain target detection results of image data, including: The hybrid encoder uses the deep image features generated by the last three multi-scale dilated convolution modules in the backbone network to perform multi-scale feature fusion to obtain encoder features; The decoder iterates and predicts the features of the encoder to obtain the target detection results of the image data.

6. The method according to claim 1, characterized in that: The multi-scale dilated convolution module also includes a basic convolution submodule and an activation function submodule.

7. A lightweight real-time target detection device, characterized in that: include: An image acquisition unit, used to acquire image data and transmit it to the RT-DETR model; The basic convolution unit is used to perform standard convolution on the collected image data; Multi-scale dilated convolution unit, used to perform multi-scale dilated convolution on shallow image features generated after standard convolution; A hybrid encoder unit for feature fusion of deep image features generated after multi-scale dilated convolution; The decoder unit is used for object detection after feature fusion from the hybrid encoder.

8. A server, characterized in that: The server comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the lightweight real-time target detection method as described in any one of claims 1-6.

9. A storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to perform the lightweight real-time target detection method as described in any one of claims 1 to 6.

Images