Lightweight target detection method, device, equipment, and storage medium

Abstract

The present invention provides a lightweight target detection method, apparatus, device, and storage medium, comprising: obtaining an image of a target to be detected; performing dimensionality reduction on the image through a first convolutional layer to obtain a reduced-dimensional image; compressing the reduced-dimensional image through multiple second convolutional layers to obtain multiple first branches; extracting first feature maps from each of the multiple first branches and concatenating the first feature maps of the multiple first branches to obtain a first concatenated feature map; downsampling the first concatenated feature map through a first pooling layer to obtain a sampled feature map; performing feature extraction on the sampled feature map through multiple block modules to obtain a feature map of the image; and performing detection based on the feature map of the image. In the feature extraction stage, the present invention adopts a cross-channel branching strategy to concatenate the feature maps of multiple branches as the basis for subsequent feature extraction, thereby expanding the reception range and retaining more low-level functions, thereby ensuring detection speed while improving accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a lightweight target detection method, device, equipment, and storage medium. Background Art

[0002] Object detection is a fundamental aspect of computer vision and is widely used in fields such as autonomous driving and safety inspection. With the recent success of deep learning in image classification tasks, convolutional neural network (CNN)-based object detection networks have gradually become mainstream. Common CNN-based object detection networks include Faster R-CNN, R-FCN, SSD, and YOLO. These object detection networks rely on complex network structures, with computational overhead reaching five figures in the millions of floating point operations (MFLOPs). They can run accurately and quickly on server GPUs. However, due to the limited computing power and memory on mobile devices, they cannot handle the excessive network parameters and computational requirements. Clearly, these object detection networks are not suitable for real-time deployment and application in mobile scenarios. Existing lightweight object detection networks include MobileNet-SSD, MobileNetV2-SSD Lite, Tiny-YOLO, and D-YOLO. However, these lightweight object detection networks do not strike a good balance between accuracy and model complexity. Summary of the Invention

[0003] In order to address the deficiencies of the prior art, the present invention provides a lightweight target detection method, apparatus, device, and storage medium, which can improve the accuracy of target detection while ensuring the detection speed.

[0004] The specific technical solution proposed by the present invention is: a lightweight target detection method, the target detection method comprising:

[0005] Acquire an image of the target to be detected;

[0006] Performing dimensionality reduction on the image through a first convolutional layer to obtain a dimensionality-reduced image;

[0007] Compressing the reduced-dimensional image through multiple second convolutional layers to obtain multiple first branches, each of which has the same number of channels;

[0008] Extracting first feature maps of the plurality of first branches respectively and splicing the first feature maps of the plurality of first branches to obtain a first spliced ​​feature map, wherein the first feature maps of the plurality of first branches are sequentially increased in depth;

[0009] Downsample the first concatenated feature map through the first pooling layer to obtain a sampled feature map;

[0010] Extract features from the sampled feature map through multiple block modules to obtain a feature map of the image;

[0011] Detection is performed based on the feature map of the image to obtain a detection result of the target to be detected.

[0012] Furthermore, before extracting features from the sampled feature map through a plurality of block modules to obtain the feature map of the image, the target detection method further includes:

[0013] Compressing the sampled feature maps through multiple second convolutional layers to obtain multiple second branches, each of which has the same number of channels;

[0014] Extracting the second feature maps of the plurality of second branches respectively and splicing the second feature maps of the plurality of second branches to obtain a second spliced ​​feature map, wherein the second feature maps of the plurality of second branches are sequentially increased in depth;

[0015] Accordingly, the sampled feature map is subjected to feature extraction through multiple block modules to obtain the feature map of the image, including:

[0016] The second spliced ​​feature map is subjected to feature extraction through multiple block modules to obtain a feature map of the image.

[0017] Furthermore, for the first branch / second branch with the smallest depth, the reduced-dimensional image / sampled feature map is compressed through a second convolutional layer, including:

[0018] The reduced-dimensional image / sampled feature map is sequentially passed through the second pooling layer and the second convolutional layer for pooling and compression respectively.

[0019] Furthermore, respectively extracting the plurality of first feature maps of the first branches / the second feature maps of the second branches and splicing the plurality of first feature maps of the first branches / the second feature maps of the second branches to obtain a first spliced ​​feature map / a second spliced ​​feature map includes:

[0020] The output of the previous first branch / second branch is used as the residual part of the next first branch / second branch, and the features of the next first branch / second branch with the same depth as the residual part are fused with the residual part to obtain a cross-branch feature map after the fusion of multiple first branches / second branches;

[0021] The cross-branch feature maps after the fusion of multiple first branches / second branches are subjected to feature extraction by the third convolutional layer to obtain multiple first feature maps of the first branches / second feature maps of the second branches respectively;

[0022] The first feature maps of the multiple first branches are spliced ​​and fused with the image after dimensionality reduction / the second feature maps of the multiple second branches are spliced ​​and fused with the sampled feature map to obtain a first spliced ​​feature map / a second spliced ​​feature map.

[0023] Furthermore, the sampled feature map / the second spliced ​​feature map is subjected to feature extraction through a plurality of block modules to obtain a feature map of the image, including:

[0024] Passing the sampled feature map / the second spliced ​​feature map through multiple block modules in sequence to obtain a first-scale feature map, a second-scale feature map, and a third-scale feature map;

[0025] Downsampling the third scale feature map to obtain a fourth scale feature map;

[0026] Upsampling the third scale feature map and the fourth scale feature map respectively to obtain a first upsampled feature map and a second upsampled feature map;

[0027] The second-scale feature map, the first up-sampled feature map, and the second up-sampled feature map are fused to obtain a feature map of the image.

[0028] Furthermore, performing detection according to the feature map of the image to obtain a detection result of the target to be detected includes:

[0029] Passing the feature map of the image through the RPN network to obtain the feature map of the image in the RPN network and the candidate box containing the target to be detected;

[0030] Generate a channel attention feature map according to the feature map of the image in the RPN network;

[0031] Fusing the channel attention feature map with the feature map of the image to obtain a fused feature map;

[0032] Obtaining a feature map of the target to be detected according to the candidate frame and the fused feature map;

[0033] A detection result of the target to be detected is obtained according to the feature map of the target to be detected.

[0034] Furthermore, generating a channel attention feature map according to the feature map of the image in the RPN network includes:

[0035] Splitting the feature map of the image in the RPN network into a first sub-feature map and a second sub-feature map, where the number of channels of the first sub-feature map and the second sub-feature map are equal;

[0036] Obtaining a channel attention weight according to the first sub-feature map;

[0037] Multiplying the channel attention weight by the second sub-feature map obtains a channel attention feature map.

[0038] The present invention also provides a lightweight target detection device, comprising:

[0039] An acquisition module is used to acquire an image of a target to be detected;

[0040] A dimensionality reduction module, configured to reduce the dimensionality of the image through a first convolutional layer to obtain a reduced-dimensional image;

[0041] A compression module, configured to compress the image after dimensionality reduction through a plurality of second convolutional layers to obtain a plurality of first branches, wherein the plurality of first branches have the same number of channels;

[0042] a splicing module, configured to respectively extract the first feature maps of the plurality of first branches and splice the first feature maps of the plurality of first branches to obtain a first spliced ​​feature map, wherein the first feature maps of the plurality of first branches are sequentially increased in depth;

[0043] A sampling module, configured to downsample the first concatenated feature map through the first pooling layer to obtain a sampled feature map;

[0044] A feature extraction module is used to extract features from the sampled feature map through multiple block modules to obtain a feature map of the image;

[0045] The detection module is used to perform detection based on the feature map of the image to obtain the detection result of the target to be detected.

[0046] The present invention also provides a device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the target detection method as described in any one of the above items.

[0047] The present invention also provides a computer-readable storage medium having computer instructions stored thereon, and when the computer instructions are executed by a processor, the target detection method as described in any one of the above items is implemented.

[0048] The target detection method proposed in the present invention first compresses the reduced-dimensional image through multiple second convolutional layers in the feature extraction stage to obtain multiple first branches, then extracts the first feature maps of the multiple first branches respectively and splices the first feature maps of the multiple first branches to obtain a first spliced ​​feature map, and adopts a cross-channel branching strategy to splice the feature maps of the multiple branches as the basis for subsequent feature extraction, thereby expanding the receiving range and retaining more low-level functions through information interaction between multiple channel branches, thereby improving the accuracy of target detection while ensuring the detection speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The technical solutions and other beneficial effects of the present invention will be made apparent by describing in detail the specific embodiments of the present invention in conjunction with the accompanying drawings.

[0050] Figure 1 Schematic diagram of the target detection method in an embodiment of the present application;

[0051] Figure 2 Schematic diagram of a cross-channel branch feature extraction module in an embodiment of the present application;

[0052] Figure 3 2 is another schematic diagram of a cross-channel branch feature extraction module in an embodiment of the present application;

[0053] Figure 4 Schematic diagram of a multi-scale feature fusion module in an embodiment of the present application;

[0054] Figure 5 Schematic diagram of a detection network in an embodiment of the present application;

[0055] Figure 6 Schematic diagram of a channel self-attention network in an embodiment of the present application;

[0056] Figure 7 is a schematic diagram of a target detection device in an embodiment of the present application;

[0057] Figure 8 Schematic diagram of the structure of the device in the embodiment of the present application. DETAILED DESCRIPTION

[0058] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention may be implemented in many different forms, and the present invention should not be construed as being limited to the specific embodiments set forth herein. Instead, these embodiments are provided to explain the principles of the present invention and its practical application, thereby enabling others skilled in the art to understand the various embodiments of the present invention and various modifications suitable for specific intended applications. In the accompanying drawings, the same reference numerals will be used throughout to represent the same elements.

[0059] CNN-based object detection networks are divided into two categories based on whether they include a region proposal network (RPN): one-stage and two-stage. One-stage object detection networks can directly regress and predict object categories and bounding boxes from feature maps. Their simpler network structure and higher efficiency make them more suitable for lightweight research. Two-stage object detection networks, however, can achieve better detection performance due to the added step of candidate region selection. Currently, most research on lightweight object detection networks is based on one-stage networks, such as MobileNet-SSD, MobileNetV2-SSD Lite, Tiny-YOLO, D-YOLO, and Pelee. There are also two-stage lightweight object detection networks, such as Light-Head R-CNN. However, existing one-stage and two-stage lightweight object detection networks struggle to strike a good balance between accuracy and model complexity.

[0060] Based on the above problems, the present application provides a lightweight target detection method, which is a two-stage lightweight target detection method, including a feature extraction stage and a detection stage. In the feature extraction stage, a cross-channel branch strategy is adopted to add cross-channel branches to the structure of the existing lightweight classification network, and the feature maps of multiple branches are spliced ​​as the basis for subsequent feature extraction, thereby expanding the receiving range and retaining more low-level functions through information interaction between multiple channel branches, thereby improving the accuracy of target detection while ensuring the detection speed. Specifically, the present application first obtains an image of the target to be detected, reduces the dimension of the image of the target to be detected through a first convolutional layer to obtain a reduced-dimensional image, compresses the reduced-dimensional image into multiple first branches with the same number of channels through a second convolutional layer, extracts the first feature maps of the multiple first branches respectively, and splices the first feature maps of the multiple first branches to obtain a first spliced ​​feature map, wherein the first feature maps of the multiple first branches increase in depth in sequence, and then downsamples the first spliced ​​feature map through a first pooling layer to obtain a sampled feature map, extracts features from the sampled feature map through multiple block modules to obtain a feature map of the image, and finally performs detection based on the feature map of the image to obtain a detection result of the target to be detected.

[0061] The following is a detailed description of the target detection method of the present application using the lightweight classification network ShuffleNetV2 as an example of the lightweight classification network of the present application. It should be noted that the lightweight classification network ShuffleNetV2 is used as the lightweight classification network of the present application only as an example and is not used to limit the target detection method of the present application. The lightweight classification network of the present application can also adopt other lightweight classification networks, such as Tiny-Darknet, MoblieNetV2, PeleeNet, etc.

[0062] Reference Figure 1 The lightweight target detection method provided in this embodiment includes the following steps:

[0063] S1. Acquire an image of the target to be detected;

[0064] S2. Reduce the dimension of the image through the first convolutional layer to obtain a reduced-dimensional image;

[0065] S3, compressing the reduced-dimensional image through multiple second convolutional layers to obtain multiple first branches, wherein the multiple first branches have the same number of channels;

[0066] S4, respectively extracting the first feature maps of the plurality of first branches and splicing the first feature maps of the plurality of first branches to obtain a first spliced ​​feature map, wherein the first feature maps of the plurality of first branches are sequentially increased in depth;

[0067] S5. Downsample the first concatenated feature map through the first pooling layer to obtain a sampled feature map;

[0068] S6. Extract features from the sampled feature map through multiple block modules to obtain a feature map of the image;

[0069] S7. Perform detection based on the feature map of the image to obtain a detection result of the target to be detected.

[0070] The feature extraction network of the target detection method in this embodiment is an improvement based on the lightweight classification network ShuffleNetV2. The specific structure of the feature extraction network is shown in the following table, which includes a first convolution layer (Convolution), a cross-channel branch feature extraction module, a first pooling layer (MaxPooling), and multiple block modules (ShuffleV2 block) cascaded in sequence. The first convolution layer (Convolution) and the first pooling layer (MaxPooling) are the stem stage, and the multiple block modules are the stage2, stage3, and stage4 stages. Specifically, the feature extraction network includes 16 block modules, among which the stage2 stage includes a block module with a step size of 2 and 3 block modules with a step size of 1, the stage3 stage includes a block module with a step size of 2 and 7 block modules with a step size of 1, and the stage4 stage includes a block module with a step size of 2 and 3 block modules with a step size of 1.

[0071] Table 1 Structure of feature extraction network

[0072]

[0073] The target detection method in this embodiment adopts a cross-channel branching strategy. A cross-channel branch feature extraction module is added to the stem stage of the ShuffleNet V2 network. The feature maps of multiple branches are spliced ​​together and serve as the basis for feature extraction in subsequent stages 2, 3, and 4. This expands the reception range and retains more low-level functions through information exchange between multiple channel branches, thereby improving the accuracy of target detection while ensuring detection speed.

[0074] Specifically, the image of the target to be detected obtained in step S1 is input into the first convolution layer (Convolution), the convolution kernel size of the first convolution layer (Convolution) is 3╳3, the step size is 2, and the image is reduced in dimension through the first convolution layer (Convolution) to obtain a reduced-dimensional image.

[0075] Reference Figure 2, the cross-channel branch feature extraction module in this embodiment includes multiple branch modules and a splicing layer (Concat). The multiple branch modules are used to compress the image after dimensionality reduction into multiple first branches with the same number of channels and extract the first feature maps of the multiple first branches. The first feature maps of the multiple first branches increase in depth. Each branch module includes a second convolution layer (1╳1Conv) with a convolution kernel size of 1╳1. The multiple branch modules compress the image after dimensionality reduction into multiple first branches with the same number of channels through the second convolution layer (1╳1 Conv). The first branch module only includes the second convolution layer (1╳1 Conv). Starting from the second branch module, each branch module includes a convolution layer (3╳3 Conv) with a convolution kernel size of 3╳3 and the number of convolution layers (3╳3 Conv) increases successively to extract the first feature maps of the multiple first branches that increase in depth. Finally, the first feature maps of the multiple first branches are spliced ​​together through the splicing layer (Concat) to obtain the first spliced ​​feature map. Figure 2 It shows the case where the feature extraction module includes 4 branch modules, and the first branches of the 4 branch modules are a1 to a4 respectively. Of course, this is only for illustration and is not intended to be limiting. The number of branch modules can be set according to actual needs.

[0076] Preferably, the first branch module in this embodiment adds a second pooling layer (Pool) in front of the second convolutional layer (1╳1 Conv), and first performs pooling processing on the reduced-dimensional image through the second pooling layer (Pool) to increase the receptive field of the first branch module, retain the main features while reducing parameters. In addition, based on the first branch module including the second pooling layer (Pool), in order to retain more information of the original image, the cross-channel branch feature extraction module adds a branch module a0 that only includes the second convolutional layer (1╳1 Conv) in front of the first branch module. This branch module performs channel compression on the reduced-dimensional image through the second convolutional layer (1╳1 Conv) and directly outputs it to the concatenation layer (Concat).

[0077] Reference Figure 3 As the network depth increases, the model will degenerate. To solve this problem, a residual connection is added to the cross-channel branch feature extraction module in this embodiment. The output of the previous branch is used as the residual part of the next branch and is fused with the features of the same depth of the next branch before feature extraction. Figure 3 Shown in Figure 2 The structure of the cross-channel branch feature extraction module with residual connection is added on the basis of the structure of the cross-channel branch feature extraction module in . Specifically, step S4 includes:

[0078] S41, using the output of the previous first branch as the residual part of the next first branch and fusing the features of the next first branch with the same depth as the residual part with the residual part to obtain a cross-branch feature map after the fusion of the first branch;

[0079] S42, performing feature extraction on the cross-branch feature map obtained by fusion of the first branches through a third convolutional layer to obtain first feature maps of multiple first branches respectively;

[0080] S43: Splice the first feature maps of the multiple first branches and fuse them with the image after dimensionality reduction to obtain a first spliced ​​feature map.

[0081] Taking the case where the feature extraction module includes 4 branch modules, and the first branches of the 4 branch modules are a1 to a4 as an example, the output of the previous first branch is used as the residual part of the next first branch, and the features of the next first branch with the same depth as the residual part are fused with the residual part. Specifically, the output of the first branch module is fused with the feature map obtained by the second branch module after the second convolution layer (1╳1 Conv) to obtain the fused cross-branch feature map of the second branch module, and then the fused cross-branch feature map of the second branch module is subjected to feature extraction through the third convolution layer to obtain the first feature map of the second branch module; the output of the second branch module is fused with the feature map obtained after the third branch module passes through the second convolution layer (1╳1 Conv) and the convolution layer (3╳3 Conv) to obtain the fused cross-branch feature map of the third branch module, and then the fused cross-branch feature map of the third branch module is subjected to feature extraction through the third convolution layer to obtain the first feature map of the third branch module; the output of the third branch module is fused with the feature map obtained after the fourth branch module passes through the second convolution layer (1╳1 Conv) to obtain the first feature map of the third branch module. Conv), the feature maps obtained after the two convolution layers (3╳3 Conv) are fused to obtain the fused cross-branch feature map of the fourth branch module, and then the fused cross-branch feature map of the fourth branch module is extracted through the third convolution layer to obtain the first feature map of the fourth branch module. Among them, the output γ of the four branch modules is i It is expressed as follows:

[0082]

[0083] Among them, α represents the convolution operation on the reduced-dimensional image through the second convolutional layer (1╳1 Conv), S represents the convolution operation through the convolutional layer (3╳3 Conv), i∈{1,2,......,k}, k is the number of branch modules. It should be noted here that α1 is the convolution operation performed on the reduced-dimensional image after the second pooling layer (Pool) and then the convolution operation is performed through the second convolutional layer (1╳1 Conv).

[0084] After obtaining the output of the 4-branch module γ i Finally, the outputs of the four branch modules and the output of branch module a0 are input to the concatenation layer (Concat) for concatenation and fusion with the reduced-dimensional image to obtain the first concatenated feature map. It should be noted that the third convolutional layer refers to the convolutional layer (3╳3Conv) connected to the concatenation layer (Concat) in each branch module. The same principle applies to the case where the number of branch modules is greater than four, so it will not be repeated here.

[0085] The cross-channel branch feature extraction module in this embodiment can supplement the original input features by adding residual connections, thereby preventing model degradation.

[0086] In step S5, the first spliced ​​feature map is input into the first pooling layer (MaxPooling) for downsampling to obtain the sampled feature map. The convolution kernel size of the first pooling layer (MaxPooling) is 3╳3, the step size is 2, and the first pooling layer (MaxPooling) adopts the maximum pooling method for pooling. Downsampling can further reduce the amount of data calculation.

[0087] In order to further expand the network's reception range and retain more details, in another implementation of this embodiment, a cross-channel branch feature extraction module is also added after the first pooling layer (MaxPooling) of the ShuffleNetV2 network. At this time, the structure of the feature extraction network in another implementation of this embodiment is shown in the following table:

[0088] Table 2 Another structure of feature extraction network

[0089]

[0090]

[0091] In another embodiment of the present invention, the target detection method further includes, before step S6:

[0092] S600: compress the sampled feature maps through a second convolutional layer (1╳1 Conv) to obtain multiple second branches, wherein the multiple second branches have the same number of channels;

[0093] S601. Extract the second feature maps of the plurality of second branches respectively and splice the second feature maps of the plurality of second branches to obtain a second spliced ​​feature map, wherein the second feature maps of the plurality of second branches increase in depth.

[0094] Specifically, the sampled feature map is input again into the cross-channel branch feature extraction module, and multiple branch modules are used to compress the sampled feature map into multiple second branches with the same number of channels and extract the second feature maps of the multiple second branches, and the second feature maps of the multiple second branches increase in depth. The sampled feature map is compressed into multiple second branches with the same number of channels through the second convolutional layer (1╳1 Conv), and then the second feature maps of the multiple second branches that increase in depth are extracted through the convolutional layer (3╳3 Conv). Finally, the second feature maps of the multiple second branches are spliced ​​together through the splicing layer (Concat) to obtain the second spliced ​​feature map. The acquisition process of the second spliced ​​feature map is the same as that of the first spliced ​​feature map, and will not be repeated here.

[0095] Similarly, in order to solve the problem of model degradation, a residual connection is also added to the cross-channel branch feature extraction module after the first pooling layer (MaxPooling). That is, the output of the previous branch is used as the residual part of the next branch and is fused with the features of the same depth of the next branch before feature extraction. For the specific structure, see Figure 3 As shown, step S601 includes:

[0096] S6011, using the output of the previous second branch as the residual part of the next second branch and fusing the features of the next second branch with the same depth as the residual part with the residual part to obtain a cross-branch feature map after the second branch fusion;

[0097] S6012: Perform feature extraction on the cross-branch feature map obtained by fusion of the second branches through a third convolutional layer to obtain a plurality of second feature maps of the second branches respectively;

[0098] S6013: Splice the second feature maps of the multiple second branches and fuse them with the image after dimensionality reduction to obtain a second spliced ​​feature map.

[0099] The process of obtaining the second spliced ​​feature map after adding the residual connection to the cross-channel branch feature extraction module is the same as the process of obtaining the first spliced ​​feature map after adding the residual structure to the cross-channel branch feature extraction module, and will not be repeated here.

[0100] In step S6, feature extraction is performed on the sampled feature map through multiple block modules to obtain a feature map of the image. Specifically, feature extraction is performed on the second spliced ​​feature map through multiple block modules to obtain a feature map of the image.

[0101] Since the lightweight network has a weak ability to extract features and cannot retain a large number of channel features, preferably, the target detection method of this embodiment adds a multi-scale feature fusion module on the basis of the ShuffleNet V2 network. The multi-scale feature fusion module fuses the features output by the stage3 and stage4 stages of the ShuffleNet V2 network, thereby combining low-resolution information with high-resolution information, which can effectively supplement the global context information between the multi-scale feature maps.

[0102] Reference Figure 4 The following describes in detail the process of applying the multi-scale feature fusion module to the target detection method of this embodiment. Step S6 includes:

[0103] S61, pass the sampled feature map through multiple block modules to obtain a first scale feature map, a second scale feature map, and a third scale feature map in sequence;

[0104] S62. Downsample the third-scale feature map to obtain a fourth-scale feature map;

[0105] S63, upsampling the third scale feature map and the fourth scale feature map respectively to obtain a first upsampled feature map and a second upsampled feature map;

[0106] S64: Fuse the second scale feature map, the first upsampled feature map, and the second upsampled feature map to obtain a feature map F of the image. mfm .

[0107] In this embodiment, the first-scale feature map is the output of the stage 2 stage. Specifically, the feature map obtained by sequentially passing the second spliced ​​feature map through a block module with a step size of 2 and three block modules with a step size of 1 is the first-scale feature map; the second-scale feature map is the output of the stage 3 stage. Specifically, the feature map obtained by sequentially passing the first-scale feature map through a block module with a step size of 2 and seven block modules with a step size of 1 is the second-scale feature map; the third-scale feature map is the output of the stage 4 stage. Specifically, the feature map obtained by sequentially passing the second-scale feature map through a block module with a step size of 2 and three block modules with a step size of 1 is the third-scale feature map.

[0108] When the detection result is classification, high-resolution information has a greater impact on classification than low-resolution information. Therefore, in order to retain more high-resolution information, in step S62, the third-scale feature map is downsampled to obtain a fourth-scale feature map having more high-resolution information. Preferably, to further reduce the amount of data calculation, this embodiment uses depthwise separable convolution (3×3 DW Conv) to downsample the third-scale feature map, with a convolution kernel size of 3×3.

[0109] To ensure data dimensionality consistency, in step S63, the third scale feature map and the fourth scale feature map need to be upsampled so that the data dimensions of the obtained first upsampled feature map and the second upsampled feature map are consistent with the data dimensions of the second scale feature map. Preferably, this embodiment uses a bilinear interpolation method to upsample the third scale feature map and the fourth scale feature map, and the upsampling is achieved by a convolution layer with a convolution kernel size of 1×1 (1×1 Conv).

[0110] Preferably, in order to further ensure the consistency of data dimensions, in step S63, it is necessary to adjust the dimension of the second scale feature map to obtain a dimension-upgraded feature map. The dimension adjustment is achieved by a convolution layer (1╳1 Conv) with a convolution kernel size of 1╳1, thereby ensuring that the data dimension of the dimension-upgraded feature map is consistent with the data dimension of the first upsampled feature map and the second upsampled feature map. Correspondingly, in step S64, the second scale feature map, the first upsampled feature map, and the second upsampled feature map are fused, specifically by fusing the dimension-upgraded feature map, the first upsampled feature map, and the second upsampled feature map to obtain the feature map F of the image. mfm .

[0111] Since low-resolution information, i.e., shallow feature information, has a relatively small effect on the detection result when it is classified, and the amount of low-resolution information data is large, it will greatly increase the amount of calculation. Therefore, considering the amount of calculation and the impact on the detection result, in step S64, the low-resolution information only selects the second-scale feature map and the first up-sampled feature map, and the high-resolution information selects the second up-sampled feature map. The second-scale feature map, the first up-sampled feature map, and the second up-sampled feature map are fused to obtain the feature map of the image, thereby realizing the combination of low-resolution information and high-resolution information, effectively supplementing the global context information between the multi-scale feature maps, and avoiding information loss. It should be noted here that in this embodiment, only three different levels of feature information are selected for fusion, and the high-resolution information only selects the second up-sampled feature map. In actual application, the fourth-scale feature map can be further downsampled according to the amount of calculation or the impact on the detection result to obtain more high-resolution information and select more levels of feature information for fusion.

[0112] Reference Figure 5 The detection network in the object detection method of this embodiment is an improvement based on the existing lightweight detection network, specifically, an improvement based on the existing Light-Head R-CNN network. The Light-Head R-CNN network includes an RPN, a PSROI (position sensitive ROI pooling) layer, and a fully connected layer. The detection network in this embodiment adds a channel self-attention network to the Light-Head R-CNN network.

[0113] Specifically, the RPN includes a cascaded fourth convolutional layer (DW Conv), a fifth convolutional layer (1╳1 Conv), and a candidate region extraction module (ROIs). It should be noted that the Light-Head R-CNN network is used as an example and is not intended to be limiting. A channel self-attention network can also be added to other lightweight detection networks as the detection network in this embodiment.

[0114] Specifically, step S7 includes:

[0115] S71. Pass the feature map of the image through the RPN network to obtain the feature map of the image in the RPN network and the candidate box containing the target to be detected;

[0116] S72. Generate a channel attention feature map based on the feature map of the image in the RPN network;

[0117] S73, fusing the channel attention feature map with the image feature map to obtain a fused feature map;

[0118] S74. Obtain a feature map of the target to be detected based on the candidate frame and the fused feature map;

[0119] S75. Obtain a detection result of the target to be detected according to the feature map of the target to be detected.

[0120] In step S71, the feature map of the image is passed through the fourth convolutional layer (DW Conv) and the fifth convolutional layer (1╳1Conv) in sequence to obtain the feature map of the image in the RPN network. The size of the convolution kernel of the fifth convolutional layer (1╳1 Conv) is 1╳1. Preferably, in order to further reduce the amount of data calculation, this embodiment uses depthwise separable convolution to convolve the feature map of the image through the fourth convolutional layer (DWConv). The feature map of the image in the RPN network is passed through the candidate region extraction module (ROIs) to obtain a candidate box containing the target to be detected.

[0121] In order to solve the problems of weak feature extraction capability of lightweight networks and loss of spatial information around the target detection area, this embodiment adds a channel self-attention network on the basis of the existing Light-Head R-CNN network. The channel self-attention network optimizes the feature distribution of the feature map input to the PSROI (position sensitive ROI pooling) layer, so that the output feature map pays more attention to the area related to detection, thereby improving the accuracy of the detection results.

[0122] Specifically, step S72 includes:

[0123] S721, dividing the feature map of the image in the RPN network into a first sub-feature map and a second sub-feature map, wherein the number of channels of the first sub-feature map and the second sub-feature map is equal;

[0124] S722. Obtain a channel attention weight according to the first sub-feature map;

[0125] S723. Multiply the channel attention weight by the second sub-feature map to obtain a channel attention feature map.

[0126] Reference Figure 6 The channel self-attention network in this embodiment includes a first segmentation module and a channel attention weight acquisition module. The first segmentation module is used to convert the image feature map F in the RPN network into rpn The image is divided into a first sub-feature map F1 and a second sub-feature map F2, wherein the number of channels of the first sub-feature map F1 and the second sub-feature map F2 is equal. The segmentation here refers to directly dividing the channels equally. For example, if the number of channels of the feature map of the image in the RPN network is 8, the data corresponding to the 1st to 4th channels are used as the first sub-feature map F1, and the data corresponding to the 5th to 8th channels are used as the second sub-feature map F2.

[0127] The first sub-feature map F1 is input into the channel attention weight acquisition module, and the channel attention weight K is obtained through the channel attention weight acquisition module, wherein the channel attention weight acquisition module includes the second segmentation module, the group convolution layer (Group Conv), the depth-wise separable convolution layer (DW Conv), the softmax layer, the third pooling layer (Avg pool), and the sixth convolution layer (1╳1 Conv).

[0128] The second segmentation module is used to segment the first sub-feature map F1 into a third sub-feature map F3 and a fourth sub-feature map F4, wherein the number of channels of the third sub-feature map F3 and the fourth sub-feature map F4 are equal. The segmentation here is to directly divide the channels equally. Continuing with the example that the number of channels of the feature map of the image in the RPN network is 8, after segmentation by the first segmentation module, the number of channels of the first sub-feature map F1 is 4, then the data corresponding to the 1st to 2nd channels is used as the third sub-feature map F3, and the data corresponding to the 3rd to 4th channels is used as the fourth sub-feature map F4.

[0129] The third sub-feature map F3 and the fourth sub-feature map F4 are respectively input into the group convolution layer (Group Conv) and the depth-wise separable convolution layer (DW Conv) for convolution processing. The outputs of the group convolution layer (Group Conv) and the depth-wise separable convolution layer (DWConv) are fused and then processed in sequence through the softmax layer, the third pooling layer (Avg pool), and the sixth convolution layer (1╳1Conv) to obtain the channel attention weight K. Among them, the third pooling layer (Avg pool) adopts the mean pooling method for pooling, and the sixth convolution layer (1╳1 Conv) is used for dimensionality increase processing so that the dimension of the channel attention weight K is consistent with the dimension of the second sub-feature map F2.

[0130] After obtaining the channel attention weight K, the channel attention weight K is multiplied by the second sub-feature map F2 to obtain the channel attention feature map.

[0131] In step S73, the channel attention feature map is finally fused with the feature map of the image through the channel self-attention network to obtain a fused feature map. The channel self-attention network in this embodiment combines channel separation with the self-attention mechanism. Through channel separation, the information between each channel can interact with each other, which significantly reduces the complexity of the network structure, thereby greatly reducing the number of parameters. The self-attention mechanism can suppress background features and highlight foreground features. In addition, by combining the channel attention feature map with the feature map of the image, the field of view of each spatial position is expanded, enriching the output function.

[0132] In this embodiment, in steps S74 to S75, the candidate box is mapped to the fused feature map through the PSROI (position sensitive ROI pooling) layer, and the feature map of the target to be detected is extracted from the fused feature map according to the candidate box. The feature map of the target to be detected is passed through the fully connected layer to obtain the detection result of the target to be detected, wherein the category probability is obtained through the fully connected layer and classification is performed according to the category probability, that is, the detection result is classification, and the position offset information is obtained through the fully connected layer and the position of the target is obtained according to the position offset information, that is, the detection result is regression.

[0133] The target detection method in this embodiment is mainly applied to mobile terminal devices. Before deploying the target detection algorithm to the mobile terminal device, it is necessary to first use the training data to train the network model constructed according to the target detection method of this embodiment on the server. After the training is completed, the network model is evaluated using the evaluation data to obtain the network model with the best performance. Finally, the network model with the best performance is deployed to the mobile terminal through the onnx tool to implement the target detection algorithm, detect real data and visualize the detection results.

[0134] The object detection method in this embodiment is verified on the public dataset PASCAL VOC. The experimental results show that the object detection method in this embodiment only requires 528 MFLOPs to achieve an accuracy of 70.6mAP, achieving a good balance between accuracy and model complexity.

[0135] The following describes in detail the verification structure of the object detection method in this embodiment on the public dataset PASCAL VOC.

[0136] The image was scaled to 320×320 as input, and the network model constructed according to the object detection method of this embodiment was trained on an NVIDIA TITAN RTX with 24GB RAM. During the training phase, we used a stochastic gradient optimizer with a learning rate of 0.0001 and a weight decay of 0.001. All datasets were randomly divided into a training set (60%), a validation set (20%), and a test set (20%) so that the data in the training phase, validation phase, and test phase had similar distributions. Millions of floating-point operations (MFLOPs) are defined here to measure the complexity and efficiency of lightweight network models. The performance of the model can be evaluated by the mean average precision (mAP). Different methods were used to detect PASCAL VOC data under the premise of consistent training parameters. The MFLOP and mAP results corresponding to different methods are shown in Table 3.

[0137] Table 3 Comparison of results of different methods on PASCAL VOC dataset

[0138]

[0139] Compared to most recent models based on large-scale object detectors (e.g., YOLOv2, SSD300, SSD321, and R-FCN), the object detection method in this embodiment (our model in Table 2) has a significant advantage in terms of model complexity. Therefore, the object detection method in this embodiment is more suitable for the needs of mobile terminal devices.

[0140] Comparing the target detection method in this embodiment with existing lightweight detection algorithms, it can be seen that the target detection method in this embodiment (our model in Table 2) has much smaller MFLOPs than Tiny-YOLO, D-YOLO, and MobileNet-SSD, and has higher accuracy than Tiny-YOLO, D-YOLO, and MobileNet-SSD. Compared with Pelee, the target detection method in this embodiment (our model in Table 2) can produce similar accuracy with only half the model complexity. It can be seen that the target detection method in this embodiment can achieve a good balance between accuracy and model complexity.

[0141] Reference Figure 7 This embodiment also provides a target detection device corresponding to the above-mentioned target detection method, which includes an acquisition module 1, a dimensionality reduction module 2, a compression module 3, a splicing module 4, a sampling module 5, a feature extraction module 6, and a detection module 7.

[0142] Specifically, the acquisition module 1 is used to acquire an image of the target to be detected, the dimensionality reduction module 2 is used to reduce the dimensionality of the image through a first convolutional layer to obtain a reduced-dimensional image, the compression module 3 is used to compress the reduced-dimensional image through multiple second convolutional layers respectively to obtain multiple first branches, wherein the multiple first branches have the same number of channels, the splicing module 4 is used to extract the first feature maps of the multiple first branches respectively and splice the first feature maps of the multiple first branches to obtain a first spliced ​​feature map, wherein the first feature maps of the multiple first branches increase in depth in sequence, the sampling module 5 is used to downsample the first spliced ​​feature map through a first pooling layer to obtain a sampled feature map, the feature extraction module 6 is used to extract features from the sampled feature map through multiple block modules to obtain a feature map of the image, and the detection module 7 is used to perform detection based on the feature map of the image to obtain a detection result of the target to be detected.

[0143] The splicing module 4 in this embodiment is also used to use the output of the previous first branch as the residual part of the next first branch and fuse the features of the next first branch with the same depth as the residual part and the residual part to obtain the cross-branch feature map after the fusion of the first branch, and perform feature extraction on the cross-branch feature map after the fusion of the first branch through the third convolutional layer to obtain the first feature maps of multiple first branches respectively, and splice the first feature maps of the multiple first branches and fuse them with the image after dimensionality reduction to obtain the first spliced ​​feature map.

[0144] The compression module 3 is also used to compress the sampled feature maps through multiple second convolutional layers to obtain multiple second branches, wherein the multiple second branches have the same number of channels. The splicing module 4 is also used to extract the second feature maps of the multiple second branches respectively and splice the second feature maps of the multiple second branches to obtain a second spliced ​​feature map, wherein the second feature maps of the multiple second branches increase in depth.

[0145] The splicing module 4 in this embodiment is also used to use the output of the previous second branch as the residual part of the next second branch and fuse the features of the next second branch with the same depth as the residual part and the residual part to obtain a plurality of cross-branch feature maps after the fusion of the second branches, and to extract features from the cross-branch feature maps after the fusion of the plurality of second branches through the third convolutional layer to obtain a plurality of second feature maps of the second branches respectively, and to splice the second feature maps of the plurality of second branches and fuse them with the sampled feature map to obtain a second spliced ​​feature map.

[0146] The feature extraction module 6 is also used to extract features from the second spliced ​​feature map through multiple block modules to obtain a feature map of the image. Specifically, the feature extraction module 6 is used to sequentially obtain a first-scale feature map, a second-scale feature map, and a third-scale feature map from the sampled feature map / second spliced ​​feature map through multiple block modules, and to downsample the third-scale feature map to obtain a fourth-scale feature map, and to upsample the third-scale feature map and the fourth-scale feature map respectively to obtain a first upsampled feature map and a second upsampled feature map, and to fuse the second-scale feature map, the first upsampled feature map, and the second upsampled feature map to obtain a feature map of the image.

[0147] The detection module 7 in this embodiment is specifically used to pass the feature map of the image through the RPN network to obtain the feature map of the image in the RPN network and the candidate box containing the target to be detected, and to generate a channel attention feature map based on the feature map of the image in the RPN network, and to fuse the channel attention feature map with the feature map of the image to obtain a fused feature map, and to obtain the feature map of the target to be detected based on the candidate box and the fused feature map, and to obtain the detection result of the target to be detected based on the feature map of the target to be detected.

[0148] The detection module 7 is also used to segment the feature map of the image in the RPN network into a first sub-feature map and a second sub-feature map, wherein the number of channels of the first sub-feature map and the second sub-feature map is equal, and to obtain the channel attention weight according to the first sub-feature map, and to multiply the channel attention weight by the second sub-feature map to obtain the channel attention feature map.

[0149] Reference Figure 8This embodiment provides a device including a memory 100, a processor 200, and a network interface 202. The memory 100 stores a computer program, and the processor 200 executes the computer program to implement the target detection method in this embodiment.

[0150] The memory 100 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0151] The processor 200 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the target detection method in this embodiment can be completed by an integrated logic circuit of hardware in the processor 200 or instructions in the form of software. The processor 200 can also be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc., and can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.

[0152] The memory 100 is used to store a computer program. After receiving an execution instruction, the processor 200 executes the computer program to implement the target detection method in this embodiment.

[0153] This embodiment further provides a computer storage medium, in which a computer program is stored. The processor 200 is configured to read and execute the computer program stored in the computer storage medium 201 to implement the target detection method of this embodiment.

[0154] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer storage medium or transmitted from one computer storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0155] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0156] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0158] The above is only a specific implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A lightweight target detection method, characterized in that: The target detection method comprises: Acquire an image of the target to be detected; Performing dimensionality reduction on the image through a first convolutional layer to obtain a dimensionality-reduced image; Compressing the reduced-dimensional image through multiple second convolutional layers to obtain multiple first branches, each of which has the same number of channels; Extracting first feature maps of the plurality of first branches respectively and splicing the first feature maps of the plurality of first branches to obtain a first spliced ​​feature map, wherein the first feature maps of the plurality of first branches are sequentially increased in depth; Downsample the first concatenated feature map through the first pooling layer to obtain a sampled feature map; Extract features from the sampled feature map through multiple block modules to obtain a feature map of the image; Performing detection based on the feature map of the image to obtain a detection result of the target to be detected; The step of performing detection based on the feature map of the image to obtain a detection result of the target to be detected includes: Passing the feature map of the image through the RPN network to obtain the feature map of the image in the RPN network and the candidate box containing the target to be detected; Generate a channel attention feature map according to the feature map of the image in the RPN network; Fusing the channel attention feature map with the feature map of the image to obtain a fused feature map; Obtaining a feature map of the target to be detected according to the candidate frame and the fused feature map; A detection result of the target to be detected is obtained according to the feature map of the target to be detected.

2. The target detection method according to claim 1, wherein: Before extracting features from the sampled feature map through a plurality of block modules to obtain the feature map of the image, the target detection method further includes: Compressing the sampled feature maps through multiple second convolutional layers to obtain multiple second branches, each of which has the same number of channels; Extracting the second feature maps of the plurality of second branches respectively and splicing the second feature maps of the plurality of second branches to obtain a second spliced ​​feature map, wherein the second feature maps of the plurality of second branches are sequentially increased in depth; Accordingly, the sampled feature map is subjected to feature extraction through multiple block modules to obtain the feature map of the image, including: The second spliced ​​feature map is subjected to feature extraction through multiple block modules to obtain a feature map of the image.

3. The target detection method according to claim 2, wherein: For the first branch / second branch with the smallest depth, compressing the reduced-dimensional image / sampled feature map through a second convolutional layer includes: The reduced-dimensional image / sampled feature map is sequentially passed through the second pooling layer and the second convolutional layer for pooling and compression respectively.

4. The target detection method according to claim 3, wherein: Extracting the first feature maps of the multiple first branches / the second feature maps of the second branches respectively and splicing the multiple first feature maps of the first branches / the second feature maps of the second branches to obtain a first spliced ​​feature map / a second spliced ​​feature map, including: The output of the previous first branch / second branch is used as the residual part of the next first branch / second branch, and the features of the next first branch / second branch with the same depth as the residual part are fused with the residual part to obtain a cross-branch feature map after the fusion of multiple first branches / second branches; The cross-branch feature maps after the fusion of multiple first branches / second branches are subjected to feature extraction by the third convolutional layer to obtain multiple first feature maps of the first branches / second feature maps of the second branches respectively; The first feature maps of the multiple first branches are spliced ​​and fused with the image after dimensionality reduction / the second feature maps of the multiple second branches are spliced ​​and fused with the sampled feature map to obtain a first spliced ​​feature map / a second spliced ​​feature map.

5. The target detection method according to claim 1 or 2, characterized in that: The sampled feature map / the second spliced ​​feature map is subjected to feature extraction through multiple block modules to obtain a feature map of the image, including: Passing the sampled feature map / the second spliced ​​feature map through multiple block modules in sequence to obtain a first-scale feature map, a second-scale feature map, and a third-scale feature map; Downsampling the third scale feature map to obtain a fourth scale feature map; Upsampling the third scale feature map and the fourth scale feature map respectively to obtain a first upsampled feature map and a second upsampled feature map; The second-scale feature map, the first up-sampled feature map, and the second up-sampled feature map are fused to obtain a feature map of the image.

6. The target detection method according to claim 1, wherein: Generating a channel attention feature map according to the feature map of the image in the RPN network, including: Splitting the feature map of the image in the RPN network into a first sub-feature map and a second sub-feature map, where the number of channels of the first sub-feature map and the second sub-feature map are equal; Obtaining a channel attention weight according to the first sub-feature map; Multiplying the channel attention weight by the second sub-feature map obtains a channel attention feature map.

7. A lightweight target detection device, characterized in that: The target detection device comprises: An acquisition module is used to acquire an image of a target to be detected; A dimensionality reduction module, configured to reduce the dimensionality of the image through a first convolutional layer to obtain a reduced-dimensional image; A compression module, configured to compress the image after dimensionality reduction through a plurality of second convolutional layers to obtain a plurality of first branches, wherein the plurality of first branches have the same number of channels; a splicing module, configured to respectively extract the first feature maps of the plurality of first branches and splice the first feature maps of the plurality of first branches to obtain a first spliced ​​feature map, wherein the first feature maps of the plurality of first branches are sequentially increased in depth; A sampling module, configured to downsample the first concatenated feature map through the first pooling layer to obtain a sampled feature map; A feature extraction module is used to extract features from the sampled feature map through multiple block modules to obtain a feature map of the image; A detection module, configured to perform detection based on the feature map of the image and obtain a detection result of the target to be detected; Among them, the detection module is further used to: pass the feature map of the image through the RPN network to obtain the feature map of the image in the RPN network and the candidate box containing the target to be detected; generate a channel attention feature map based on the feature map of the image in the RPN network; fuse the channel attention feature map with the feature map of the image to obtain a fused feature map; obtain the feature map of the target to be detected based on the candidate box and the fused feature map; obtain the detection result of the target to be detected based on the feature map of the target to be detected.

8. A device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the computer program to implement the target detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the target detection method according to any one of claims 1 to 6 is implemented.

Images