Lightweight small target detection method inspired by ventral visual pathway

Abstract

The invention aims to provide a lightweight small target detection method inspired by a ventral visual pathway, and the method comprises the following steps: A, constructing a neural network which comprises a VCB network, a VDE denoising module, a PAN module, and an ITT Head module; b, inputting the original image into a VCB network to obtain VCB output features, and respectively inputting the VCB output features into a VDE denoising module and a PAN module; c, a VDE denoising module performs background interference suppression on the VCB output features, obtains VDE output features after enhancing detail features, and inputs the VDE output features into a PAN module; and D, performing multi-scale feature fusion on the VCB output features and the VDE output features by the PAN module to obtain specially-integrated layers with different resolutions, and respectively inputting the specially-integrated layers into the two ITT Head modules for detection to obtain a final result. According to the method, the identification capability of similar small targets can be enhanced, and the method has excellent performance and robustness in a small target detection task.

Description

Technical Field

[0001] The present invention relates to the field of computer image processing, and in particular to a lightweight small target detection method inspired by a ventral visual pathway. Background Art

[0002] Object detection is a basic task in computer vision, which involves the recognition and location of objects in images. Small object detection is a subfield of the field of object detection, focusing on identifying and locating objects with small size, small area, or occupying a small proportion in the image in images or videos. Based on different target positioning methods, object detection methods can be divided into two categories: single-stage detectors and two-stage detectors. Most of the existing small object detection methods use higher input image resolution or high-resolution feature maps, or use larger and more complex two-stage model detection, and have achieved good performance in adapting to the special needs of the UAV aerial photography small target dataset (VisDrone2019) and the aerial image dataset (AI-TODv2).

[0003] Although the current small target detection algorithms have achieved good performance, this has sacrificed the performance of other aspects of the model (such as model size, computational complexity, etc.). Although these small target detection algorithms have made significant progress, compared with biological vision systems, deep learning models have complexity and black box properties, which increases the unexplainability. Summary of the invention

[0004] The present invention aims to provide a lightweight small target detection method inspired by the ventral visual pathway, which can enhance the recognition ability of similar small targets by simulating the visual mechanism of biological visual cortex, and has excellent performance and robustness in small target detection tasks.

[0005] The technical solution of the present invention is as follows:

[0006] The lightweight small target detection method inspired by the ventral visual pathway comprises the following steps:

[0007] A. Constructing a neural network, wherein the neural network includes a VCB network, a VDE denoising module, a PAN module, and an ITT Head module;

[0008] B. The original image is input into the VCB network, and is first processed by the color antagonism module CAM and the brightness and orientation contrast module BOCM to obtain the color antagonism result and the brightness and orientation adjustment result. Then, the two processing results are concatenated by the Concat operation and then processed by the cross-layer information fusion module CIFM to obtain the VCB output features, which are input into the VDE denoising module and the PAN module respectively.

[0009] C. The VDE denoising module suppresses background interference on the VCB output features and strengthens the detail features to obtain the VDE output features, which are then input into the PAN module.

[0010] D. The PAN module performs multi-scale feature fusion on the VCB output features and the VDE output features to obtain feature layers of different resolutions, which are input into the two ITT Head modules for detection respectively to obtain a series of detection boxes and category data, which is the final result.

[0011] The processing process in the color antagonism module CAM is as follows:

[0012] The input result is split into three channel images: blue, green, and red;

[0013] The blue channel image is convolved to obtain the blue channel result;

[0014] The green channel image is convolved to obtain the green channel result;

[0015] The red channel image is convolved to obtain the red channel result;

[0016] The green channel result and the red channel result are concatenated by Concat operation to obtain a yellow concatenated result, and the yellow concatenated result is concatenated with the blue channel result to obtain a first concatenated result;

[0017] The green channel result and the red channel result are subjected to an anti-antagonism operation to obtain a second anti-antagonism result;

[0018] The first anti-antagonism result and the second anti-antagonism result are added and fused to obtain an output result.

[0019] The process of performing the anti-antagonistic operation on the yellow splicing result and the blue channel result is as follows:

[0020] The yellow splicing result is processed by 1*1 convolution, and the blue channel result is processed by 3*3 convolution. The two convolution results are added and fused to obtain the first anti-antagonism result;

[0021] The process of performing the anti-antagonistic operation on the green channel result and the red channel result is as follows:

[0022] The red channel result is processed by 1*1 convolution, and the green channel result is processed by 3*3 convolution. The two convolution results are added and fused to obtain the second antagonism result.

[0023] The processing process in the brightness azimuth contrast model BOCM is as follows: the input result is divided into two paths, the first path is multiplied by the set constant α, 1<α<2, to obtain the dark processing result; the second path is multiplied by the set constant β, 0<β<1, to obtain the bright processing result;

[0024] The dark processing result is sequentially subjected to convolution and azimuth convolution to obtain a first processing result;

[0025] The dark processing result and the bright processing result are added and fused to obtain a second processing result;

[0026] The bright processing result is sequentially subjected to convolution and azimuth convolution to obtain a third processing result;

[0027] The first processing result, the second processing result, and the third processing result are added and fused to obtain an output result.

[0028] The processing process in the azimuth convolution is as follows: the input result is divided into four paths, which are respectively passed through the azimuth convolution AZConv1 module, the azimuth convolution AZConv2 module, the azimuth convolution AZConv3 module, and the azimuth convolution AZConv4 module; in the azimuth convolution AZConv1 module, the azimuth convolution AZConv2 module, the azimuth convolution AZConv3 module, and the azimuth convolution AZConv4 module, after masking part of the convolution kernel value by Mask, it is multiplied with the input result to obtain four azimuth convolution results;

[0029] After the processing results of the azimuth convolution AZConv2 module and the processing results of the azimuth convolution AZConv3 module are added and fused, the obtained fusion results are added and fused with the processing results of the azimuth convolution AZConv1 module and the processing results of the azimuth convolution AZConv4 module to obtain the output result;

[0030] Among them, the Mask modes in the azimuth convolution AZConv1 module, azimuth convolution AZConv2 module, azimuth convolution AZConv3 module, and azimuth convolution AZConv4 module are 0°, 45°, 90°, and 135°, respectively.

[0031] The processing process in the VDE denoising module is as follows:

[0032] The input result is divided into two paths. The first path is processed by 1*1 convolution to obtain the first path result. The second path is processed by the spatial channel attention module CBAM to obtain the CBAM processing result. The CBAM processing result is divided into two parts through the split operation. The first part is processed by 3*3 convolution to obtain the first 3*3 convolution processing result.

[0033] The second part is divided into two parts after the split operation to obtain two second split results; one of the second split results is divided into two paths, one path is processed by 1*1 convolution, and the other path is processed by 3*3 convolution. After the two convolution results are added and fused, the first fusion result is obtained; the second split result is divided into two paths, one path is processed by 3*3 convolution, and the other path is processed by 5*5 convolution. After the two convolution results are added and fused, the second fusion result is obtained; the first fusion result and the second fusion result are spliced ​​by the Concat operation, and then processed by 3*3 convolution, and then spliced ​​with the first 3*3 convolution processing result by the Concat operation. The spliced ​​result and the first path result are input into the reverse attention module for processing, and the result obtained is the output result.

[0034] The processing process in the ITT Head module is as follows:

[0035] After the input result is convolved through structural reparameterization, the Repconv result is divided into two paths.

[0036] The first path is processed by the spatial channel attention module CBAM, convolution layer, convolution layer, and ordinary convolution layer without pooling and activation layer in sequence to obtain the first processing result, which is the Bbox detection frame;

[0037] The first processing result is processed by a common convolution layer without pooling and activation layer and a convolution layer in turn to obtain a backup result. The second processing result is obtained by adding and fusion of the result obtained by the convolution layer, convolution layer, and common convolution layer without pooling and activation layer, which is the Class category.

[0038] The method of the present invention solves the problem of small target detection by constructing a lightweight, simple and effective framework. We obtain the model design ideas from the perspective of the V1 / V2-V4-IT ventral pathway to explain the reason for the improvement of network performance; secondly, by designing modules inspired by biological vision mechanisms such as the VCB network, VDE denoising module and ITT Head module, hierarchical connections in the ventral pathway are realized. Experiments have shown that the method of the present invention promotes the model's ability to detect small target objects.

[0039] The method of the present invention uses insights from biological vision mechanisms to improve small object detection. This improved design enables the model to have excellent performance and robustness in small target detection tasks, thereby improving the detection performance of small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1It is a schematic diagram of the structure of the neural network of Example 1 of the present invention; (A) is a network structure framework diagram, the solid line represents the actual process of image features, and the dotted line represents the pointing detection result diagram. BOCM, CAM, and CIFM are respectively the brightness and azimuth contrast module, the color antagonism module, and the cross-layer information fusion module. The white dotted box in the detection result diagram represents the area with similar small targets, which are enlarged and displayed in A(1), A(2), A(3), and A(4) respectively. A(1) shows the detection effect of Van and Car; A(2) shows the detection effect of Tricycle; A(3) shows the detection effect of Awning-Tricycle, bicycle, and motor; A(4) shows the detection effect of pedestrian, people, and motor. (B) is the biological visual information transmission pathway-the ventral pathway.

[0041] Figure 2 This is a schematic diagram of the structure of the color antagonism module CAM of Example 1;

[0042] Figure 3 The schematic diagram of the structure of the brightness and orientation contrast model BOCM of Example 1; (A) shows the organization of the color antagonistic receptive field of biological vision V1 / V2, brightness and orientation contrast, and cross-layer information transmission within V2. (B) shows the network structure diagram of VCBNet.

[0043] Figure 4 Schematic diagram of the structure of the azimuthal convolution of Example 1;

[0044] Figure 5 Schematic diagram of the structure of the VDE denoising module of Example 1;

[0045] Figure 6 It is a structural schematic diagram of the ITT Head module of Example 1;

[0046] Figure 7 Comparison charts showing the results of Ground Truth, YOLOv8m-P2, and Example 1 on the Visdrone2019 test set. DETAILED DESCRIPTION

[0047] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0048] Example 1

[0049] The lightweight small target detection method inspired by the ventral visual pathway comprises the following steps:

[0050] A. Construct a neural network, such as Figure 1 As shown, the neural network includes a VCB network, a VDE denoising module, a PAN module, and an ITT Head module;

[0051] B. The original image is input into the VCB network, and is first processed by the color antagonism module CAM and the brightness and orientation contrast module BOCM to obtain the color antagonism result and the brightness and orientation adjustment result. Then, the two processing results are concatenated by the Concat operation and then processed by the cross-layer information fusion module CIFM to obtain the VCB output features, which are input into the VDE denoising module and the PAN module respectively.

[0052] like Figure 2 As shown, the processing process in the color antagonism module CAM is as follows:

[0053] The input result is split into three channel images: blue, green, and red;

[0054] The blue channel image is convolved to obtain the blue channel result;

[0055] The green channel image is convolved to obtain the green channel result;

[0056] The red channel image is convolved to obtain the red channel result;

[0057] The green channel result and the red channel result are concatenated by Concat operation to obtain a yellow concatenated result, and the yellow concatenated result is concatenated with the blue channel result to obtain a first concatenated result;

[0058] The green channel result and the red channel result are subjected to an anti-antagonism operation to obtain a second anti-antagonism result;

[0059] The first anti-antagonism result and the second anti-antagonism result are added and fused to obtain an output result.

[0060] Antagonistic operation by Figure 2 The CATConv module is used, wherein the process of performing the anti-antagonistic operation on the yellow splicing result and the blue channel result is as follows:

[0061] The yellow splicing result is processed by 1*1 convolution, and the blue channel result is processed by 3*3 convolution. The two convolution results are added and fused to obtain the first anti-antagonism result;

[0062] The process of performing the anti-antagonistic operation on the green channel result and the red channel result is as follows:

[0063] The red channel result is processed by 1*1 convolution, and the green channel result is processed by 3*3 convolution. The two convolution results are added and fused to obtain the second antagonism result.

[0064] like Figure 3As shown, the processing process in the brightness azimuth contrast model BOCM is as follows: the input result is divided into two paths, the first path is multiplied by the set constant α, 1<α<2, to obtain the dark processing result; the second path is multiplied by the set constant β, 0<β<1, to obtain the bright processing result;

[0065] The dark processing result is sequentially subjected to convolution and azimuth convolution to obtain a first processing result;

[0066] The dark processing result and the bright processing result are added and fused to obtain a second processing result;

[0067] The bright processing result is sequentially subjected to convolution and azimuth convolution to obtain a third processing result;

[0068] The first processing result, the second processing result, and the third processing result are added and fused to obtain an output result.

[0069] like Figure 4 As shown, the processing process in the azimuth convolution is as follows: the input result is divided into four paths, which are respectively passed through the azimuth convolution AZConv1 module, the azimuth convolution AZConv2 module, the azimuth convolution AZConv3 module, and the azimuth convolution AZConv4 module; in the azimuth convolution AZConv1 module, the azimuth convolution AZConv2 module, the azimuth convolution AZConv3 module, and the azimuth convolution AZConv4 module, after masking part of the convolution kernel value by Mask, multiply it with the input result to obtain four azimuth convolution results;

[0070] After the processing results of the azimuth convolution AZConv2 module and the processing results of the azimuth convolution AZConv3 module are added and fused, the obtained fusion results are added and fused with the processing results of the azimuth convolution AZConv1 module and the processing results of the azimuth convolution AZConv4 module to obtain the output result;

[0071] Among them, the Mask modes in the azimuth convolution AZConv1 module, azimuth convolution AZConv2 module, azimuth convolution AZConv3 module, and azimuth convolution AZConv4 module are 0°, 45°, 90°, and 135°, respectively.

[0072] C. The VDE denoising module suppresses background interference on the VCB output features and strengthens the detail features to obtain the VDE output features, which are then input into the PAN module.

[0073] like Figure 5 As shown, the processing process in the VDE denoising module is as follows:

[0074] The input result is divided into two paths. The first path is processed by 1*1 convolution to obtain the first path result. The second path is processed by the spatial channel attention module CBAM to obtain the CBAM processing result. The CBAM processing result is divided into two parts through the split operation. The first part is processed by 3*3 convolution to obtain the first 3*3 convolution processing result.

[0075] The second part is divided into two parts after the split operation to obtain two second split results; one of the second split results is divided into two paths, one path is processed by 1*1 convolution, and the other path is processed by 3*3 convolution. After the two convolution results are added and fused, the first fusion result is obtained; the second split result is divided into two paths, one path is processed by 3*3 convolution, and the other path is processed by 5*5 convolution. After the two convolution results are added and fused, the second fusion result is obtained; the first fusion result and the second fusion result are spliced ​​by the Concat operation, and then processed by 3*3 convolution, and then spliced ​​with the first 3*3 convolution processing result by the Concat operation. The spliced ​​result and the first path result are input into the reverse attention module for processing, and the result obtained is the output result.

[0076] D. The PAN module performs multi-scale feature fusion on the VCB output features and the VDE output features to obtain feature layers of different resolutions, which are input into the two ITT Head modules for detection respectively to obtain a series of detection boxes and category data, which is the final result.

[0077] like Figure 6 As shown, the processing process in the ITT Head module is as follows:

[0078] After the input result is convolved through structural reparameterization, the Repconv result is divided into two paths.

[0079] The first path is processed by the spatial channel attention module CBAM, convolution layer, convolution layer, and ordinary convolution layer without pooling and activation layer in sequence to obtain the first processing result, which is the Bbox detection frame;

[0080] The first processing result is processed by a common convolution layer without pooling and activation layer and a convolution layer in turn to obtain a backup result. The second processing result is obtained by adding and fusion of the result obtained by the convolution layer, convolution layer, and common convolution layer without pooling and activation layer, which is the Class category.

[0081] Example 2

[0082] For quantitative performance evaluation of small target datasets, we adopt the same evaluation criteria as those in references 1-5. The specific evaluation is shown in formula (1).

[0083]

[0084] Where class_num represents the number of categories, AP represents the accuracy, and its calculation formula (2) is as follows:

[0085]

[0086] Where Precision and Recall represent the precision and recall respectively, as shown in the following formulas (3) and (4):

[0087]

[0088] Among them, TP represents the number of correctly predicted prediction boxes [IoU>threshold], FP represents the number of redundant prediction boxes detected as the same GT [IoU<threshold], and FN represents the number of prediction boxes that have not detected GT.

[0089] Document 1: N.Wang et al., "NAS-FCOS: Fast neural architecture search for object detection," in proceedings of the IEEE / CVF conference on computervision and pattern recognition, 2020, pp.11943-11951.

[0090] Document 2: C.Feng, Y.Zhong, Y.Gao, MRScott, and W.Huang, "Tood: Task-alignedone-stage object detection," in 2021 IEEE / CVF International Conference on Computer Vision (ICCV), 2021, pp.3490-3499: IEEE Computer Society.

[0091] Document 3: D.Ouyang et al., "Efficient multi-scale attention module with cross-spatial learning," in ICASSP 2023-2023IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp.1-5: IEEE.

[0092] Document 4: Z.Chen, H.Ji, Y.Zhang, Z.Zhu, and Y.Li, "High-Resolution FeaturePyramid Network for Small Object Detection On Drone View," IEEE Transactionson Circuits and Systems for Video Technology, 2023.

[0093] Document 5: Y.Li, Y.Wang, Z.Ma,

[0094] The parameters used in references 1-5 are the same as those in the original text, and are guaranteed to be the optimal parameters of the model.

[0095] Table 1 summarizes the experimental data of References 1-5 and Example 1 on the drone aerial photography dataset (VisDrone2019). The current mAP comparison results of this method and the VisDrone2019 validation set are shown, indicating that this method has significant advantages over other mAP indicator methods.

[0096] Table 1 Comparison of mAP results of the validation set of the drone aerial photography dataset VisDrone2019.

[0097]

[0098] Table 2 verifies the effectiveness of the bionic model design. An ablation experiment was performed on the UAV aerial photography dataset VisDrone2019. The specific experimental method is to replace and add the VCBNet, VDE, and ITT modules representing the V2, V4, and IT visual cortices in Example 1, and replace or add some visual features of the visual cortex to the baseline. It can be seen from Table 2 that the application of the corresponding modules successively improves the small target detection performance of the model.

[0099] Table 2 Qualitative analysis of Example 1 in VisDrone2019.

[0100]

[0101] Example 3

[0102] Figure 7 The results of Ground Truth, YOLOv8m-P2, and Example 1 are shown in Figure 1. Ground Truth represents the real situation of the dataset, and YOLOv8m-P2 is a small target model proposed by YOLOv8 for small target detection. Due to the dense distribution and mutual occlusion of small targets, there are a large number of undetected or detected similar categories in YOLOv8m-P2. The detection results of Example 1 of this model are excellent, with a small gap compared with Ground Truth.

Claims

1. A lightweight small target detection method inspired by the ventral visual pathway, characterized in that: The following steps are involved: A. Constructing a neural network, wherein the neural network includes a VCB network, a VDE denoising module, a PAN module, and an ITT Head module; B. The original image is input into the VCB network, and is first processed by the color antagonism module CAM and the brightness and orientation contrast module BOCM to obtain the color antagonism result and the brightness and orientation adjustment result. Then, the two processing results are concatenated by the Concat function and then processed by the cross-layer information fusion module CIFM to obtain the VCB output features, which are input into the VDE denoising module and the PAN module respectively. C. The VDE denoising module suppresses background interference on the VCB output features and strengthens the detail features to obtain the VDE output features, which are then input into the PAN module. D. The PAN module performs multi-scale feature fusion on the VCB output features and the VDE output features to obtain feature layers of different resolutions, which are input into the two ITT Head modules for detection respectively to obtain a series of detection boxes and category data, which is the final result.

2. The lightweight small target detection method inspired by the ventral visual pathway as claimed in claim 1, characterized in that: The processing process in the color antagonism module CAM is as follows: The input result is split into three channel images: blue, green, and red; The blue channel image is convolved to obtain the blue channel result; The green channel image is convolved to obtain the green channel result; The red channel image is convolved to obtain the red channel result; The green channel result and the red channel result are concatenated by the Concat function to obtain a yellow concatenated result. The yellow concatenated result is concatenated with the blue channel result to obtain a first concatenated result. The green channel result and the red channel result are subjected to an anti-antagonism operation to obtain a second anti-antagonism result; The first anti-antagonism result and the second anti-antagonism result are added and fused to obtain an output result.

3. The lightweight small target detection method inspired by the ventral visual pathway as claimed in claim 2, characterized in that: The process of performing the anti-antagonistic operation on the yellow splicing result and the blue channel result is as follows: The yellow splicing result is processed by 1*1 convolution, and the blue channel result is processed by 3*3 convolution. The two convolution results are added and fused to obtain the first anti-antagonism result; The process of performing the anti-antagonistic operation on the green channel result and the red channel result is as follows: The red channel result is processed by 1*1 convolution, and the green channel result is processed by 3*3 convolution. The two convolution results are added and fused to obtain the second antagonism result.

4. The lightweight small target detection method inspired by the ventral visual pathway as claimed in claim 1, characterized in that: The processing process in the brightness azimuth contrast model BOCM is as follows: the input result is divided into two paths, the first path is multiplied by the set constant α, 1<α<2, to obtain the dark processing result; the second path is multiplied by the set constant β, 0<β<1, to obtain the bright processing result; The dark processing result is sequentially subjected to convolution and azimuth convolution to obtain a first processing result; The dark processing result and the bright processing result are added and fused to obtain a second processing result; The bright processing result is sequentially subjected to convolution and azimuth convolution to obtain a third processing result; The first processing result, the second processing result, and the third processing result are added and fused to obtain an output result.

5. The lightweight small target detection method inspired by the ventral visual pathway as claimed in claim 4, characterized in that: The processing process in the azimuth convolution is as follows: the input result is divided into four paths, which are respectively passed through the azimuth convolution AZConv1 module, the azimuth convolution AZConv2 module, the azimuth convolution AZConv3 module, and the azimuth convolution AZConv4 module; in the azimuth convolution AZConv1 module, the azimuth convolution AZConv2 module, the azimuth convolution AZConv3 module, and the azimuth convolution AZConv4 module, after masking part of the convolution kernel value by Mask, it is multiplied with the input result to obtain four azimuth convolution results; After the processing results of the azimuth convolution AZConv2 module and the processing results of the azimuth convolution AZConv3 module are added and fused, the obtained fusion results are added and fused with the processing results of the azimuth convolution AZConv1 module and the processing results of the azimuth convolution AZConv4 module to obtain the output result; Among them, the Mask modes in the azimuth convolution AZConv1 module, azimuth convolution AZConv2 module, azimuth convolution AZConv3 module, and azimuth convolution AZConv4 module are 0°, 45°, 90°, and 135°, respectively.

6. The lightweight small target detection method inspired by the ventral visual pathway as claimed in claim 1, characterized in that: The processing process in the VDE denoising module is as follows: The input result is divided into two paths. The first path is processed by 1*1 convolution to obtain the first path result. The second path is processed by the spatial channel attention module CBAM to obtain the CBAM processing result. The CBAM processing result is divided into two parts through the split operation. The first part is processed by 3*3 convolution to obtain the first 3*3 convolution processing result. The second part is divided into two parts after the split operation to obtain two second split results; one of the second split results is divided into two paths, one path is processed by 1*1 convolution, and the other path is processed by 3*3 convolution. After the two convolution results are added and fused, the first fusion result is obtained; the second split result is divided into two paths, one path is processed by 3*3 convolution, and the other path is processed by 5*5 convolution. After the two convolution results are added and fused, the second fusion result is obtained; the first fusion result and the second fusion result are spliced ​​by the Concat function, and then processed by 3*3 convolution, and then spliced ​​with the first 3*3 convolution processing result Concat function, the spliced ​​result and the first path result are input into the reverse attention module for processing, and the result obtained is the output result.

7. The lightweight small target detection method inspired by the ventral visual pathway as claimed in claim 1, characterized in that: The processing process in the ITT Head module is as follows: After the input result is convolved through structural reparameterization, the Repconv result is divided into two paths. The first path is processed by the spatial channel attention module CBAM, convolution layer, convolution layer, and ordinary convolution layer without pooling and activation layer in sequence to obtain the first processing result, which is the Bbox detection frame; The first processing result is processed by a common convolution layer and a convolution layer without pooling and activation layers to obtain a backup result. The second path is processed by the convolution layer, convolution layer, and ordinary convolution layer without pooling and activation layer in sequence. The obtained result is added and fused with the backup result to obtain the second processing result, which is the Class category.

Images