A lightweight small target detection method inspired by the ventral visual pathway

Abstract

The present application aims to provide a lightweight small target detection method inspired by the ventral visual pathway, comprising the following steps: A, constructing a neural network, including a VCB network, a VDE denoising module, a PAN module, and an ITT Head module; B, inputting an original image into the VCB network to obtain VCB output features, which are respectively input into the VDE denoising module and the PAN module; C, the VDE denoising module suppresses background interference on the VCB output features, strengthens the detail features, and obtains VDE output features which are input into the PAN module; D, the PAN module performs multi-scale feature fusion on the VCB output features and the VDE output features to obtain a feature integration layer with different resolutions, which are respectively input into two ITT Head modules for detection to obtain the final result. The present application can enhance the recognition ability of similar small targets and has excellent performance and robustness in small target detection tasks.

Description

Technical Field

[0001] The present invention relates to the field of computer image processing, and in particular to a lightweight small target detection method inspired by the ventral visual pathway. Background Art

[0002] Object detection is a fundamental task in computer vision, involving the recognition and localization of objects in images. Small object detection is a subfield of the field of object detection, focusing on identifying and localizing objects in images or videos that are small in size, small in area, or occupy a small proportion of the image. Based on different target localization methods, object detection methods can be divided into two categories: single-stage detectors and two-stage detectors. Most existing small object detection methods use higher input image resolution or high-resolution feature maps, or use larger and more complex two-stage models for detection, achieving good performance in adapting to the special needs of the UAV Aerial Small Object Dataset (VisDrone2019) and the Aerial Image Dataset (AI-TODv2).

[0003] Although current small object detection algorithms have achieved good performance, this has sacrificed other aspects of the model's performance (such as model size, computational complexity, etc.). Although these small object detection algorithms have made significant progress, compared with biological vision systems, deep learning models have the complexity and black-box nature, which increases the unexplainability. Summary of the Invention

[0004] This paper aims to provide a lightweight small target detection method inspired by the ventral visual pathway. By simulating the visual mechanism of the biological visual cortex, this method can enhance the recognition ability of similar small targets and achieve excellent performance and robustness in small target detection tasks.

[0005] The technical solutions of the present invention are as follows:

[0006] The lightweight small target detection method inspired by the ventral visual pathway includes the following steps:

[0007] A. Constructing a neural network, wherein the neural network includes a VCB network, a VDE denoising module, a PAN module, and an ITT Head module;

[0008] B. The original image is input into the VCB network and first processed by the color antagonism module CAM and the brightness and orientation contrast module BOCM to obtain the color antagonism result and the brightness and orientation adjustment result. The two processed results are then concatenated by the Concat operation and processed by the cross-layer information fusion module CIFM to obtain the VCB output features, which are input into the VDE denoising module and the PAN module respectively.

[0009] C. The VDE denoising module suppresses background interference on the VCB output features and enhances the detail features to obtain the VDE output features, which are then input into the PAN module.

[0010] D. The PAN module performs multi-scale feature fusion on the VCB output features and the VDE output features to obtain feature layers of different resolutions, which are input into the two ITT Head modules for detection respectively, and a series of detection boxes and category data are obtained, which is the final result.

[0011] The processing process in the color antagonism module CAM is as follows:

[0012] The input result is split into three channel images: blue, green, and red;

[0013] The blue channel image is convolved to obtain the blue channel result;

[0014] The green channel image is convolved to obtain the green channel result;

[0015] The red channel image is convolved to obtain the red channel result;

[0016] After the green channel result and the red channel result are concatenated through the Concat operation, a yellow splicing result is obtained. The yellow splicing result is then subjected to an anti-antagonism operation on the blue channel result to obtain a first anti-antagonism result.

[0017] The green channel result and the red channel result are subjected to an anti-antagonism operation to obtain a second anti-antagonism result;

[0018] The first anti-antagonism result and the second anti-antagonism result are added and fused to obtain an output result.

[0019] The process of performing the anti-antagonistic operation on the yellow splicing result and the blue channel result is as follows:

[0020] The yellow splicing result is processed by 1*1 convolution, and the blue channel result is processed by 3*3 convolution. The two convolution results are added and fused to obtain the first anti-antagonism result;

[0021] The process of performing the anti-antagonism operation on the green channel result and the red channel result is as follows:

[0022] The red channel result is processed by 1*1 convolution, and the green channel result is processed by 3*3 convolution. The two convolution results are added and fused to obtain the second anti-antagonism result.

[0023] The processing process in the BOCM is as follows: the input result is divided into two paths, the first path is multiplied by a set constant α, 1<α<2, to obtain a dark processing result; the second path is multiplied by a set constant β, 0<β<1, to obtain a bright processing result;

[0024] The dark processing result is sequentially subjected to convolution and azimuth convolution to obtain a first processing result;

[0025] The dark processing result and the bright processing result are added and fused to obtain a second processing result;

[0026] The bright processing result is sequentially subjected to convolution and azimuth convolution to obtain the third processing result;

[0027] The first processing result, the second processing result, and the third processing result are added and fused to obtain the output result.

[0028] The processing process in the azimuth convolution is as follows: the input result is divided into four paths, which pass through the azimuth convolution AZConv1 module, the azimuth convolution AZConv2 module, the azimuth convolution AZConv3 module, and the azimuth convolution AZConv4 module respectively; in the azimuth convolution AZConv1 module, the azimuth convolution AZConv2 module, the azimuth convolution AZConv3 module, and the azimuth convolution AZConv4 module, the convolution kernel part is masked by the mask, and then multiplied with the input result to obtain four azimuth convolution results;

[0029] After the processing results of the azimuth convolution module AZConv2 and the azimuth convolution module AZConv3 are added and fused, the obtained fusion results are added and fused with the processing results of the azimuth convolution module AZConv1 and the azimuth convolution module AZConv4 to obtain the output result;

[0030] Among them, the Mask modes in the azimuth convolution AZConv1 module, azimuth convolution AZConv2 module, azimuth convolution AZConv3 module, and azimuth convolution AZConv4 module are 0°, 45°, 90°, and 135°, respectively.

[0031] The processing process in the VDE denoising module is as follows:

[0032] The input result is divided into two paths. The first path is processed by 1*1 convolution to obtain the first path result; the second path is processed by the spatial channel attention module CBAM to obtain the CBAM processing result. The CBAM processing result is divided into two parts through the split operation. The first part is processed by 3*3 convolution to obtain the first 3*3 convolution processing result;

[0033] The second part is divided into two parts after the split operation to obtain two second split results; one of the second split results is divided into two paths, one path is processed by 1*1 convolution, and the other path is processed by 3*3 convolution. After the two convolution results are added and fused, the first fusion result is obtained; the second split result is divided into two paths, one path is processed by 3*3 convolution, and the other path is processed by 5*5 convolution. After the two convolution results are added and fused, the second fusion result is obtained; the first fusion result and the second fusion result are spliced ​​by the Concat operation, and then processed by 3*3 convolution, and then spliced ​​with the first 3*3 convolution processing result by the Concat operation. The spliced ​​result and the first path result are input into the reverse attention module for processing, and the result obtained is the output result.

[0034] The processing process in the ITT Head module is as follows:

[0035] After the input result is convolved by structural reparameterization, the Repconv result is divided into two paths.

[0036] The first path is processed in sequence by the spatial channel attention module CBAM, convolution layer, convolution layer, and ordinary convolution layer without pooling and activation layer to obtain the first-path processing result, which is the Bbox detection frame;

[0037] The first-path processing result is processed in sequence by a common convolution layer without pooling and activation layer, and then a convolution layer to obtain a backup result. The second-path processing result is obtained by adding and fusion of the result obtained by the convolution layer, convolution layer, and common convolution layer without pooling and activation layer to obtain the second-path processing result, which is the Class category.

[0038] This method addresses the problem of small object detection by constructing a lightweight, simple, and effective framework. We first derive model design insights from the V1 / V2-V4-IT ventral pathway to explain the improved network performance. Secondly, we implement hierarchical connections within the ventral pathway by designing modules inspired by biological vision mechanisms, including the VCB network, VDE denoising module, and ITT Head module. Experiments demonstrate that this method improves the model's ability to detect small objects.

[0039] The method of the present invention uses insights from biological vision mechanisms to improve small object detection. This improved design enables the model to have excellent performance and robustness in small target detection tasks, thereby improving the detection performance of small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1This is a schematic diagram of the structure of the neural network of Example 1 of the present invention; (A) is a network structure framework diagram, the solid line represents the actual process of image features, and the dotted line represents the pointing detection result diagram. BOCM, CAM, and CIFM are respectively the brightness and orientation contrast module, the color antagonism module, and the cross-layer information fusion module. The white dotted box in the detection result diagram represents the area with similar small targets, which are magnified and displayed in A(1), A(2), A(3), and A(4) respectively. A(1) shows the detection effect of Van and Car; A(2) shows the detection effect of Tricycle; A(3) shows the detection effect of Awning-Tricycle, bicycle, and motor; A(4) shows the detection effect of pedestrian, people, and motor. (B) is the biological visual information transmission path - the ventral pathway.

[0041] Figure 2 This is a schematic structural diagram of the color antagonism module CAM in Example 1;

[0042] Figure 3 Schematic diagram of the BOCM architecture of Example 1. (A) illustrates the V1 / V2 color antagonistic receptive field organization, luminance-orientation contrast, and cross-layer information transmission within V2. (B) illustrates the VCBNet network architecture.

[0043] Figure 4 Schematic diagram of the structure of the azimuthal convolution of Example 1;

[0044] Figure 5 Schematic diagram of the structure of the VDE denoising module of Example 1;

[0045] Figure 6 This is a schematic structural diagram of the ITT Head module of Example 1;

[0046] Figure 7 Comparison charts showing the results of Ground Truth, YOLOv8m-P2, and Example 1 on the Visdrone2019 test set. DETAILED DESCRIPTION

[0047] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0048] Example 1

[0049] The lightweight small target detection method inspired by the ventral visual pathway includes the following steps:

[0050] A. Construct a neural network, such as Figure 1 As shown, the neural network includes a VCB network, a VDE denoising module, a PAN module, and an ITT Head module;

[0051] B. The original image is input into the VCB network and first processed by the color antagonism module CAM and the brightness and orientation contrast module BOCM to obtain the color antagonism result and the brightness and orientation adjustment result. The two processed results are then concatenated by the Concat operation and processed by the cross-layer information fusion module CIFM to obtain the VCB output features, which are input into the VDE denoising module and the PAN module respectively.

[0052] like Figure 2 As shown, the processing process in the color antagonism module CAM is as follows:

[0053] The input result is split into three channel images: blue, green, and red;

[0054] The blue channel image is convolved to obtain the blue channel result;

[0055] The green channel image is convolved to obtain the green channel result;

[0056] The red channel image is convolved to obtain the red channel result;

[0057] After the green channel result and the red channel result are concatenated through the Concat operation, a yellow splicing result is obtained. The yellow splicing result is then subjected to an anti-antagonism operation on the blue channel result to obtain a first anti-antagonism result.

[0058] The green channel result and the red channel result are subjected to an anti-antagonism operation to obtain a second anti-antagonism result;

[0059] The first anti-antagonism result and the second anti-antagonism result are added and fused to obtain an output result.

[0060] Anti-antagonistic operation by Figure 2 The CATConv module is used, wherein the process of performing the anti-antagonistic operation on the yellow splicing result and the blue channel result is as follows:

[0061] The yellow splicing result is processed by 1*1 convolution, and the blue channel result is processed by 3*3 convolution. The two convolution results are added and fused to obtain the first anti-antagonism result;

[0062] The process of performing the anti-antagonism operation on the green channel result and the red channel result is as follows:

[0063] The red channel result is processed by 1*1 convolution, and the green channel result is processed by 3*3 convolution. The two convolution results are added and fused to obtain the second anti-antagonism result.

[0064] like Figure 3As shown, the processing process in the brightness azimuth contrast model BOCM is as follows: the input result is divided into two paths, the first path is multiplied by the set constant α, 1<α<2, to obtain the dark processing result; the second path is multiplied by the set constant β, 0<β<1, to obtain the bright processing result;

[0065] The dark processing result is sequentially subjected to convolution and azimuth convolution to obtain a first processing result;

[0066] The dark processing result and the bright processing result are added and fused to obtain a second processing result;

[0067] The bright processing result is sequentially subjected to convolution and azimuth convolution to obtain the third processing result;

[0068] The first processing result, the second processing result, and the third processing result are added and fused to obtain the output result.

[0069] like Figure 4 As shown, the processing process in the azimuth convolution is as follows: the input result is divided into four paths, which pass through the azimuth convolution AZConv1 module, the azimuth convolution AZConv2 module, the azimuth convolution AZConv3 module, and the azimuth convolution AZConv4 module respectively; in the azimuth convolution AZConv1 module, the azimuth convolution AZConv2 module, the azimuth convolution AZConv3 module, and the azimuth convolution AZConv4 module, the convolution kernel part is masked by the Mask, and then multiplied with the input result to obtain four azimuth convolution results;

[0070] After the processing results of the azimuth convolution module AZConv2 and the azimuth convolution module AZConv3 are added and fused, the obtained fusion results are added and fused with the processing results of the azimuth convolution module AZConv1 and the azimuth convolution module AZConv4 to obtain the output result;

[0071] Among them, the Mask modes in the azimuth convolution AZConv1 module, azimuth convolution AZConv2 module, azimuth convolution AZConv3 module, and azimuth convolution AZConv4 module are 0°, 45°, 90°, and 135°, respectively.

[0072] C. The VDE denoising module suppresses background interference on the VCB output features and enhances the detail features to obtain the VDE output features, which are then input into the PAN module.

[0073] like Figure 5 As shown, the processing process in the VDE denoising module is as follows:

[0074] The input result is divided into two paths. The first path is processed by 1*1 convolution to obtain the first path result; the second path is processed by the spatial channel attention module CBAM to obtain the CBAM processing result. The CBAM processing result is divided into two parts through the split operation. The first part is processed by 3*3 convolution to obtain the first 3*3 convolution processing result;

[0075] The second part is divided into two parts after the split operation to obtain two second split results; one of the second split results is divided into two paths, one path is processed by 1*1 convolution, and the other path is processed by 3*3 convolution. After the two convolution results are added and fused, the first fusion result is obtained; the second split result is divided into two paths, one path is processed by 3*3 convolution, and the other path is processed by 5*5 convolution. After the two convolution results are added and fused, the second fusion result is obtained; the first fusion result and the second fusion result are spliced ​​by the Concat operation, and then processed by 3*3 convolution, and then spliced ​​with the first 3*3 convolution processing result by the Concat operation. The spliced ​​result and the first path result are input into the reverse attention module for processing, and the result obtained is the output result.

[0076] D. The PAN module performs multi-scale feature fusion on the VCB output features and the VDE output features to obtain feature layers of different resolutions, which are input into the two ITT Head modules for detection respectively, and a series of detection boxes and category data are obtained, which is the final result.

[0077] like Figure 6 As shown, the processing process in the ITT Head module is as follows:

[0078] After the input result is convolved by structural reparameterization, the Repconv result is divided into two paths.

[0079] The first path is processed in sequence by the spatial channel attention module CBAM, convolution layer, convolution layer, and ordinary convolution layer without pooling and activation layer to obtain the first-path processing result, which is the Bbox detection frame;

[0080] The first-path processing result is processed in sequence by a common convolution layer without pooling and activation layer, and then a convolution layer to obtain a backup result. The second-path processing result is obtained by adding and fusion of the result obtained by the convolution layer, convolution layer, and common convolution layer without pooling and activation layer to obtain the second-path processing result, which is the Class category.

[0081] Example 2

[0082] For quantitative performance evaluation of small target datasets, we adopt the same evaluation criteria as those in References 1-5. The specific evaluation is shown in Formula (1).

[0083]

[0084] Where class_num represents the number of categories, AP represents the accuracy, and its calculation formula (2) is as follows:

[0085]

[0086] Precision and Recall represent the precision and recall rates, respectively, as shown in the following formulas (3) and (4):

[0087]

[0088] Among them, TP represents the number of correctly predicted prediction boxes [IoU>threshold], FP represents the number of redundant prediction boxes detected as the same GT [IoU<threshold], and FN represents the number of prediction boxes that did not detect GT.

[0089] Document 1: N.Wang et al., "NAS-FCOS: Fast neural architecture search for object detection," in proceedings of the IEEE / CVF conference on computervision and pattern recognition, 2020, pp.11943-11951.

[0090] Document 2: C.Feng, Y.Zhong, Y.Gao, MRScott, and W.Huang, "Tood: Task-alignedone-stage object detection," in 2021 IEEE / CVF International Conference on Computer Vision (ICCV), 2021, pp.3490-3499: IEEE Computer Society.

[0091] Document 3: D.Ouyang et al., "Efficient multi-scale attention module with cross-spatial learning," in ICASSP 2023-2023IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp.1-5: IEEE.

[0092] Document 4: Z.Chen, H.Ji, Y.Zhang, Z.Zhu, and Y.Li, "High-Resolution FeaturePyramid Network for Small Object Detection On Drone View," IEEE Transactionson Circuits and Systems for Video Technology, 2023.

[0093] Document 5: Y.Li, Y.Wang, Z.Ma,

[0094] The parameters used in references 1-5 are the same as those in the original text and are guaranteed to be the optimal parameters for the model.

[0095] Table 1 summarizes the experimental data of References 1-5 and Example 1 on the UAV aerial photography dataset (VisDrone2019). The mAP comparison results of this method and the VisDrone2019 validation set are shown, indicating that this method has a significant advantage over other mAP indicator methods.

[0096] Table 1 Comparison of mAP results on the validation set of the UAV aerial photography dataset VisDrone2019.

[0097]

[0098] Table 2 verifies the effectiveness of the bionic model design. An ablation experiment was conducted on the UAV aerial photography dataset VisDrone2019. The specific experimental method is to replace and add the VCBNet, VDE, and ITT modules representing the V2, V4, and IT visual cortices in Example 1, and replace or add some visual features of the visual cortex to the baseline. It can be seen from Table 2 that the application of the corresponding modules successively improves the small target detection performance of the model.

[0099] Table 2 Qualitative analysis of Example 1 in VisDrone 2019

[0100]

[0101] Example 3

[0102] Figure 7 The results of Ground Truth, YOLOv8m-P2, and Example 1 are shown. Ground Truth represents the actual data set, while YOLOv8m-P2 is a small object model developed by YOLOv8 for small object detection. Due to the dense distribution and mutual occlusion of small objects, YOLOv8m-P2 has a large number of undetected or detected objects of similar categories. However, Example 1, our model, achieves excellent detection results, with a small difference compared to Ground Truth.

Claims

1. A lightweight small target detection method inspired by the ventral visual pathway, characterized by: The following steps are involved: A. Constructing a neural network, wherein the neural network includes a VCB network, a VDE denoising module, a PAN module, and an ITT Head module; B. The original image is input into the VCB network and processed by the color antagonism module CAM and the brightness and orientation contrast module BOCM respectively to obtain the color antagonism result and the brightness and orientation adjustment result. The two processed results are then concatenated by the Concat function and processed by the cross-layer information fusion module CIFM to obtain the VCB output features, which are input into the VDE denoising module and the PAN module respectively. C. The VDE denoising module suppresses background interference on the VCB output features and enhances the detail features to obtain the VDE output features, which are then input into the PAN module. D. The PAN module performs multi-scale feature fusion on the VCB output features and the VDE output features to obtain feature layers of different resolutions, which are input into the two ITT Head modules for detection respectively, and a series of detection boxes and category data are obtained, which is the final result; The processing process in the color antagonism module CAM is as follows: The input result is split into three channel images: blue, green, and red; The blue channel image is convolved to obtain the blue channel result; The green channel image is convolved to obtain the green channel result; The red channel image is convolved to obtain the red channel result; The green channel result and the red channel result are concatenated by the Concat function to obtain the yellow splicing result. The yellow splicing result is antagonized with the blue channel result to obtain the first antagonistic result. The green channel result and the red channel result are subjected to an antagonistic operation to obtain a second antagonistic result; The first antagonistic result and the second antagonistic result are added and fused to obtain an output result; The processing process in the brightness azimuth contrast module BOCM is as follows: the input result is divided into two paths, the first path is multiplied by a set constant α, 1<α<2, to obtain a dark processing result; the second path is multiplied by a set constant β, 0<β<1, to obtain a bright processing result; The dark processing result is sequentially subjected to convolution and azimuth convolution to obtain a first processing result; The dark processing result and the bright processing result are added and fused to obtain a second processing result; The bright processing result is sequentially subjected to convolution and azimuth convolution to obtain the third processing result; The first processing result, the second processing result, and the third processing result are added and fused to obtain the output result.

2. The lightweight small target detection method inspired by the ventral visual pathway according to claim 1, characterized in that: The process of performing the antagonistic operation on the yellow splicing result and the blue channel result is as follows: The yellow splicing result is processed by 1*1 convolution, and the blue channel result is processed by 3*3 convolution. The two convolution results are added and fused to obtain the first antagonistic result; The process of performing the antagonistic operation on the green channel result and the red channel result is as follows: The red channel result is processed by 1*1 convolution, and the green channel result is processed by 3*3 convolution. The two convolution results are added and fused to obtain the second antagonistic result.

3. The lightweight small target detection method inspired by the ventral visual pathway according to claim 1, characterized in that: The processing process in the azimuth convolution is as follows: the input result is divided into four paths, which pass through the azimuth convolution AZConv1 module, the azimuth convolution AZConv2 module, the azimuth convolution AZConv3 module, and the azimuth convolution AZConv4 module respectively; in the azimuth convolution AZConv1 module, the azimuth convolution AZConv2 module, the azimuth convolution AZConv3 module, and the azimuth convolution AZConv4 module, the convolution kernel part is masked by the mask, and then multiplied with the input result to obtain four azimuth convolution results; After the processing results of the azimuth convolution module AZConv2 and the azimuth convolution module AZConv3 are added and fused, the obtained fusion results are added and fused with the processing results of the azimuth convolution module AZConv1 and the azimuth convolution module AZConv4 to obtain the output result; Among them, the Mask modes in the azimuth convolution AZConv1 module, azimuth convolution AZConv2 module, azimuth convolution AZConv3 module, and azimuth convolution AZConv4 module are 0°, 45°, 90°, and 135°, respectively.

4. The lightweight small target detection method inspired by the ventral visual pathway according to claim 1, characterized in that: The processing process in the VDE denoising module is as follows: The input result is divided into two paths. The first path is processed by 1*1 convolution to obtain the first path result; the second path is processed by the spatial channel attention module CBAM to obtain the CBAM processing result. The CBAM processing result is divided into two parts through the split operation. The first part is processed by 3*3 convolution to obtain the first 3*3 convolution processing result; The second part is divided into two parts after the split operation to obtain two second split results; one of the second split results is divided into two paths, one path is processed by 1*1 convolution, and the other path is processed by 3*3 convolution. After the two convolution results are added and fused, the first fusion result is obtained; the second split result is divided into two paths, one path is processed by 3*3 convolution, and the other path is processed by 5*5 convolution. After the two convolution results are added and fused, the second fusion result is obtained; the first fusion result and the second fusion result are spliced ​​by the Concat function, and then processed by 3*3 convolution, and then spliced ​​with the first 3*3 convolution result by the Concat function. The spliced ​​result and the first path result are input into the reverse attention module for processing, and the result obtained is the output result.

5. The lightweight small target detection method inspired by the ventral visual pathway according to claim 1, characterized in that: The processing process in the ITT Head module is as follows: After the input result is convolved through structural reparameterization, the Repconv result is divided into two paths. The first path is processed by the spatial channel attention module CBAM, convolution layer, convolution layer, and ordinary convolution layer without pooling and activation layer in sequence to obtain the first-path processing result, which is the Bbox detection frame; The first processing result is processed by the spatial channel attention module CBAM and the convolution layer in sequence to obtain the backup result; The second path is processed in sequence by the convolution layer, convolution layer, and ordinary convolution layer without pooling and activation layer. The obtained result is added and fused with the backup result to obtain the second processing result, which is the Class category.

Images