Multi-scale feature weight enabling method and system and image target detection method

Abstract

The invention discloses a multi-scale feature weight enabling method, a multi-scale feature weight enabling system and an image target detection method, and belongs to a visual image target detection method based on deep learning. Multi-level features of a detection image are input through a feature extraction network, and the multi-level features comprise a shallow-level feature map, a middle-level feature map and a deep-level feature map; three times of weight enabling are carried out on the multi-level features to obtain an output feature map, multi-scale target detection in a detection image is carried out, specific fusion is carried out on feature maps of different levels to carry out weight enabling, and fusion and information interaction of shallow-layer features and deep-layer features in the multi-scale features are enhanced. Deep learning prediction is carried out on the output feature map through the visual detection model, so that information complementation between shallow and deep features is realized, and the high-precision detection requirement of a multi-scale target in image target detection is met; the method is especially suitable for visual reasoning tasks such as surface defect detection of complex industrial products and detection of specific ground targets by unmanned aerial vehicles.

Description

Technical Field

[0001] The present invention discloses a multi-scale feature weight empowerment method, system and image target detection method, which belong to a visual image target detection method based on deep learning. Background Art

[0002] Deep learning-based object detection methods are widely used in visual reasoning tasks such as surface defect detection for complex industrial products and drone-based ground target detection. Due to the diverse and multi-scale nature of surface defects in complex industrial products and drone aerial imagery, defect features in acquired product images and ground-based features in aerial images often exhibit multi-scale characteristics. Multi-scale object detection is a major challenge for deep learning-based object detection. It is well known that in deep learning-based object detection methods, deep features extracted by feature extraction networks typically contain strong semantic information but weak localization information; shallow features, on the other hand, exhibit strong localization information but weak semantic information. Deep learning-based object detection methods often incorporate feature pyramid models and their related variants, such as FPN and PAN. By fusing and interacting shallow and deep features extracted by the feature extraction networks in deep learning object detection models, the network's detection performance for multi-scale objects can be improved. For example, patent application CN202411458061.2 discloses a method for multi-view feature extraction using a lightweight Transformer-CNN fusion network. However, existing feature fusion networks tend to adopt a simple feature addition or connection strategy. This feature fusion method will indiscriminately enhance the positioning and semantic information of all targets (including background noise). Although the target features are amplified, the background noise is also amplified. This is not only of no substantial help for multi-scale target detection, but may also lead to false detection and missed detection due to excessive amplification of complex background noise. It fails to fully realize the information complementarity between shallow and deep features, and cannot meet the high-precision detection requirements of multi-scale targets in images. Summary of the Invention

[0003] The technical problem solved by the present invention is: to address the problem that the feature fusion network in the existing deep learning target detection method treats features of different scales indiscriminately, resulting in indiscriminate enhancement of the detected target and background, and to provide a multi-scale feature weighting enabling method, system and image target detection method to improve the detection accuracy of the deep learning-based target detection method for multi-scale targets in the image.

[0004] The present invention is implemented by the following technical solutions:

[0005] The present invention first discloses a multi-scale feature weight empowerment method, which inputs multi-level features of a detection image through a feature extraction network, including a shallow feature map, an intermediate feature map and a deep feature map, and performs three weight empowerments on the multi-level features to obtain an output feature map to perform multi-scale target detection in the detection image. In the first weight empowerment, the shallow feature map and the intermediate feature map are preliminarily fused to generate a first weight feature map, and the deep feature map is re-weighted using the first weight feature map to obtain a first output feature map; in the second weight empowerment, the first output feature map and the shallow feature map are preliminarily fused to generate a second weight feature map, and the intermediate feature map is re-weighted using the second weight feature map to obtain a second output feature map; in the third weight empowerment, the first output feature map and the second output feature map are preliminarily fused to generate a third weight feature map, and the shallow feature map is re-weighted using the third weight feature map to obtain a third output feature map.

[0006] In the multi-scale feature weight empowerment method of the present invention, further, in the first weight empowerment, the shallow feature map and the intermediate feature map are fused to obtain a first fused feature map, a first weight feature map is generated from the first fused feature map, the first fused feature map is fused with the deep feature map, and then the fused feature map is element-wise multiplied with the first weight feature map to obtain a first intermediate feature map, and the first fused feature map, the first intermediate feature map and the shallow feature map are spliced ​​to obtain a first output feature map.

[0007] In the multi-scale feature weight empowerment method of the present invention, further, in the first weight empowerment, the shallow feature map and the intermediate feature map are both downsampled and fused element by element to obtain a first fused feature map, and the first fused feature map is fused element by element with the deep feature map.

[0008] In the multi-scale feature weight empowerment method of the present invention, further, in the second weight empowerment, the shallow feature map and the first output feature map are fused to obtain a second fused feature map, a second weight feature map is generated from the second fused feature map, the second fused feature map is fused with the intermediate feature map, and then the fused feature map is element-wise multiplied with the second weight feature map to obtain a second intermediate feature map, and the second intermediate feature map, the first output feature map and the shallow feature map are spliced ​​to obtain the second output feature map.

[0009] In the multi-scale feature weight empowerment method of the present invention, further, in the second weight empowerment, the shallow feature map is downsampled and the upsampled first output feature map is element-by-element added and fused to obtain a second fused feature map, and the second fused feature map is element-by-element added and fused with the downsampled intermediate feature map.

[0010] In the multi-scale feature weight empowerment method of the present invention, further, in the third weight empowerment, the first output feature map and the second output feature map are fused to obtain a third fused feature map, a third weight feature map is generated from the third fused feature map, the third fused feature map is fused with the shallow feature map, and then the third weight feature map is multiplied element-by-element with the third weight feature map to obtain a third intermediate feature map, and the third intermediate feature map, the third fused feature map and the first output feature map are spliced ​​to obtain a third output feature map.

[0011] In the multi-scale feature weight empowerment method of the present invention, further, the first output feature map and the second output feature map are both upsampled and fused element by element to obtain a third fused feature map, and the third fused feature map is fused element by element with the shallow feature map.

[0012] In the multi-scale feature weight enabling method of the present invention, further, the first weight feature map, the second weight feature map and the third weight feature map are all generated by the fused feature maps through the Sigmoid function.

[0013] The present invention also discloses a multi-scale feature weight empowerment system, including a feature extraction module, a first weight empowerment module, a second weight empowerment module, and a third weight empowerment module, wherein the feature extraction module extracts multi-level features of the detection image, including a shallow feature map, an intermediate feature map, and a deep feature map; the first weight empowerment module inputs the shallow feature map, the intermediate feature map, and the deep feature map, preliminarily fuses the shallow feature map and the intermediate feature map to generate a first weight feature map, re-empowers the deep feature map through the first weight feature map, and outputs a first output feature map; The second weight empowerment module inputs the shallow feature map, the intermediate feature map and the first input feature map, preliminarily fuses the first output feature map and the shallow feature map to generate a second weight feature map, re-empowers the intermediate feature map through the second weight feature map, and outputs the second output feature map; the third weight empowerment module inputs the shallow feature map, the first output feature map and the second output feature map, preliminarily fuses the first output feature map and the second output feature map to generate a third weight feature map, re-empowers the shallow feature map through the third weight feature map, and obtains the third output feature map.

[0014] The present invention also discloses an image target detection method using the above-mentioned multi-scale feature weight empowerment method, taking YOLOv5 as a reference input detection image, adopting Darknet-53 as a feature extraction network, extracting multi-level features in the detection image, selecting shallow feature maps, middle feature maps and deep feature maps, and performing three-fold weight empowerment using the above-mentioned multi-scale feature weight empowerment method of the present invention. The obtained output feature maps are respectively connected to the three prediction heads of YOLOv5, and the target detection results of the detection image are output.

[0015] The multi-scale feature weighting method and system of the present invention fully considers the characteristics of features at different levels and applies differentiated weighting strategies to specifically amplify the feature information of multi-scale targets while suppressing noise generated by complex backgrounds. Specifically, three multi-scale feature maps of different levels, from shallow to deep, are selected as input. The multi-scale feature maps are weighted three times to deeply fuse the shallow and deep features in the feature maps.

[0016] In the first weight empowerment, the first weight feature map is generated by preliminarily fusing the shallow feature map and the intermediate feature map, and then the deep feature map is re-weighted to enhance the positioning information of the deep feature map and suppress noise.

[0017] In the second weight empowerment, the shallow feature map, the middle feature map and the first output feature map after the first weight empowerment are used. Figure 3 The second weighted feature map is generated by preliminarily fusing the shallow feature map and the first quick feature map, and then the middle-level feature map is re-empowered to enhance the positioning information and semantic information of the middle-level feature map.

[0018] In the third weight empowerment, the shallow feature map, the first output feature map after the first weight empowerment, and the second output feature map after the second weight empowerment are used. Figure 3 The third weighted feature map is generated by preliminarily fusing the second output feature map and the first output feature map, and then the shallow feature map is re-weighted to enhance the semantic information of the shallow feature map and suppress noise.

[0019] Before fusion, since the input feature layers have different scales, the three modules must first perform upsampling or downsampling operations to unify the feature layers of different scales to the same size to ensure the effectiveness of fusion.

[0020] In summary, the multi-scale feature weight empowerment method and system provided by the present invention performs weight empowerment by performing specific fusion of feature maps at different levels, thereby enhancing the fusion and information interaction of shallow features and deep features in multi-scale features, realizing information complementarity between shallow and deep features, and performing deep learning prediction on the output feature map through the visual detection model, thereby meeting the high-precision detection requirements of multi-scale targets in image target detection, and is particularly suitable for visual reasoning tasks such as surface defect detection of complex industrial products and detection of specific ground targets by drones.

[0021] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1Schematic diagram of a multi-scale feature weight enabling system in an embodiment.

[0023] Figure 2 This is a flow chart of the first weight empowerment process of the multi-scale feature weight empowerment system in the embodiment.

[0024] Figure 3 This is a flow chart of the second weight empowerment process of the multi-scale feature weight empowerment system in the embodiment.

[0025] Figure 4 This is a flow chart of the third weight empowerment process of the multi-scale feature weight empowerment system in the embodiment. DETAILED DESCRIPTION

[0026] Example

[0027] See also Figure 1 , the figure shows a specific implementation scheme of the multi-scale feature weight empowerment system of the present invention, which extracts multi-level features in the detection image according to the feature extraction network in the deep learning convolutional neural network, and inputs the multi-level features of the detection image through a multi-stage feature extraction network, such as a four-stage feature extraction network, and selects the last three levels of feature maps as the input of the multi-scale feature weight empowerment system, which are successively recorded as the first-level feature map, the second-level feature map and the third-level feature map, wherein the first-level feature map is a shallow feature map, the second-level feature map is an intermediate feature map, and the third-level feature map is a deep feature map. The multi-level features are weighted three times to obtain the output feature map to perform multi-scale target detection in the detection image.

[0028] Specifically, the multi-scale feature weight empowerment system includes a feature extraction module, a first weight empowerment module, a second weight empowerment module and a third weight empowerment module. The shallow feature map, the intermediate feature map and the deep feature map are used as inputs for weight empowerment, and the shallow feature map, the intermediate feature map and the deep feature map are weight empowered three times to obtain the output feature map. Among them, the feature extraction module extracts multi-level features of the detection image, including shallow feature maps, intermediate feature maps and deep feature maps; the first weight empowerment module inputs the shallow feature map, the intermediate feature map and the deep feature map, preliminarily fuses the shallow feature map and the intermediate feature map to generate a first weight feature map, re-empowers the deep feature map through the first weight feature map, and outputs a first output feature map; the second weight empowerment module inputs the shallow feature map, the intermediate feature map and the first input feature map, preliminarily fuses the first output feature map and the shallow feature map to generate a second weight feature map, re-empowers the intermediate feature map through the second weight feature map, and outputs a second output feature map; the third weight empowerment module inputs the shallow feature map, the first output feature map and the second output feature map, preliminarily fuses the first output feature map and the second output feature map to generate a third weight feature map, re-empowers the shallow feature map through the third weight feature map, and obtains a third output feature map.

[0029] The specific process of the multi-scale feature weight empowerment method for extracting multi-level features in the detection image by the feature extraction network is as follows: Figure 2 、 Figure 3 and Figure 4 shown.

[0030] The first weight empowerment is as follows Figure 2 As shown in the figure, the input of the first weight empowerment is the feature extraction network to extract multi-level features of different scales in the detection image: shallow feature map F2, intermediate feature map F2 and deep feature map F4, where the number of channels of the shallow feature map F2 and the intermediate feature map F3 are both 128, and the number of channels of the deep feature map F4 is 256.

[0031] In this embodiment, the resolutions of feature maps at different levels are different. The resolution of the deep feature map is 1 / 2 of the intermediate feature map, and the resolution of the intermediate feature map is 1 / 2 of the previous feature map. Therefore, when weighting, the shallow feature map F2 is first downsampled twice to obtain F2', and the intermediate feature map F3 is downsampled once to obtain F3', so that the resolutions of feature maps at different levels are consistent.

[0032] The first fusion feature map F is obtained by element-by-element addition of F2' and F3'. 23 ´, by the first fusion feature map F 23 ´´Through a standard 3×3 convolution layer, the number of channels is increased to 256, and the output is the feature map , feature map Generate the first weight feature map through the Sigmoid function , the first weight feature map It reflects the importance of the shallow feature map F2 and the intermediate feature map F3.

[0033] The first fusion feature map F 23 ´Feature map after convolution The feature map is obtained by element-by-element addition and fusion with the deep feature map F4 , feature map With the first weight feature map Perform element-by-element multiplication to obtain the first intermediate feature map .

[0034] The first fusion feature map F 23 ´, First intermediate feature map It is concatenated with the sampled shallow feature map F2' to obtain the first output feature map F4''.

[0035] The second weight empowerment is as follows Figure 3 As shown in Figure 1, the input of the second weight empowerment is the shallow feature map F2, the intermediate feature map F3 and the first output feature map F4´´. The number of channels of the shallow feature map F2 is 128, the number of channels of the intermediate feature map F3 is 128, and the number of channels of the first output feature map F4´´ is 64.

[0036] First, the shallow feature map F2 is downsampled to obtain F 22 ´, the intermediate feature map F3 is obtained by downsampling once 33 ´, the first output feature map F4´´ is obtained by upsampling once 44 ´.

[0037] After sampling, F 22 ´ and F 44 Perform element-by-element addition and fusion to obtain the second fusion feature map F 24 ´, by the second fusion feature map F 24 ´Through a standard 3×3 convolution layer, the number of channels is increased to 256 to obtain the feature map , feature map Generate the second weight feature map through the Sigmoid function .

[0038] The second fusion feature map F 24 ´Feature map after convolution F after sampling with the intermediate feature map F3 33 ' Perform element-by-element addition and fusion to obtain , feature map With the second weight feature map Perform element-by-element multiplication to obtain the second intermediate feature map .

[0039] The second intermediate feature map , the first output feature map F after sampling 44 ´ and the shallow feature map F after sampling 22 ´ is concatenated to obtain the second output feature map F3´´.

[0040] The third weight empowerment is as follows Figure 4 As shown, the input of the third weight empowerment is the shallow feature map F2, the first output feature map F4´´ and the second output feature map F3´´. The number of channels of the shallow feature map F2 is 128, the number of channels of the first output feature map F4´´ is 64, and the number of channels of the second output feature map F3´´ is 64.

[0041] First, the first output feature map F4'' is upsampled twice to obtain F 42 ´, the second output feature map F3´´ is obtained by upsampling once 32 ´.

[0042] The first output feature map F after sampling 42 ´ and the second output feature map F after sampling 32 Perform element-by-element addition and fusion to obtain the third fusion feature map F 34 ´, by the third fusion feature map F 34 ´Through a standard 3×3 convolution layer, the number of channels is increased to 256 to obtain the feature map , feature map Generate the third weight feature map through the Sigmoid function .

[0043] The third fusion feature map F 34 ´Feature map after convolution After element-by-element addition and fusion with the shallow feature map F2, we get , feature map With the third weight feature map Perform element-by-element multiplication to obtain the third intermediate feature map .

[0044] The third intermediate feature map , the third fusion feature map F 34 ´ and the first output feature map F after sampling 42 ´ is concatenated to obtain the third output feature map F2´´.

[0045] Finally, the first output feature map F4´´, the second output feature map F3´´, and the third output feature map F2´´ are respectively connected to the three prediction heads of the image target detection model for detecting targets in the image. Taking the YOLOv5 image target detection model as an example, YOLOv5 is used as the reference input detection image, Darknet-53 is used as the feature extraction network, and multi-level features in the detection image are extracted. Then, shallow feature maps, mid-level feature maps, and deep feature maps are selected, and weight empowerment is performed three times using the multi-scale feature weight empowerment method described above in this embodiment. The obtained first output feature map, second output feature map, and third output feature map are respectively connected to the three prediction heads of YOLOv5. YOLOv5 performs deep learning prediction on the output feature maps, and outputs the target detection results of the detection image.

[0046] The multi-scale feature weighting method and system of this embodiment were applied to the visual inspection task of detecting substation equipment from infrared images captured by drones. The deep learning convolutional neural network YOLOv5 was used as the visual inspection model. YOLOv5's feature extraction network used Darknet-53. The extracted feature maps were denoted as C1, C2, C3, C4, and C5, respectively, from shallow to deep layers. Feature maps C3, C4, and C5 were selected as input, i.e., F2 = C3, F3 = C4, and F4 = C5.

[0047] In order to verify the effectiveness of this embodiment, the conventional YOLOv5 model is used as a benchmark to compare the detection effect, parameter amount, computational complexity and inference speed of substation equipment in infrared images after adding this embodiment. The experimental results are shown in Table 1. It can be seen from Table 1 that the detection accuracy mAP of substation equipment after adding this embodiment is 50 It has been improved by 1.6%, the number of parameters Params has increased slightly, the computational complexity FLOPs has decreased, and the inference speed FPS has increased.

[0048] Table 1 Ablation experiment of this embodiment in YOLOv5.

[0049] .

[0050] Then, we selected the currently mainstream multi-scale feature fusion structures for comparative experiments, including the multi-scale feature sequence fusion structure (SSFF), Gather and Distribuite (GaD), HFAMPAN, BiSFPN, RepNCSPFPN, GFPN, Lowlevel-Feature-Alignment (LFA), and FPN+PAN structures. We replaced the Neck of YOLOv5 with the above multi-scale feature fusion structures and compared the detection accuracy of multi-scale objects with this embodiment. The results are shown in Table 2.

[0051] Table 2 Comparison between this embodiment and other multi-scale feature fusion structures.

[0052] .

[0053] From the data analysis in Table 2, we can see that this embodiment has a high mAP 50 It achieved the best mAP of 97.4%, which is 0.4% higher than the second best RepNCSPFPN 97%. 50:95 In terms of indicators, this embodiment achieved the second best performance of 75.8%, while the best performance was SSFF, mAP 50:95 It reached 76.1%, while the mAP of SSFF 50 Only 95.5, lower than the mAP of the present invention 50 Although the number of parameters and inference speed (FPS) of this embodiment are not outstanding in Table 2, the computational complexity (FLOPs) of this embodiment is the lowest in Table 2, being 80% of the computational complexity (FLOPs) of the second FPN+PAN. Furthermore, the inference speed (FPS) of this embodiment reaches 107.5 frames per second, meeting the requirements of industrial real-time detection.

[0054] In this document, the directions or positional relationships indicated by terms such as "up", "down", "front", "back", "left", "right", "top", "bottom", "inside", "outside", "vertical", and "horizontal" are based on the directions or positional relationships shown in the accompanying drawings and are only for the clarity of the technical solution and the convenience of description, and therefore should not be understood as limiting the present invention.

[0055] As used herein, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion of elements other than the listed elements and may also include additional elements not specifically listed.

[0056] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. Multi-scale feature weight empowerment method, characterized by: The multi-level features of the detection image are input through the feature extraction network, including shallow feature maps, mid-level feature maps and deep feature maps. The multi-level features are weighted three times to obtain the output feature map to detect multi-scale targets in the detection image, where: In the first weight empowerment, the shallow feature map and the intermediate feature map are initially fused to generate the first weight feature map, and the deep feature map is re-weighted by the first weight feature map to obtain the first output feature map; In the second weight empowerment, the first output feature map and the shallow feature map are preliminarily fused to generate a second weight feature map, and the intermediate feature map is re-weighted by the second weight feature map to obtain the second output feature map; In the third weight empowerment, the first output feature map and the second output feature map are preliminarily fused to generate a third weight feature map. The shallow feature map is re-weighted through the third weight feature map to obtain the third output feature map.

2. The multi-scale feature weighting method according to claim 1, characterized in that: In the first weight empowerment, the shallow feature map and the intermediate feature map are fused to obtain a first fused feature map, a first weight feature map is generated from the first fused feature map, the first fused feature map is fused with the deep feature map, and then the fused feature map is element-wise multiplied with the first weight feature map to obtain a first intermediate feature map, and the first fused feature map, the first intermediate feature map and the shallow feature map are spliced ​​to obtain a first output feature map.

3. The multi-scale feature weighting method according to claim 2, characterized in that: In the first weight empowerment, the shallow feature map and the intermediate feature map are both downsampled and fused element by element to obtain a first fused feature map, and the first fused feature map is fused element by element with the deep feature map.

4. The multi-scale feature weighting method according to claim 1, characterized in that: In the second weight empowerment, the shallow feature map and the first output feature map are fused to obtain a second fused feature map, a second weight feature map is generated from the second fused feature map, the second fused feature map is fused with the intermediate feature map, and then the resultant is element-wise multiplied with the second weight feature map to obtain a second intermediate feature map, and the second intermediate feature map, the first output feature map and the shallow feature map are spliced ​​to obtain a second output feature map.

5. The multi-scale feature weighting method according to claim 4, characterized in that: In the second weight empowerment, the shallow feature map is downsampled and fused with the upsampled first output feature map element by element to obtain a second fused feature map, and the second fused feature map is added and fused with the downsampled intermediate feature map element by element.

6. The multi-scale feature weighting method according to claim 1, characterized in that: In the third weight empowerment, the first output feature map and the second output feature map are fused to obtain a third fused feature map, a third weighted feature map is generated from the third fused feature map, the third fused feature map is fused with the shallow feature map, and then multiplied element-by-element with the third weighted feature map to obtain a third intermediate feature map, and the third intermediate feature map, the third fused feature map and the first output feature map are spliced ​​to obtain a third output feature map.

7. The multi-scale feature weighting method according to claim 6, characterized in that: The first output feature map and the second output feature map are both upsampled and fused element by element to obtain a third fused feature map, and the third fused feature map is fused element by element with the shallow feature map.

8. The multi-scale feature weighting method according to claim 1, characterized in that: The first weight feature map, the second weight feature map and the third weight feature map are all generated by the fused feature maps through the Sigmoid function.

9. Multi-scale feature weight empowerment system, characterized by: include: Feature extraction module, which extracts multi-level features of the detection image, including shallow feature maps, mid-level feature maps, and deep feature maps; The first weight empowerment module inputs a shallow feature map, an intermediate feature map, and a deep feature map, preliminarily fuses the shallow feature map and the intermediate feature map to generate a first weighted feature map, re-empowers the deep feature map through the first weighted feature map, and outputs a first output feature map; The second weight empowerment module inputs the shallow feature map, the intermediate feature map and the first input feature map, preliminarily fuses the first output feature map and the shallow feature map to generate a second weighted feature map, re-empowers the intermediate feature map through the second weighted feature map, and outputs the second output feature map; The third weight empowerment module inputs the shallow feature map, the first output feature map and the second output feature map, preliminarily fuses the first output feature map and the second output feature map to generate a third weighted feature map, and re-empowers the shallow feature map through the third weighted feature map to obtain the third output feature map.

10. An image target detection method, characterized in that: YOLOv5 is used as a reference input detection image, Darknet-53 is used as a feature extraction network, and after extracting multi-level features in the detection image, a shallow feature map, a middle feature map, and a deep feature map are selected. Three weight empowerment operations are performed using the multi-scale feature weight empowerment method described in any one of claims 1 to 8. The obtained output feature maps are respectively connected to the three prediction heads of YOLOv5, and the target detection result of the detection image is output.

Images