A target detection system and method based on multi-scale feature calibration

Abstract

The present invention relates to a target detection system and method based on multi-scale feature calibration. By setting a trunk module, a calibration module and a detection head module, the trunk module uses a composite convolution method to extract features of a detected image m times to obtain m feature images, thereby achieving the technical effect of obtaining features of different scales and increasing the output of a high-resolution feature layer to capture the details of small targets; then the calibration module calibrates the m feature images output by the trunk module by calibrating shallow features with deep features during the feature fusion process, thereby sampling the most representative information from deep features for each position, reducing the probability of missed detection, and alleviating the problem of losing key details due to spatial dislocation or information dilution during the feature fusion process. In addition, due to the setting of the detection head module, the use of shared parameter convolution can greatly reduce the amount of parameters and calculations, and solve the problem of inconsistent scales of targets detected by each detection head.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a target detection system and method based on multi-scale feature calibration. Background Art

[0002] The essence of target detection is to detect and locate objects by combining feature extraction and classification. The target detection task in industrial scenarios faces the challenges of complex background, dense distribution of small targets and limited resources, which puts higher requirements on the lightweight and detection accuracy of target detection systems or methods.

[0003] In the prior art, the target detection system based on deep neural network is divided into two-stage detection system and single-stage detection system. The two-stage detection system first generates several candidate regions, and then extracts and classifies the features of these candidate regions. However, the two-stage detection system has the problems of high computational complexity, large number of model parameters, difficulty in parameter optimization and slow detection speed, and is not suitable for deployment on resource-constrained edge devices.

[0004] The single-stage detection system does not need to generate candidate regions, but directly generates the bounding box and category prediction of the target in an end-to-end manner. Since the entire process only requires one step, its detection speed is relatively fast. However, in complex industrial scenes, the single-stage detection system in the existing technology has technical defects such as low signal-to-noise ratio of small targets and missed detection, so its performance in small target detection tasks in complex industrial scenes is limited. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to overcome the technical defects of the target detection system in the prior art, such as missed detection and difficulty in deployment on resource-constrained edge devices. In order to overcome the above defects of the prior art, the present invention provides a target detection system and method based on multi-scale feature calibration, including a target detection system based on multi-scale feature calibration and a target detection method based on multi-scale feature calibration.

[0006] The present invention provides a target detection system based on multi-scale feature calibration, comprising:

[0007] The main module is configured to perform m feature extractions on the image under test through a composite convolution method to obtain m feature images, and output the feature images obtained this time after each feature extraction of the first (m-1) times and use them as the operation object of the next feature extraction, and output the feature images obtained this time after the mth feature extraction;

[0008] A calibration module is configured to calibrate the m feature images output by the backbone module by calibrating shallow features with deep features during feature fusion to obtain m calibration images;

[0009] The detection head module is configured to perform shared parameter convolution detection and feature scaling on the m calibration images output by the calibration module to obtain the corresponding detection targets;

[0010] in:

[0011] m ≥ 2;

[0012] The calibration module communicates with the trunk module, and the detection head module communicates with the calibration module.

[0013] The target detection system based on multi-scale feature calibration disclosed in the present invention is provided with a trunk module, a calibration module and a detection head module, the calibration module communicates with the trunk module, and the detection head module communicates with the calibration module. Since the trunk module uses a composite convolution method to extract features of the image under test m times to obtain m feature images, it is possible to achieve the technical effect of obtaining features of different scales and increasing the output of the high-resolution feature layer to capture the details of small targets; then, the calibration module calibrates the m feature images output by the trunk module by calibrating the shallow features through deep features in the feature fusion process, so that the most representative information can be sampled from the deep features for each position, reducing the probability of missed detection, and alleviating the problem of losing key details due to spatial dislocation or information dilution during the feature fusion process. Moreover, due to the setting of the detection head module, its use of shared parameter convolution can greatly reduce the amount of parameters and the amount of calculation, and solve the problem of inconsistent scales of the targets detected by each detection head. Finally, while maintaining a small amount of parameters and the amount of calculation, a better detection effect is achieved, making the entire system easy to deploy on resource-constrained edge devices.

[0014] In a possible implementation, the backbone module is a network structure formed by connecting one fast pyramid pooling module, multiple first composite convolution modules, and no less than m first C2f network modules in series, the first part of the network structure is one of the first composite convolution modules, and the end of the network structure is one of the fast pyramid pooling modules;

[0015] Wherein (m-1) of the first C2f network modules communicate with the calibration module, and the fast pyramid pooling module communicates with the calibration module;

[0016] This scheme sets multiple first composite convolution modules and no less than m first C2f network modules to increase the high-resolution feature layer to capture small target details, which helps the calibration module to sample the most representative information from the deep features at each position, and can further alleviate the problem of losing key details due to spatial misalignment or information dilution during the feature fusion process.

[0017] In a possible implementation, the calibration module includes:

[0018] The encoder is configured to encode the m feature images output by the backbone module (m-1) times by upsampling, concatenating and extracting feature images to obtain (m-1) encoded images;

[0019] A decoder is configured to decode and calibrate spatial features of the (m-1) coded images obtained by the encoder and the feature image output by the backbone module for the mth time by combining downsampling, splicing, extracting feature maps and spatial feature calibration to obtain m calibrated images;

[0020] in:

[0021] The encoder communicates with (m-1) first C2f network modules and the fast pyramid pooling module respectively, the decoder communicates with the encoder and the fast pyramid pooling module respectively, and the detection head module communicates with the decoder;

[0022] This scheme improves the operating mechanism of the encoder and decoder to use deep features to calibrate shallow features, which can reduce the number of operating parameters and improve the model's ability to detect dense small targets.

[0023] In a possible implementation manner, the encoder is a network structure formed by (m-1) sub-encoders connected in series, each of the sub-encoders except the sub-encoder at the end includes one upsampling module, one first splicing module and one second C2f network module sequentially arranged along its running direction, and the sub-encoder at the end includes one upsampling module and one first splicing module sequentially arranged along its running direction;

[0024] The upsampling module in the first sub-encoder communicates with the fast pyramid pooling module, the (m-1) first C2f network modules communicating with the encoder communicate with all the first splicing modules in a one-to-one connection manner, and all the second C2f network modules and the first splicing module in the last sub-encoder communicate with the decoder;

[0025] This solution can ensure the high efficiency of deep feature calibration of shallow features, enhance the detection capability of small targets, and keep the model lightweight.

[0026] In a possible implementation manner, the decoder is a network structure formed by m sub-decoders connected in series;

[0027] Each of the sub-decoders except the sub-decoder at the end includes 1 spatial feature calibration module, 1 downsampling module, 1 second splicing module and 1 third C2f network module, and the sub-decoder at the end includes 1 third C2f network module;

[0028] In each of the sub-decoders except the sub-decoder at the end: the down-sampling module communicates with the third C2f network module, and the second splicing module communicates with the down-sampling module;

[0029] The spatial feature calibration modules in each of the sub-decoders communicate with the third C2f network module in the sub-decoder located at the end, and each of the spatial feature calibration modules also communicates with the third C2f network module in the sub-decoder to which it belongs;

[0030] The third C2f network module in the first sub-decoder communicates with the first splicing module in the last sub-encoder;

[0031] The second splicing module in the second-to-last sub-decoder communicates with the fast pyramid pooling module, and the remaining second splicing modules communicate with all the second C2f network modules in a one-to-one manner;

[0032] All of the spatial feature calibration modules and the third C2f network module in the sub-decoder at the end communicate with the detection head module;

[0033] On the one hand, the decoder structure formed by this scheme is combined with the encoder structure of the previous scheme to form a pyramid convolutional feature network structure with multi-scale feature calibration, which can reduce the loss of shallow features caused by the increase in the sampling depth of the model; on the other hand, by introducing the spatial feature calibration module, deep features can be used to calibrate shallow features, further solving the problems of gradual dilution of feature information and misalignment of spatial features, and enabling the network to adapt to different scale changes.

[0034] In a possible implementation, the downsampling module is a Down downsampling module; using the Down downsampling module in the decoder to improve the downsampling operation can retain the significant features of the region and suppress unimportant details and background, thereby further enhancing the detection capability of small targets and keeping the model lightweight.

[0035] In a possible implementation, the spatial feature calibration module includes a first convolution unit, an upsampling unit, a second convolution unit, a first splicing unit, a first group of convolution units, a third convolution unit, a second group of convolution units, a fourth convolution unit, a first calibration unit, a split unit, a second calibration unit, a first dot product unit, a second dot product unit, a second splicing unit, and a gated mask unit;

[0036] In each of the spatial feature calibration modules:

[0037] The first convolution unit communicates with the third C2f network module in the sub-decoder to which it belongs, the upsampling unit communicates with the first convolution unit, and the second convolution unit communicates with the third C2f network module in the sub-decoder located at the end;

[0038] The first splicing unit communicates with the upsampling unit and the second convolution unit respectively, the third convolution unit communicates with the first splicing unit, the fourth convolution unit communicates with the third convolution unit, the splitting unit communicates with the fourth convolution unit, and the gated mask unit communicates with the splitting unit;

[0039] The first group of convolution units communicate with the upsampling unit and the first splicing unit respectively, the first calibration unit communicates with the first group of convolution units and the splitting unit respectively, and the first point product unit communicates with the first calibration unit and the gated mask unit respectively;

[0040] The second group of convolution units communicates with the second convolution unit and the first splicing unit respectively, the second calibration unit communicates with the second group of convolution units and the splitting unit respectively, and the second point product unit communicates with the second calibration unit and the gated mask unit respectively;

[0041] The second concatenation unit communicates with the first dot product unit and the second dot product unit respectively;

[0042] By setting the first convolution unit and the second convolution unit, deep features and shallow features can be collected in parallel, thereby ensuring that the spatial feature calibration module calibrates the shallow features with the deep features, thereby capturing more deep feature information to alleviate the problem of spatial feature misalignment; at the same time, the spatial feature calibration module with the above structure can achieve spatial feature calibration by dividing the features into multiple groups of sub-features along the channel dimension, transferring these sub-features using a parameter-optimizable downsampling method, and integrating a gating mechanism to adaptively fuse cross-level features.

[0043] In a possible implementation, the spatial feature calibration module is configured to operate by performing the following steps:

[0044] A1: Run the following two processes in parallel:

[0045] (1) performing a convolution operation on an image output by the third C2f network module in the sub-decoder belonging to the same sub-decoder through the first convolution unit to obtain a first image having a preset number of channels, and then performing upsampling processing on the first image through the upsampling unit by bilinear interpolation to obtain a sampled image;

[0046] (2) performing a convolution operation on the calibration image output by the third C2f network module in the last sub-decoder through the second convolution unit to obtain a second image having the same number of channels as the first image;

[0047] A2: Run the following three processes in parallel:

[0048] (1) dividing the channel dimension of the sampled image into a plurality of sub-feature groups by means of the first group of convolution units in a grouped convolution manner to obtain a first feature group map;

[0049] (2) performing image stitching on the sampled image and the second image by the first stitching unit to obtain a stitched image;

[0050] (3) dividing the channel dimension of the second image into a plurality of sub-feature groups by using the second group of convolution units in a grouped convolution manner to obtain a second feature group map;

[0051] A3: performing convolution operations on the stitched image through the third convolution unit and the fourth convolution unit in sequence to obtain a third image;

[0052] A4: The third image is subjected to an offset splitting operation by the splitting unit to obtain an offset image and offset map , and then the offset map and the offset map to the first calibration unit, the second calibration unit and the gated mask unit;

[0053] A5: Run the following four processes in parallel:

[0054] (1) Using the offset map by the first calibration unit and the offset map Adjusting the spatial position of each position on the first feature group image using a feature resampling calibration function to compensate for the spatial misalignment caused by downsampling, thereby obtaining a first calibration image;

[0055] (2) The offset map is processed by the gated mask unit Perform gated mask operation to obtain mask information , the mask information Pass it to the first dot product unit;

[0056] (3) The offset map is processed by the gated mask unit Perform gated mask operation to obtain mask information , the mask information Pass it to the second dot product unit;

[0057] (4) Using the offset map by the second calibration unit and the offset map Adjusting the spatial position of each position on the second feature group image using a feature resampling calibration function to compensate for the spatial misalignment caused by downsampling, thereby obtaining a second calibration image;

[0058] A6: Run the following two processes in parallel;

[0059] (1) The mask information is transformed into Performing a dot product operation on the first calibration image to obtain a first dot product image;

[0060] (2) The mask information is transformed into Performing a dot product operation on the second calibration image to obtain a second dot product image;

[0061] A7: performing image stitching on the first dot product image and the second dot product image by the second stitching unit to obtain the calibration image;

[0062] This scheme achieves spatial feature calibration by dividing image features into multiple groups of sub-features along the channel dimension, transferring these sub-features using an optimizable sampling method, and integrating a gating mechanism to adaptively fuse cross-level features.

[0063] In a possible implementation, the detection head module includes m shared parameter convolution detection heads, and the m shared parameter convolution detection heads communicate with all the spatial feature calibration modules and the third C2f network module in the sub-decoder at the end in a one-to-one manner;

[0064] Each of the shared parameter convolution detection heads performs parameter optimization through a positioning loss function and a classification loss function;

[0065] Each of the shared parameter convolution detection heads includes a second composite convolution module, a third composite convolution module, a fourth composite convolution module and a feature scaling unit which are sequentially arranged along the running direction thereof;

[0066] This scheme can realize parameter sharing. In addition to reducing the number of parameters, it can also solve the problem of imbalance in the number of updateable samples at different levels, thereby significantly improving the detection performance of targets of different scales.

[0067] Another technical solution of the present invention is to provide a target detection method based on multi-scale feature calibration, comprising the following steps:

[0068] S1: Perform m feature extractions on the image under test in a composite convolution manner through the backbone module to obtain m feature images;

[0069] S2: Calibrate the m feature images output by the backbone module by using a calibration module to calibrate shallow features with deep features during feature fusion to obtain m calibration images;

[0070] S3: The m calibration images output by the calibration module are subjected to shared parameter convolution detection and feature scaling respectively by the detection head module to obtain the corresponding detection targets.

[0071] The method disclosed in the present application performs m feature extractions on the tested image in a composite convolution manner through a backbone module to obtain m feature images, thereby achieving the technical effect of adding a high-resolution feature layer to capture the details of small targets; then the m feature images output by the backbone module are calibrated by a calibration module in a manner of calibrating shallow features through deep features, and finally the m calibration images output by the calibration module are respectively subjected to shared parameter convolution detection and feature scaling through a detection head module. This not only achieves sampling of the most representative information from the deep features for each position, reduces the probability of missed detection, and alleviates the problem of losing key details due to spatial dislocation or information dilution during the feature fusion process, but also has better detection effect and is easy to be deployed on resource-constrained edge devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 A schematic diagram of the structure of a target detection system based on multi-scale feature calibration disclosed in an embodiment of the present application;

[0073] Figure 2 A structural schematic diagram of a trunk module when m=2 disclosed in an embodiment of the present application;

[0074] Figure 3 A structural schematic diagram of a trunk module when m=3 disclosed in an embodiment of the present application;

[0075] Figure 4 A structural schematic diagram of a trunk module when m=4 disclosed in an embodiment of the present application;

[0076] Figure 5This is a schematic diagram of the operation flow of the downsampling module disclosed in the embodiments of the present application;

[0077] Figure 6 This is a schematic diagram of the structure of the spatial feature calibration module disclosed in the embodiments of the present application;

[0078] Figure 7 Schematic diagram of the structure of the shared parameter convolution detection head disclosed in the embodiment of the present application;

[0079] Figure 8 It is a flow chart of the method disclosed in the embodiments of this application;

[0080] Fig. 9 This is a comparison diagram of a target detection system based on multi-scale feature calibration disclosed in an embodiment of the present application and several common systems.

[0081] Description of reference numerals:

[0082] 1. Backbone module, 11. Fast pyramid pooling module, 12. First composite convolution module, 13. First C2f network module, 2. Calibration module, 21. Encoder, 211. Upsampling module, 212. First splicing module, 213. Second C2f network module, 22. Decoder, 221. Spatial feature calibration module, 22101. First convolution unit, 22102. Upsampling unit, 22103. Second convolution unit, 22104. First splicing unit, 22105. First group of convolution units, 22106. Third convolution unit, 22107. Second group of convolution units, 22108. Fourth convolution unit, 22109. First calibration Unit, 22110, splitting unit, 22111, second calibration unit, 22112, first dot multiplication unit, 22113, second dot multiplication unit, 22114, second splicing unit, 22115, gated mask unit, 222, downsampling module, 223, second splicing module, 224, third C2f network module, 3, detection head module, 31, shared parameter convolution detection head, 311, second composite convolution module, 312, third composite convolution module, 313, fourth composite convolution module, 314, feature scaling unit. DETAILED DESCRIPTION

[0083] First, those skilled in the art should understand that these implementations are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can make adjustments to them as needed to adapt to specific application scenarios.

[0084] In the embodiments of the present application, unless otherwise clearly specified and limited, the communication or communication connection between the first feature and the second feature refers to the transmission of information between the first feature and the second feature. This information transmission can be either unidirectional or bidirectional, and the communication connection can be achieved by electrical connection of wires, radio connection, electrical connection of electromagnetic media (such as semiconductors), communication achieved by channels, etc.

[0085] In the embodiments of the present application, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0086] The present application is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0087] See also Figure 1 to Figure 5 As shown, the embodiment of the present application discloses a target detection system based on multi-scale feature calibration, and the structural schematic diagram of the system is shown in Figure 1 As shown, the system includes a trunk module 1, a calibration module 2 and a detection head module 3, wherein the trunk module 1 is provided with m output ports, namely P2, P3, ..., Pm+1, the calibration module 2 communicates with the m output ports of the trunk module 1, and the detection head module 3 communicates with the calibration module 2.

[0088] Please continue to see Figure 1 As shown, in this system, the backbone module 1 is configured to perform m feature extractions on the image under test by means of a composite convolution to obtain m feature images, and after each feature extraction of the first (m-1) times, the feature image obtained this time is output and used as the operation object for the next feature extraction, and after the mth feature extraction, the feature image obtained this time is output; the calibration module 2 is configured to calibrate the m feature images output by the backbone module 1 by means of deep feature calibration of shallow feature in the feature fusion process to obtain m calibration images; the detection head module 3 is configured to perform shared parameter convolution detection and feature scaling on the m calibration images output by the calibration module 2 to obtain the corresponding detection targets. Where m is an integer and m≥2, and the specific size of m can be determined according to the number and size of the detection targets.

[0089] See also Figure 2 to Figure 4As shown, in this embodiment, the backbone module 1 is a network structure formed by a fast pyramid pooling (SpatialPyramid Pooling – Fast, referred to as SPPF) module 11, a plurality of first compound convolution modules 12 and no less than m first C2f network modules 13 connected in series, the head of the network structure is a first compound convolution module 12, and the end of the network structure is a fast pyramid pooling module 11. Among them, the (m-1) first C2f network modules 13 communicate with the calibration module 2, and the fast pyramid pooling module 11 communicates with the calibration module 2.

[0090] Figure 2 What is shown is a structural schematic diagram of the backbone module 1 when m=2, which is formed by three first composite convolution modules 12 and two first C2f network modules 13 connected in series. Figure 3 What is shown is a structural schematic diagram of the backbone module 1 when m=3, which is formed by four first composite convolution modules 12 and three first C2f network modules 13 connected in series. Figure 4 What is shown is a structural schematic diagram of the backbone module 1 when m=4, which is formed by five first composite convolution modules 12 and four first C2f network modules 13 connected in series.

[0091] As a preferred embodiment, the following Figure 4 The structure of the backbone module 1 when m=4 is taken as an example for detailed description.

[0092] Please continue to see Figure 1 As shown, in the embodiment of m=4, the calibration module 2 includes an encoder 21 and a decoder 22. The encoder 21 communicates with the three first C2f network modules 13 and the fast pyramid pooling module 11 respectively. Specifically, there are three first C2f network modules 13 in the backbone module 1, each of which is provided with an output port, namely P2, P3 and P4, as shown in FIG. Figure 4 As shown, the encoder 21 communicates with P2, P3 and P4 respectively, and the last output terminal is P5, which is set on the fast pyramid pooling module 11 in the trunk module 1, and the encoder 21 also communicates with P5; the decoder 22 communicates with the encoder 21 and P4 on the fast pyramid pooling module 11 respectively, and the detection head module 3 communicates with the decoder 22.

[0093] In the calibration module 2, the encoder 21 is configured to encode the four feature images output by the backbone module 1 three times by upsampling, splicing and extracting feature maps to obtain three encoded images; the decoder 22 is configured to decode and perform spatial feature calibration on the three encoded images obtained by the encoder 21 and the feature image outputted by the backbone module 1 for the fourth time (i.e., outputted through P5) by combining downsampling, splicing, extracting feature maps and spatial feature calibration to obtain four calibration images.

[0094] Please continue to see Figure 1 As shown, in the embodiment where m=4, the encoder 21 is a network structure formed by three sub-encoders connected in series, and each sub-encoder except the sub-encoder at the end includes an upsampling module 211, a first splicing module 212 and a second C2f network module 213 arranged in sequence along its running direction, and the sub-encoder at the end includes an upsampling module 211 and a first splicing module 212 arranged in sequence along its running direction; the upsampling module 211 in the first sub-encoder communicates with the fast pyramid pooling module 11, and the three first C2f network modules 13 communicating with the encoder 21 communicate with all the first splicing modules 212 in a one-to-one connection manner, and all the second C2f network modules 213 and the first splicing module 212 in the sub-encoder at the end communicate with the decoder 22.

[0095] Please continue to see Figure 1 As shown, in the embodiment of m=4, the decoder 22 is a network structure formed by 4 sub-decoders connected in series; each sub-decoder except the sub-decoder at the end includes a spatial feature calibration module 221 (abbreviated as SFC in the figure), a downsampling module 222, a second splicing module 223 and a third C2f network module 224, and the sub-decoder at the end includes a third C2f network module 224; in each sub-decoder except the sub-decoder at the end: the downsampling module 222 communicates with the third C2f network module 224, and the second splicing module 223 communicates with the downsampling module 222; the spatial feature calibration module 221 in each sub-decoder communicates with the sub-decoder at the end. The third C2f network module 224 in the sub-decoder at the end communicates with each other, and each spatial feature calibration module 221 also communicates with the third C2f network module 224 in the sub-decoder to which it belongs; the third C2f network module 224 in the sub-decoder at the first position communicates with the first splicing module 212 in the sub-encoder at the end; the second splicing module 223 in the sub-decoder at the second-to-last position communicates with the fast pyramid pooling module 11, and the remaining second splicing modules 223 communicate with all the second C2f network modules 213 in a one-to-one manner; all spatial feature calibration modules 221 and the third C2f network module 224 in the sub-decoder at the end communicate with the detection head module 3.

[0096] The downsampling operation of the ordinary downsampling module is often accompanied by a large number of parameters and computational overhead, and is inefficient when processing multi-scale features. In order to solve the above problems, while enhancing the system's ability to detect small targets and maintaining the lightweight characteristics of the system, the downsampling module 222 in this embodiment is a Down downsampling module, and its operation process is as follows Figure 5 shown.

[0097] See also Figure 5 As shown, the Down sampling module in this embodiment is configured to perform the following steps: first, average pooling is performed using an average pooling unit with a size of 2 and a step size of 1 to smooth the feature map output by the second stitching module 223, while removing noise and retaining overall details; then, the feature map after average pooling is divided into two branch images x 1 and x 2 ; Then use the convolution unit with a convolution kernel of 3×3 to convolve the image x 1 Perform convolution operation to obtain the first processing result; for image x 2 Then, the pooling operation is performed using a maximum pooling unit with a size of 3 and a step size of 2, and the convolution operation is performed using a convolution unit with a convolution kernel of 1×1 to obtain the second processing result, so as to retain the significant features of the region and suppress unimportant details and background; finally, the first processing result is concatenated with the second processing result to obtain the down-sampling result. The down-sampling module enables the network to better retain and extract important features, efficiently reduce the spatial dimension, and achieve the purpose of reducing the number of parameters and reducing computational overhead.

[0098] See also Figure 6As shown, in this embodiment, the spatial feature calibration module 221 includes a first convolution unit 22101, an upsampling unit 22102, a second convolution unit 22103, a first splicing unit 22104, a first group of convolution units 22105, a third convolution unit 22106, a second group of convolution units 22107, a fourth convolution unit 22108, a first calibration unit 22109, a split unit 22110, a second calibration unit 22111, a first dot multiplication unit 22112, a second dot multiplication unit 22113, a second splicing unit 22114 and a gated mask unit 22115. In each spatial feature calibration module 221: the first convolution unit 22101 communicates with the third C2f network module 224 in the sub-decoder to which it belongs, the upsampling unit 22102 communicates with the first convolution unit 22101, and the second convolution unit 22103 communicates with the third C2f network module 224 in the sub-decoder located at the end; the first splicing unit 22104 communicates with the upsampling unit 22102 and the second convolution unit 22103 respectively, the third convolution unit 22106 communicates with the first splicing unit 22104, the fourth convolution unit 22108 communicates with the third convolution unit 22106, the splitting unit 22110 communicates with the fourth convolution unit 22108, and the gated mask unit 22115 communicates with the splitting unit 22110; the first group of convolution units 22105 The first calibration unit 22109 communicates with the first group of convolution units 22105 and the splitting unit 22110 respectively, and the first point multiplication unit 22112 communicates with the first calibration unit 22109 and the gated mask unit 22115 respectively; the second group of convolution units 22107 communicates with the second convolution unit 22103 and the first splicing unit 22104 respectively, the second calibration unit 22111 communicates with the second group of convolution units 22107 and the splitting unit 22110 respectively, and the second point multiplication unit 22113 communicates with the second calibration unit 22111 and the gated mask unit 22115 respectively; the second splicing unit 22114 communicates with the first point multiplication unit 22112 and the second point multiplication unit 22113 respectively.

[0099] Please continue to see Figure 6 As shown, in this embodiment, the spatial feature calibration module 221 is configured to run by executing the following steps:

[0100] A1: Run the following two processes in parallel:

[0101] (1) performing a convolution operation on the image output by the third C2f network module 224 in the sub-decoder belonging thereto through the first convolution unit 22101 to obtain a first image with a preset number of channels, and then performing upsampling processing on the first image by bilinear interpolation upsampling through the upsampling unit 22102 to obtain a sampled image;

[0102] (2) The calibration image output by the third C2f network module 224 in the last sub-decoder is convolved by the second convolution unit 22103 to obtain a second image having the same number of channels as the first image.

[0103] A2: Run the following three processes in parallel:

[0104] (1) Dividing the channel dimension of the sampled image into a plurality of sub-feature groups by means of a first group of convolution units 22105 in a grouped convolution manner to obtain a first feature group map;

[0105] (2) stitching the sampled image and the second image together through the first stitching unit 22104 to obtain a stitched image;

[0106] (3) The channel dimension of the second image is divided into a plurality of sub-feature groups by a second group of convolution units 22107 in a grouped convolution manner to obtain a second feature group map.

[0107] A3: The convolution operation is performed on the stitched image through the third convolution unit 22106 and the fourth convolution unit 22108 in sequence to obtain a third image.

[0108] A4: The third image is subjected to an offset splitting operation by the splitting unit 22110 to obtain an offset map and offset map , and then the offset map and offset map Passed to the first calibration unit 22109, the second calibration unit 22111 and the gated mask unit 22115.

[0109] A5: Run the following four processes in parallel:

[0110] (1) Using the offset map by the first calibration unit 22109 and offset map , adjusting the spatial position of each position on the first feature group image by a feature resampling calibration function to compensate for the spatial misalignment caused by downsampling, thereby obtaining a first calibration image;

[0111] (2) The offset map is processed by gated mask unit 22115 Perform gated mask operation to obtain mask information , the mask information Passed to the first dot product unit 22112;

[0112] (3) The offset map is processed by gated mask unit 22115 Perform gated mask operation to obtain mask information , the mask information Passed to the second dot product unit 22113;

[0113] (4) Using the offset map by the second calibration unit 22111 and offset map , the spatial position of each position on the second feature group image is adjusted by the feature resampling calibration function to compensate for the spatial misalignment caused by downsampling, and a second calibration image is obtained.

[0114] The expression of the feature resampling calibration function used in this embodiment is as follows:

[0115] ,

[0116] In the formula,

[0117] Represents the pixel height of the feature group map to be adjusted by the function;

[0118] Represents the pixel width of the feature group map to be adjusted by the function;

[0119] Represents the pixel position in the feature group map that the function is to adjust The characteristic value of

[0120] Representative offset map The pixel position in The offset of

[0121] Representative offset map The pixel position in The offset of

[0122] Represents pixel position The feature resampling calibration value of .

[0123] A6: Run the following two processes in parallel;

[0124] (1) The mask information is converted into Performing a dot product operation with the first calibration image to obtain a first dot product image;

[0125] (2) The mask information is converted into Perform a dot product operation with the second calibration image to obtain a second dot product image.

[0126] A7: The first dot product image and the second dot product image are stitched together by the second stitching unit 22114 to obtain the calibration image.

[0127] The detection head is the key to target detection, but the original detection head of the benchmark model has some limitations, mainly manifested in the large number of parameters of this type of detection head. Each detection head needs to go through two 3×3 convolutions and one 1×1 convolution to extract image information, accounting for 20% of the total network computation. At the same time, the single-scale prediction structure adopted by this type of detection head cannot effectively handle multi-scale targets. It can only predict from one scale of the feature map, ignoring the feature information of other scales. In order to overcome these defects, the detection head module 3 in this embodiment includes m shared parameter convolution detection heads 31, and the shared parameter convolution detection head 31 can enhance the multi-scale perception ability of the detection head with smaller parameter and computational complexity.

[0128] See also Figure 1 and Figure 7 As shown, in the embodiment where m=4, the detection head module 3 includes four shared parameter convolution detection heads 31 (referred to as LSPC detection heads in the figure), and these four shared parameter convolution detection heads 31 communicate with all the spatial feature calibration modules 221 and the third C2f network module 224 in the sub-decoder at the end in a one-to-one manner, and each shared parameter convolution detection head 31 performs parameter optimization through positioning loss function and classification loss function.

[0129] See also Figure 7 As shown, in this embodiment, each shared parameter convolution detection head 31 includes a second compound convolution module 311 with a convolution kernel of 1×1, a third compound convolution module 312 with a convolution kernel of 3×3, a fourth compound convolution module 313 with a convolution kernel of 3×3, and a feature scaling (Scale) unit 314, which are arranged in sequence along its running direction.

[0130] Since the neural network has multi-scale semantic invariance, shared parameters can not only reduce the amount of parameters, but also solve the problem of imbalance in the number of learnable samples at different levels, and significantly improve the detection performance of targets of different scales. For the batch normalization layer (BN layer) in the second composite convolution module 311, the third composite convolution module 312, and the fourth composite convolution module 313, this embodiment adopts a separate calculation method. This is because directly introducing the batch normalization layer in the shared parameter detection head may cause errors in its sliding average. During the forward propagation in the training phase, the batch normalization layer in each shared parameter convolution detection head 31 will independently calculate the mean and variance of the input feature map through the current branch, and then perform subsequent calculations. In addition, in order to solve the problem of inconsistent target scales detected by different detection heads, we use a feature scaling unit 314 to scale the features. By sharing the parameter convolution detection head 31, the multi-scale perception ability of the detection head can be enhanced with a smaller amount of parameters and calculations.

[0131] Next, this embodiment will further disclose an applicable method of the target detection system for multi-scale feature calibration, see Figure 8 As shown, the method comprises the following steps:

[0132] S1: Perform m feature extractions on the image under test in a composite convolution manner through the backbone module 1 to obtain m feature images. Specifically, when m=4, perform 4 feature extractions to obtain 4 feature images, and after each of the first 3 feature extractions, the feature images obtained this time are output and used as the operation object for the next feature extraction, and after the 4th feature extraction, the feature images obtained this time are output.

[0133] S2: Calibrate the m feature images output by the backbone module 1 by using the calibration module 2 to calibrate the shallow features with the deep features to obtain m calibration images. Specifically, when m=4, calibrate the calibration module 2 disclosed in this embodiment to obtain 4 calibration images.

[0134] S3: The m calibration images output by the calibration module 2 are respectively subjected to shared parameter convolution detection and feature scaling by the detection head module 3 to obtain their corresponding detection targets.

[0135] In order to verify the performance of the target detection system based on multi-scale feature calibration disclosed in this embodiment (in order to reduce the text space occupied by drawings and tables, the target detection system based on multi-scale feature calibration is referred to as LMFC-Net in tables and drawings), this embodiment is compared with the currently popular and latest target detection systems. The results of the self-built FactorySafeDet data set are shown in Table 1, where bold represents the optimal result and underline represents the suboptimal result. From the results, it can be seen that the target detection system based on multi-scale feature calibration disclosed in this embodiment has decreased by 76.7% and 24.7% in lightweight indicators, namely parameter indicators and floating-point calculations per second indicators compared with the baseline network system, respectively, which is far superior to other existing systems, and the recall rate and accuracy indicators (mAP@0.5) are improved by 5.9% and 2.7% compared with the baseline network system, significantly reducing the missed detection rate. Comparison of various indicators such as Fig. 9 As shown, the solid line represents the lightweight index, the lower the better, and the dotted line represents the accuracy index, the higher the better. From the above results, it can be seen that the LMFC-Net we proposed has greatly improved in all aspects compared with the existing models or systems.

[0136] Table 1: Performance comparison with other systems on FactorySafeDet dataset

[0137]

[0138] In summary, the target detection system based on multi-scale feature calibration disclosed in this embodiment, by setting the trunk module 1, the calibration module 2 and the detection head module 3, and making the calibration module 2 communicate with the trunk module 1, and the detection head module 3 communicate with the calibration module 2, because the trunk module 1 uses a composite convolution method to extract features of the image under test m times to obtain m feature images, and m can be set according to demand, so that the technical effect of obtaining m scale features and increasing the output of the high-resolution feature layer to capture the details of small targets can be achieved; then the calibration module 2 calibrates the m feature images output by the trunk module 1 by calibrating the shallow features through deep features, which can achieve the most representative information from the deep features for each position, reduce the probability of missed detection, and alleviate the problem of losing key details due to spatial dislocation or information dilution in the feature fusion process. In addition, due to the setting of the detection head module 3, its use of shared parameter convolution can greatly reduce the amount of parameters and calculations, and solve the problem of inconsistent scales of the targets detected by each detection head. Ultimately, while maintaining a small number of parameters and computational complexity, better detection effects are achieved, making the entire system easy to deploy on resource-constrained edge devices.

[0139] In the description of the embodiments of the present application, it should be noted that in the description of the present application, terms such as "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is only for the convenience of description, and does not indicate or imply that the device or component must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present application.

[0140] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "in the present embodiment", "specific example", or "some examples" etc. means that the specific features, mechanisms, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0141] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An object detection system based on multi-scale feature calibration, characterized in that: include: The main module (1) is configured to perform m feature extractions on the image under test in a composite convolution manner to obtain m feature images, and after each feature extraction of the first (m-1) times, the feature image obtained this time is output and used as the operation object for the next feature extraction, and after the mth feature extraction, the feature image obtained this time is output; The calibration module (2) is configured to calibrate the m feature images output by the backbone module (1) by calibrating shallow features with deep features during feature fusion, thereby obtaining m calibration images; The detection head module (3) is configured to perform shared parameter convolution detection and feature scaling on the m calibration images output by the calibration module (2) to obtain the corresponding detection targets; in: m≥2； The calibration module (2) communicates with the trunk module (1), and the detection head module (3) communicates with the calibration module (2); The backbone module (1) is a network structure formed by serially connecting one fast pyramid pooling module (11), a plurality of first composite convolution modules (12), and no less than m first C2f network modules (13), wherein the first portion of the network structure is one of the first composite convolution modules (12), and the last portion of the network structure is one of the fast pyramid pooling modules (11); wherein (m-1) of the first C2f network modules (13) communicate with the calibration module (2), and the fast pyramid pooling module (11) communicates with the calibration module (2); The calibration module (2) comprises: The encoder (21) is configured to encode the m feature images output by the backbone module (1) (m-1) times by upsampling, splicing and extracting feature images, thereby obtaining (m-1) encoded images; The decoder (22) is configured to decode and calibrate the spatial features of the (m-1) coded images obtained by the encoder (21) and the feature image outputted by the backbone module (1) for the mth time by combining downsampling, splicing, feature map extraction and spatial feature calibration, so as to obtain m calibrated images; The encoder (21) communicates with (m-1) first C2f network modules (13) and the fast pyramid pooling module (11) respectively, the decoder (22) communicates with the encoder (21) and the fast pyramid pooling module (11) respectively, and the detection head module (3) communicates with the decoder (22); The decoder (22) is a network structure formed by connecting m sub-decoders in series; Each of the sub-decoders except the sub-decoder at the end includes a spatial feature calibration module (221), a downsampling module (222), a second splicing module (223) and a third C2f network module (224), and the sub-decoder at the end includes the third C2f network module (224); The detection head module (3) comprises m shared parameter convolution detection heads (31), and the m shared parameter convolution detection heads (31) communicate with all the spatial feature calibration modules (221) and the third C2f network module (224) in the sub-decoder located at the end in a one-to-one manner; Each of the shared parameter convolution detection heads (31) is subjected to parameter optimization through a positioning loss function and a classification loss function; Each of the shared parameter convolution detection heads (31) comprises a second composite convolution module (311), a third composite convolution module (312), a fourth composite convolution module (313) and a feature scaling unit (314) which are sequentially arranged along the running direction thereof.

2. The target detection system based on multi-scale feature calibration according to claim 1, characterized in that: The encoder (21) is a network structure formed by (m-1) sub-encoders connected in series, each of the sub-encoders except the sub-encoder at the end comprises an up-sampling module (211), a first splicing module (212) and a second C2f network module (213) arranged in sequence along its running direction, and the sub-encoder at the end comprises an up-sampling module (211) and a first splicing module (212) arranged in sequence along its running direction; The upsampling module (211) in the first sub-encoder communicates with the fast pyramid pooling module (11), the (m-1) first C2f network modules (13) communicating with the encoder (21) communicate with all the first splicing modules (212) in a one-to-one connection manner, and all the second C2f network modules (213) and the first splicing module (212) in the last sub-encoder communicate with the decoder (22).

3. The target detection system based on multi-scale feature calibration according to claim 2, characterized in that: In each of the sub-decoders except the sub-decoder at the end: the down-sampling module (222) communicates with the third C2f network module (224), and the second splicing module (223) communicates with the down-sampling module (222); The spatial feature calibration module (221) in each of the sub-decoders communicates with the third C2f network module (224) in the sub-decoder located at the end, and each of the spatial feature calibration modules (221) also communicates with the third C2f network module (224) in the sub-decoder to which it belongs; The third C2f network module (224) in the first sub-decoder communicates with the first splicing module (212) in the last sub-encoder; The second splicing module (223) in the sub-decoder at the second-to-last position communicates with the fast pyramid pooling module (11), and the remaining second splicing modules (223) communicate with all the second C2f network modules (213) in a one-to-one manner; All of the spatial feature calibration modules (221) and the third C2f network module (224) in the sub-decoder located at the end communicate with the detection head module (3).

4. The target detection system based on multi-scale feature calibration according to claim 3, characterized in that: The down sampling module (222) is a Down down sampling module.

5. The target detection system based on multi-scale feature calibration according to claim 4, characterized in that: Each of the spatial feature calibration modules (221) comprises a first convolution unit (22101), an upsampling unit (22102), a second convolution unit (22103), a first splicing unit (22104), a first group of convolution units (22105), a third convolution unit (22106), a second group of convolution units (22107), a fourth convolution unit (22108), a first calibration unit (22109), a splitting unit (22110), a second calibration unit (22111), a first dot product unit (22112), a second dot product unit (22113), a second splicing unit (22114) and a gated mask unit (22115); In each of the spatial feature calibration modules (221): The first convolution unit (22101) communicates with the third C2f network module (224) in the sub-decoder to which it belongs, the upsampling unit (22102) communicates with the first convolution unit (22101), and the second convolution unit (22103) communicates with the third C2f network module (224) in the sub-decoder located at the end; The first splicing unit (22104) communicates with the up-sampling unit (22102) and the second convolution unit (22103) respectively, the third convolution unit (22106) communicates with the first splicing unit (22104), the fourth convolution unit (22108) communicates with the third convolution unit (22106), the splitting unit (22110) communicates with the fourth convolution unit (22108), and the gated mask unit (22115) communicates with the splitting unit (22110); The first group of convolution units (22105) communicates with the upsampling unit (22102) and the first splicing unit (22104) respectively, the first calibration unit (22109) communicates with the first group of convolution units (22105) and the splitting unit (22110) respectively, and the first point product unit (22112) communicates with the first calibration unit (22109) and the gated mask unit (22115) respectively; The second group of convolution units (22107) communicates with the second convolution unit (22103) and the first splicing unit (22104) respectively, the second calibration unit (22111) communicates with the second group of convolution units (22107) and the splitting unit (22110) respectively, and the second point product unit (22113) communicates with the second calibration unit (22111) and the gated mask unit (22115) respectively; The second splicing unit (22114) communicates with the first dot product unit (22112) and the second dot product unit (22113) respectively.

6. The target detection system based on multi-scale feature calibration according to claim 5, characterized in that: Each of the spatial feature calibration modules (221) is configured to operate by executing the following steps: A1: Run the following two processes in parallel: (1) performing a convolution operation on an image output by the third C2f network module (224) in the sub-decoder belonging thereto through the first convolution unit (22101) to obtain a first image having a preset number of channels, and then performing upsampling processing on the first image by bilinear interpolation upsampling through the upsampling unit (22102) to obtain a sampled image; (2) performing a convolution operation on the calibration image output by the third C2f network module (224) in the sub-decoder located at the end through the second convolution unit (22103) to obtain a second image having the same number of channels as the first image; A2: Run the following three processes in parallel: (1) dividing the channel dimension of the sampled image into a plurality of sub-feature groups by means of the first group of convolution units (22105) in a grouped convolution manner, thereby obtaining a first feature group map; (2) performing image stitching on the sampled image and the second image by the first stitching unit (22104) to obtain a stitched image; (3) dividing the channel dimension of the second image into a plurality of sub-feature groups by means of the second group of convolution units (22107) in a grouped convolution manner, thereby obtaining a second feature group map; A3: sequentially performing convolution operations on the spliced ​​image through the third convolution unit (22106) and the fourth convolution unit (22108) to obtain a third image; A4: The third image is subjected to an offset splitting operation by the splitting unit (22110) to obtain an offset image and offset map , and then the offset map and the offset map Transmitted to the first calibration unit (22109), the second calibration unit (22111) and the gated mask unit (22115); A5: Run the following four processes in parallel: (1) Using the offset map by the first calibration unit (22109) and the offset map , adjusting the spatial position of each position on the first feature group image by a feature resampling calibration function to compensate for the spatial misalignment caused by downsampling, thereby obtaining a first calibration image; (2) The offset map is processed by the gated mask unit (22115) Perform gated mask operation to obtain mask information , the mask information Passed to the first dot product unit (22112); (3) The offset map is processed by the gated mask unit (22115) Perform gated mask operation to obtain mask information , the mask information Passed to the second dot product unit (22113); (4) Using the offset map by the second calibration unit (22111) and the offset map Adjusting the spatial position of each position on the second feature group image using a feature resampling calibration function to compensate for the spatial misalignment caused by downsampling, thereby obtaining a second calibration image; A6: Run the following two processes in parallel; (1) The mask information is transformed into Performing a dot product operation on the first calibration image to obtain a first dot product image; (2) The mask information is transformed into Performing a dot product operation on the second calibration image to obtain a second dot product image; A7: The first dot product image and the second dot product image are stitched together by the second stitching unit (22114) to obtain the calibration image.

7. A target detection method based on multi-scale feature calibration, characterized in that: The target detection system based on multi-scale feature calibration according to any one of claims 1 to 6 comprises the following steps: S1: performing m feature extractions on the image under test in a composite convolution manner through the backbone module (1) to obtain m feature images; S2: The calibration module (2) calibrates the m feature images output by the backbone module (1) in a feature fusion process by calibrating shallow features with deep features to obtain m calibration images; S3: The m calibration images output by the calibration module (2) are respectively subjected to shared parameter convolution detection and feature scaling by the detection head module (3) to obtain the corresponding detection targets.

Images