A target detection method based on multi-scale feature fusion

Abstract

The present invention specifically relates to a target detection method based on multi-scale feature fusion, including: constructing a target detection model and training it; obtaining image data of the target to be detected; inputting the image data of the target to be detected into the trained target detection model; first extracting a multi-scale feature map based on the image data; then fully fusing the low-level feature map by layer-by-layer feature fusion and feedback feature fusion to obtain a corresponding low-level fusion feature map; then fusing and updating only the high-level feature map by an attention mechanism to obtain a corresponding high-level fusion feature map; finally generating a corresponding multi-scale fusion feature map based on the low-level fusion feature map and the high-level fusion feature map; completing target detection based on the multi-scale fusion feature map. The target detection method based on multi-scale feature fusion in the present invention can fully fuse scale features and ensure the detection effect of small targets, thereby improving the accuracy and effect of target detection.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a target detection method based on multi-scale feature fusion. Background Art

[0002] Object detection technology is a core research direction in the field of computer vision, which aims to obtain the classification and location of the object of interest in the image. Object detection technology is not only the research basis of many computer vision tasks such as target tracking and semantic segmentation, but also widely used in various civil and military fields such as medical diagnosis, autonomous driving, intelligent video surveillance, and military target monitoring. With the diversification and complexity of application scenarios, an image to be detected will contain multiple objects of different scales, which makes the current object detection task face severe challenges brought by scale differences. Therefore, multi-scale object detection has become one of the research hotspots in the field of object detection.

[0003] In response to the problem of low feature information utilization in existing target detection methods, a Chinese patent with publication number CN113361528A discloses "A multi-scale target detection method and system", which includes: constructing a dilated pyramid network model including multiple convolution modules and multiple convolution branches connected in sequence; the output of each convolution module is respectively connected to a convolution branch, the convolution branch includes a convolution operation and multiple dilated convolution operations, and the convolution operation and the dilated convolution operation in the convolution branch are in a parallel relationship; the output of the convolution branch is sequentially upsampled from low to high resolution, and then element-wise added with adjacent first feature maps of the same size to obtain multiple fused feature maps; the dilated pyramid network model is trained according to the target detection data set to obtain a target detection model.

[0004] The target detection method in the above-mentioned existing scheme extracts feature information through the hole convolution of different convolution kernels, so that the convolution layer has a receptive field of different sizes, which helps the single feature layer to obtain richer multi-scale context feature information, thereby enhancing the sensitivity of each feature layer to targets of different scales. In actual target detection, the image data is input into the trained target detection model, and the target detection model obtains the corresponding multiple fused feature maps, and then completes the target detection based on the multiple fused feature maps. Therefore, the fused feature map determines the effect of target detection to a certain extent. However, when obtaining the fused feature map, most of the technical means of the existing scheme remain at the improvement of the backbone network and the improvement of the convolution method at the micro level, and no effective improvement is made on the multi-scale feature fusion method of the feature pyramid at the macro level, and no relevant means are used to refine and balance the multi-scale semantic features. Although it can extract multi-scale feature information, it is difficult to fully fuse the multi-scale feature information, especially for small targets (small-scale features). The target detection effect will be limited, which makes it difficult to ensure the accuracy of target detection. Therefore, how to design a target detection method that can fully integrate scale features and ensure small target detection effect is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is: how to provide a target detection method based on multi-scale feature fusion so as to fully integrate scale features and ensure the small target detection effect, thereby improving the accuracy and effect of target detection.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A target detection method based on multi-scale feature fusion includes the following steps:

[0008] S1: Build a target detection model and train the target detection model;

[0009] S2: Acquire image data of the target to be detected;

[0010] S3: Input the image data of the target to be detected into the trained target detection model;

[0011] The target detection model first extracts multi-scale feature maps based on image data; then fully fuses the low-level feature maps through layer-by-layer feature fusion and feedback feature fusion to obtain the corresponding low-level fusion feature maps; then only fuses and updates the high-level feature maps through the attention mechanism to obtain the corresponding high-level fusion feature maps; finally, the corresponding multi-scale fusion feature maps are generated based on the low-level fusion feature maps and the high-level fusion feature maps;

[0012] S4: Complete target detection based on multi-scale fusion feature maps.

[0013] Preferably, the target detection model includes an FPN network, an FFE network and an H-BFP network;

[0014] The FPN network is used to extract multi-scale feature maps based on image data, and then perform feature fusion layer by layer from the highest level downwards, and output high-level feature maps without feature fusion, as well as preliminary fused feature maps obtained by feature fusion of low-level feature maps;

[0015] The FFE network is used to extract multi-scale feature maps based on image data, and then perform layer-by-layer feature fusion and feedback feature fusion on the feature maps of the corresponding scales with the preliminary fusion feature maps output by the FPN network to generate the corresponding low-level fusion feature maps;

[0016] The H-BFP network is used to add and average the high-level feature map and the low-level fusion feature map, and extract the corresponding feature information; then the high-level feature map is fused and updated based on the feature information and the attention mechanism to obtain the corresponding high-level fusion feature map.

[0017] Preferably, the FFE network includes a feedback feature input module; the feedback feature input module obtains the preliminary fusion feature map output by the FPN network, and implements the input of the preliminary fusion feature map through two branches;

[0018] Branch 1 is used to input the preliminary fusion feature map as feedback information into the FFE network, so that the preliminary fusion feature map can be fused layer by layer with the feature map of the corresponding scale to output the corresponding deep fusion feature map;

[0019] Branch two is used to input the preliminary fusion feature map to the output position of the FFE network, so that the preliminary fusion feature can be feedback-fused with the deep fusion feature map output by the FFE network to generate the corresponding low-level fusion feature map.

[0020] Preferably, when the preliminary fusion feature map is fused layer by layer with the feature map of the corresponding scale, the output of the FFE network is adjusted through a 1*1 convolutional layer, and the corresponding channel size is adjusted to 256 to output the corresponding deep fusion feature map.

[0021] Preferably, the FFE network includes a global module; the global module is used to perform global feature extraction on the input preliminary fusion feature map to obtain richer feature semantic information in the preliminary fusion feature map.

[0022] Preferably, the H-BFP network first performs an operation on the high-level feature map P output by the FPN network. h Downsample to obtain the corresponding high-level feature map P h+1; Then the high-level feature map P h and P h+1 The low-level fusion feature map output by the FFE network is added and averaged to obtain the feature information after fusion of each layer; then the feature information is subjected to no local operation, and the high-level feature map P is obtained by sampling the feature information. h and P h+1 High-level fusion feature map P of corresponding scale h ′ and P′ h+1 ; Then fusion feature map P h ′ and P′ h+1 Perform convolution and activation function operations to calculate the corresponding attention feature maps respectively; finally, based on the high-level fusion feature map P h ′ and P′ h+1 The corresponding attention feature maps are calculated respectively h With P h ′, and P h+1 and P′ h+1 Weighted sum, and then update the high-level fusion feature map P h ′ and P′ h+1 .

[0023] Preferably, the high-level feature map is updated by the following formula:

[0024] P′ h =torch.sigmoid(self.conv(P′ h ))*P′ h +(1-torch.sigmoid(self.conv(P′ h )))*P h ;

[0025] P′ h+1 =torch.sigmoid(self.conv(P′ h+1 ))*P′ h+1 +(1-torch.sigmoid(self.conv(P′ h+1 )))*P h+1 ;

[0026] In the above formula: P h ′、P′ h+1 represents the high-level fusion feature map, P h , P h+1 Represents a high-level feature map, self.conv(P′ h ) indicates P h 'After convolution operation; self.conv(P h ′1) indicates P h'1 after convolution operation; torch.sigmoid(self.conv(P h ′)) indicates that the high-level fusion feature map P is not updated h ′ when calculating the attention feature map; torch.sigmoid(self.conv(P′ h +1)) indicates that the high-level fusion feature map P′ is not updated h+1 The attention feature map calculated at S

[0027] Preferably, the target detection model is trained by a dynamic balanced loss function; the dynamic balanced loss function introduces a dynamic strategy based on the balanced L1 loss function, so that dynamic label allocation can be achieved and the shape of the loss function can be adjusted.

[0028] Preferably, dynamic label allocation is expressed by the following formula:

[0029]

[0030] In the above formula: Label represents the label, T now Indicates the current threshold;

[0031] During training, we first calculate the IoUs between the proposed box and its true box to get set I, and then select the Kth maximum value from set I to update T now The threshold value.

[0032] Preferably, the dynamic balance loss function is expressed by the following formula:

[0033]

[0034] In the above formula: L b represents the dynamic balance loss, α, b, and γ are mutually constrained by the formula αln(b+1)=γ, α=0.5, γ=1.5; during training, the regression label E between the proposal box and its true box is first calculated, and then the Kth minimum value is selected from the regression label E to update β now ;

[0035] By fine-tuning the factor _tune The shape of the loss function is fine-tuned to eliminate the influence of derivative discontinuity. The formula of the fine-tuning factor is as follows:

[0036] Among them, fine _tune <γ.

[0037] Compared with the prior art, the target detection method in the present invention has the following beneficial effects:

[0038] 1. The present invention fully fuses the low-level feature maps through layer-by-layer feature fusion and feedback feature fusion, so that the semantic information of the low-level feature maps can be enhanced, and the multi-scale semantic features of the low-level feature maps can be refined and balanced, thereby generating a multi-scale fused feature map with high resolution and high semantic information, thereby improving the detection effect of small targets and small-scale features.

[0039] 2. The present invention fuses and updates high-level feature maps through the attention mechanism, so that the fusion scale of the high-level feature maps can be consistent with the low-level feature maps, and thus a multi-scale fusion feature map can be better obtained to ensure the detection performance of the model, thereby improving the accuracy and effect of target detection.

[0040] 3. The present invention can realize dynamic label allocation and adjust the shape of the loss function by training the target detection model through a dynamic balance loss function, so as to solve the inconsistency problem between fixed settings and dynamic training, and then ensure the consistency of fixed settings and dynamic training, thereby improving the training effect of the target detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0042] Figure 1 It is the logic block diagram of the target detection method;

[0043] Figure 2 This is the network structure diagram of the target detection model;

[0044] Figure 3 This is the network structure diagram of the global module;

[0045] Figure 4 It is the network structure diagram of H-BFP network;

[0046] Figure 5 To dynamically balance the gradient image and loss image of the loss function under different β value settings, "gradient" in the figure represents the gradient and "Smooth L1Loss" represents the smooth L1Loss curve;

[0047] Figure 6 Qualitative results comparison of FPN and MT-FPN. DETAILED DESCRIPTION

[0048] The following is a further detailed description through specific implementation methods:

[0049] Example:

[0050] This embodiment discloses a target detection method based on multi-scale feature fusion.

[0051] like Figure 1 As shown, the target detection method based on multi-scale feature fusion includes the following steps:

[0052] S1: Build a target detection model and train the target detection model;

[0053] S2: Acquire image data of the target to be detected;

[0054] S3: Input the image data of the target to be detected into the trained target detection model;

[0055] The target detection model first extracts multi-scale feature maps based on image data; then fully fuses the low-level feature maps through layer-by-layer feature fusion and feedback feature fusion to obtain the corresponding low-level fused feature maps; then only fuses and updates the high-level feature maps through the attention mechanism to obtain the corresponding high-level fused feature maps; finally, the corresponding multi-scale fused feature maps are generated based on the low-level fused feature maps and the high-level fused feature maps.

[0056] Among them, the low-level feature map and the high-level feature map are relative concepts, and the low-level feature map and the high-level feature map can be defined as needed when used. In this embodiment, the feature map of the highest level is defined as the high-level feature map, and the feature maps other than the highest-level feature map are defined as low-level feature maps. At the same time, the low-level fusion feature map and the high-level fusion feature map are combined to obtain the corresponding multi-scale fusion feature map.

[0057] S4: Target detection based on multi-scale fusion feature map. The solution of the present invention focuses on the generation and acquisition of multi-scale fusion feature map. Target detection based on multi-scale fusion feature map is a mature method in the prior art, which will not be described in detail here. Please refer to the relevant content disclosed in the Chinese patent "A multi-scale target detection method and system" with publication number CN113361528A.

[0058] The present invention fully fuses the low-level feature maps by layer-by-layer feature fusion and feedback feature fusion, so that the semantic information of the low-level feature maps can be enhanced, and the multi-scale semantic features of the low-level feature maps can be refined and balanced, thereby generating a multi-scale fused feature map with high resolution and high semantic information, thereby improving the detection effect of small targets and small-scale features. At the same time, the present invention fuses and updates the high-level feature maps through the attention mechanism, so that the fusion scale of the high-level feature maps can be consistent with the low-level feature maps, and then the multi-scale fused feature maps can be better obtained to ensure the detection performance of the model, thereby improving the accuracy and effect of target detection.

[0059] In the specific implementation process, Figure 2As shown, the target detection model (MT-FPN) includes FPN (feature pyramid) network, FFE (feedback feature enhancement) network and H-BFP (high-level balanced feature pyramid) network.

[0060] In this embodiment, a “fourth-order” feature pyramid network is taken as an example for description.

[0061] The FPN network is used to extract multi-scale feature maps based on image data, and then perform feature fusion layer by layer from the highest level downward to output a high-level feature map P5 without feature fusion (this embodiment only defines one high-level feature map, and other preferred embodiments can define multiple high-level feature maps as needed), and a preliminary fused feature map {P2, P3, P4} after feature fusion of the low-level feature maps (this embodiment defines other feature maps except the high-level feature map P5 as low-level feature maps).

[0062] The FFE network is used to extract multi-scale feature maps based on image data, and then perform layer-by-layer feature fusion and feedback feature fusion on the feature maps of the corresponding scales with the preliminary fused feature maps {P2, P3, P4} output by the FPN network to generate the corresponding low-level fused feature maps {P2′, P3′, P4′}.

[0063] The H-BFP network is used to add and average the high-level feature map P5 and its upsampled high-level feature map P6 and the low-level fusion feature map {P2′, P3′, P4′}, and extract the corresponding feature information. Then, based on the feature information and attention mechanism, only the high-level feature maps P5 and P6 are fused and updated to obtain the corresponding high-level fusion feature maps P5′ and P6′.

[0064] The FFE network includes a feedback feature input module; the feedback feature input module obtains the preliminary fusion feature map {P2, P3, P4} output by the FPN network, and realizes the input of the preliminary fusion feature map through two branches;

[0065] Branch one is used to input the preliminary fusion feature map {P2, P3, P4} as feedback information into the FFE network, so that the preliminary fusion feature map {P2, P3, P4} can be fused layer by layer with the feature map of the corresponding scale to output the corresponding deep fusion feature map; when the preliminary fusion feature map is fused layer by layer with the feature map of the corresponding scale, the output of the FFE network is adjusted through a 1*1 convolutional layer, and the corresponding channel size is adjusted to 256 to output the corresponding deep fusion feature map.

[0066] Branch two is used to input the preliminary fusion feature map {P2, P3, P4} to the output position of the FFE network, so that the preliminary fusion feature {P2, P3, P4} can be feedback-fused with the deep fusion feature map output by the FFE network to generate the corresponding low-level fusion feature map {P2′, P3′, P4′}.

[0067] The FFE network includes global modules; combined Figure 3 As shown, the global module is used to perform global feature extraction on the input preliminary fusion feature map to obtain richer feature semantic information in the preliminary fusion feature map.

[0068] The present invention performs global feature extraction on the preliminary fusion feature map through a global module, and can obtain richer feature semantic information in the high-level preliminary fusion feature map, thereby solving the problem of low resolution of the high-level preliminary fusion feature map, thereby helping to balance the detail information of the low-level preliminary fusion feature map.

[0069] In the specific implementation process, Figure 4 As shown in the figure, Integrate and Refine represent the Integrate strategy and Refine strategy of BFP in Libra R-CNN respectively), the H-BFP network first downsamples the high-level feature map P5 output by the FPN network to obtain the corresponding high-level feature map P6; then the high-level feature maps P5 and P6 are added and averaged with the low-level fusion feature maps output by the FFE network to obtain the feature information after fusion of each layer; then the feature information is subjected to no local operation, and high-level fusion feature maps P5′ and P6′ of corresponding scales to the high-level feature maps P5 and P6 are obtained by sampling the feature information; then convolution and activation function operations are performed on the high-level fusion feature maps P5′ and P6′, and the corresponding attention feature maps are calculated respectively; finally, based on the attention feature maps corresponding to the high-level fusion feature maps P5′ and P6′, the weighted sum of P5 and P5′ and the weighted sum of P6 and P6′ are calculated respectively, and then the high-level fusion feature maps P5′ and P6′ are updated.

[0070] Preferably, the high-level feature map is updated by the following formula:

[0071] P5′=torch.sigmoid(self.conv(P5′))*P5′+(1-torch.sigmoid(self.conv(P5′)))*P5;

[0072] P6′=torch.sigmoid(self.conv(P6′))*P6′+(1-torch.sigmoid(self.conv(P6′)))*P6;

[0073] In the above formulas: P5′ and P6′ represent high-level fusion feature maps, P5 and P6 represent high-level feature maps, self.conv(p′5) represents P5′ after convolution operation; self.conv(p′6) represents P6′ after convolution operation; torch.sigmoid(self.conv(p5′)) represents the attention feature map calculated when the high-level fusion feature map P5′ is not updated. The attention feature map is used to calculate the weighted sum of P5 and P5′, and the result covers the original P5′ as the output of the H-BFP network; torch.sigmoid(self.conv(p6′)) represents the attention feature map calculated when the high-level fusion feature map P6′ is not updated. The attention feature map is used to calculate the weighted sum of P6 and P6′, and the result covers the original P6′ as the output of the H-BFP network.

[0074] During the specific implementation process, the target detection model is trained by a dynamic balanced loss function; the dynamic balanced loss function introduces a dynamic strategy based on the balanced L1 loss function, so that dynamic label allocation can be achieved and the shape of the loss function can be adjusted.

[0075] Dynamic label allocation is expressed as follows:

[0076]

[0077] In the above formula: Label represents the label, T now Indicates the current threshold;

[0078] During training, we first calculate the IoUs between the proposed box and its true box to get set I, and then select the Kth maximum value from set I to update T now The threshold value.

[0079] The balanced L1 loss function is expressed as follows:

[0080]

[0081] The dynamic balance loss function is expressed by the following formula:

[0082]

[0083] In the above formula: L b represents the dynamic balance loss. α, b, and γ are mutually constrained by the formula αln(b+1)=γ. The optimal values ​​of α and γ are α=0.5 and γ=1.5. During training, the regression label E between the proposed box and its true box is first calculated, and then the Kth minimum value is selected from the regression label E to update β. now ;

[0084] The shape of the loss function is adjusted by fine-tuning the factor. The formula of the fine-tuning factor is as follows:

[0085] Among them, fine _tune <γ.

[0086] This embodiment finds the left and right limits of x=β on the derivative function. If the left limit is equal to the right limit and equal to L b The derivative value of , then the derivative function is continuous at x = β. In fact, the derivative function is discontinuous. Figure 5 As shown in (a), the derivative function graphs corresponding to different β values ​​are discontinuous when x = β. As the β value decreases, this problem becomes more prominent. In order to eliminate the negative impact of the derivative discontinuity, L b The derivative function graph of is fitted into a smooth and continuous pattern.

[0087] After adding the fine-tuning factor, Figure 5 (b) and Figure 5 (c) shows the loss image and gradient image of the dynamic equilibrium loss function under different β value settings. As the β value decreases, the gradient tends to saturate faster, so smaller errors contribute more to the training of the model and reduce the negative impact caused by derivative discontinuity.

[0088] The present invention trains the target detection model by dynamically balancing the loss function, which can realize dynamic label allocation and adjust the shape of the loss function, so as to solve the inconsistency problem between fixed settings and dynamic training, and further ensure the consistency of fixed settings and dynamic training, thereby improving the training effect of the target detection model.

[0089] In order to better illustrate the advantages of the target detection method of the present invention, the following experiment is disclosed in this embodiment.

[0090] 1. Datasets and Evaluation Metrics

[0091] All experiments were performed on the MS COCO detection dataset with 80 categories. MS COCO contains 115k images for training (train2017), 5k images for validation (val2017), and 20k images for testing (test-dev). In addition, the test-dev labels are not publicly released. We trained the model on train2017 and reported the results of the ablation study on val2017. We submitted the final results to the evaluation server of test-dev for comparison. The performance metrics follow the standard coco-style mean average accuracy (mAP) metric at different IoU thresholds, ranging from 0.5 to 0.95 with an interval of 0.05.

[0092] 2. Experimental Equipment

[0093] This experiment is based on the existing mmdetection implementation. mmdetection has been upgraded to v2.0 with higher baseline performance than v1.0. Performance improvement compared to the baseline has become more difficult. Therefore, the baseline was re-implemented on mmdetection v2.0 for fair comparison. The detector with resolution (1333,800) was trained and tested on 1 NVIDIA Tesla V100GPU (2 images per GPU). The model was trained for 12 batches with an initial learning rate of 0.0025. The model uses a weight decay of 0.0001 and a momentum of 0.9. The proposed MT-FPN can be applied to any FPN-based detector.

[0094] Faster R-CNN and Libra R-CNN were selected as baseline detectors. In the MT-FPN comparison test, the backbone networks were ResNet-50 and ResNet-101. In another comparison test, the baseline was equipped with MT-FPN and DBLL applied in combination. In addition, this experiment adds ResNext101-64x4d as a backbone network for comparison with SOTA detectors. Since ResNext101-64x4d takes too long to train as a backbone network, in order to save time, the baseline for the experiment to test the effect of combined application is only Faster R-CNN, and Libra R-CNN is ignored as a baseline. The latter is used for ablation experiments. The dimension of the feature pyramid channel is set to 256, and other settings follow the basic framework. All other hyperparameters in this experiment follow mmdetection.

[0095] 3. Experimental Results

[0096] 1) Results of the target detection model (MT-FPN) of the present invention

[0097] In order to verify the effectiveness of the proposed method in improving performance, MT-FPN was evaluated on COCO test-dev. As shown in Table 1, when using ResNet-50, Faster R-CNN and Libra RCNN achieved an average precision (AP) of 42.5 and 42.4 respectively by replacing FPN with MT-FPN. When using ResNet-101, Faster R-CNN based on MT-FPN achieved 43.6AP, which exceeded the gains brought by the most advanced FPN-based methods. MT-FPN has a significant improvement in AP. S The performance of the indicators shows that the proposed method (Ours) can effectively solve the problem of low accuracy in small-scale target detection.

[0098] Therefore, the MT-FPN of the present invention is effective.

[0099] Table 1 Comparison of baseline methods and the proposed method on COCO TEST-DEV

[0100]

[0101] (In Table 1, the symbol “*” indicates the result re-implemented by MMDETECTION)

[0102] 2) Results of the target detection model (MT-FPN) and the dynamic balance loss function (DBLL) of the present invention

[0103] In Table 2, the results of the combined application of MT-FPN and DBLL are shown. The table is divided into 4 groups. The first group shows the single-stage detectors. The second group shows the two-stage detectors. The third group is Faster R-CNN, which is the baseline of the experiment. The fourth group is the results of the experiment. When evaluating the effect of the combined application of MT-FPN and DBLL, Faster R-CNN uses ResNet-50 and ResNext101-64x4d as the backbone. The application of this method brings an improvement of 6.8 and 2.7 points compared to the baseline. Considering the higher complexity of the detectors in Table 2, this experiment uses the more basic Faster R-CNN as the baseline. The experimental results show that the performance obtained by the proposed method (Ours) exceeds most of the previous state-of-the-art detectors on the MS COCO benchmark, which is achieved by adding MT-FPN and DBLL. From the perspective of average accuracy and inference speed, the proposed method achieves competitive performance compared to the state-of-the-art detectors.

[0104] Table 2 Comparison of the combined application of MT-FPN and DBLL on the MS COCO benchmark

[0105]

[0106] 3) Visualization and comparison of qualitative results

[0107] Figure 6A comparison of qualitative results between FPN and MT-FPN is shown. It can be seen that MT-FPN produces satisfactory results for small, medium, and large objects, while typical FPN produces poor results. The typical FPN model occasionally misses some objects because these objects may be too small or outside the receptive field. It may also locate wrong and aliased objects. MT-FPN has higher discriminability and better performance by using various techniques such as feedback information, global modules, attention mechanism, and fine information fusion to refine and balance multi-scale semantic features. Both models are built on Faster R-CNN with ResNet-50 and 1× schedule. The images are selected from COCO val-2017. We compare the detection performance under a threshold of 0.5.

[0108] 4. Ablation Experiment

[0109] This experiment analyzes the impact of each proposed component of MT-FPN on the COCO val-2017 subset. The ablation results are reported in Table 3. We gradually add FFE and H-BFP on libra-rcnn with a ResNet-50 backbone. The dynamic balance loss (DBLL) and fine-tuning factor (fine _tune ) is based on the DLA in Table IV. The training process follows the 1× schedule (12 EPOCH). The ablation experiments are performed using the same settings for fair comparison. Because DBLL is proposed based on Balanced L1Loss. Therefore, all ablation experiments use libra-rcnn as the baseline.

[0110] 1) Ablation experiment on MT-FPN

[0111] The original BFP in Libra-rcnn balances the semantic information of each layer, while H-BFP only balances and refines the semantic information of high-level features, so replacing BFP with H-BFP will naturally affect the performance. The experimental results prove our conjecture. H-BFP reduces AP from 38.6 to 38.5. However, the proposal of H-BFP is based on the proposal of FFE. Table 3 shows that adding FFE increases AP by 3 points. And the combined use of FFE and H-BFP increases the AP value of the baseline from 38.6 to 42.4, an increase of 3.8 points. This shows that the two modules FFE and H-BFP in MT-FPN promote each other.

[0112] Table 3 Impact of each component on COCO val-2017

[0113]

[0114] 2) Ablation experiment on DBLL

[0115] Table 4 shows that DBLL improves the baseline AP value from 38.6 to 39.6. The results are greatly improved on higher IoU indicators such as AP75, verifying the effectiveness of changing the loss function during training to compensate for high-quality samples. _tune ) improves AP from 38.6 to 39.8. Therefore, dynamically changing the loss function shape and solving the discontinuity problem will bring reasonable gains, especially in high-quality indicators. Dynamically balanced loss (DBLL with fine _tune ) improves the baseline by 1.2 points AP.

[0116] Table 4 The impact of each component on COCO val-2017

[0117]

[0118] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes can be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims. At the same time, common sense such as the well-known specific structures and characteristics in the embodiments are not described in detail here. Finally, the scope of protection claimed by the present invention shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A target detection method based on multi-scale feature fusion, characterized in that: The following steps are involved: S1: Build a target detection model and train the target detection model; S2: Acquire image data of the target to be detected; S3: Input the image data of the target to be detected into the trained target detection model; The target detection model first extracts multi-scale feature maps based on image data; then fully fuses the low-level feature maps through layer-by-layer feature fusion and feedback feature fusion to obtain the corresponding low-level fusion feature maps; then only fuses and updates the high-level feature maps through the attention mechanism to obtain the corresponding high-level fusion feature maps; finally, the corresponding multi-scale fusion feature maps are generated based on the low-level fusion feature maps and the high-level fusion feature maps; The target detection model is trained by a dynamic balanced loss function; the dynamic balanced loss function introduces a dynamic strategy based on the balanced L1 loss function, so that dynamic label allocation can be achieved and the shape of the loss function can be adjusted; Dynamic label allocation is expressed by the following formula: In the above formula: Label represents the label, T now Indicates the current threshold; During training, we first calculate the IoUs between the proposed box and its true box to get set I, and then select the Kth maximum value from set I to update T now The threshold value of The dynamic balance loss function is expressed by the following formula: In the above formula: L b represents the dynamic balance loss, α, b, and γ are mutually constrained by the formula αln(b+1)=γ; during training, the regression label E between the proposed box and its true box is first calculated, and then the Kth minimum value is selected from the regression label E to update β now ; By fine-tuning the factor _tune The shape of the loss function is fine-tuned to eliminate the influence of derivative discontinuity. The formula of the fine-tuning factor is as follows: Among them, fine _tune <γ; S4: Complete target detection based on multi-scale fusion feature maps.

2. The target detection method based on multi-scale feature fusion according to claim 1, characterized in that: The target detection model includes an FPN network, an FFE network and an H-BFP network; The FPN network is used to extract multi-scale feature maps based on image data, and then perform feature fusion layer by layer from the highest level downwards, and output high-level feature maps without feature fusion, as well as preliminary fused feature maps obtained by feature fusion of low-level feature maps; The FFE network is used to extract multi-scale feature maps based on image data, and then perform layer-by-layer feature fusion and feedback feature fusion on the feature maps of the corresponding scales with the preliminary fusion feature maps output by the FPN network to generate the corresponding low-level fusion feature maps; The H-BFP network is used to add and average the high-level feature map and the low-level fusion feature map, and extract the corresponding feature information; then the high-level feature map is fused and updated based on the feature information and the attention mechanism to obtain the corresponding high-level fusion feature map.

3. The target detection method based on multi-scale feature fusion according to claim 2, characterized in that: The FFE network includes a feedback feature input module; The feedback feature input module obtains the preliminary fusion feature map output by the FPN network and implements the input of the preliminary fusion feature map through two branches; Branch 1 is used to input the preliminary fusion feature map as feedback information into the FFE network, so that the preliminary fusion feature map can be fused layer by layer with the feature map of the corresponding scale to output the corresponding deep fusion feature map; Branch two is used to input the preliminary fusion feature map to the output position of the FFE network, so that the preliminary fusion feature can be feedback-fused with the deep fusion feature map output by the FFE network to generate the corresponding low-level fusion feature map.

4. The target detection method based on multi-scale feature fusion according to claim 3, characterized in that: When the preliminary fusion feature map is fused layer by layer with the feature map of the corresponding scale, the output of the FFE network is adjusted through a 1*1 convolutional layer, and the corresponding channel size is adjusted to 256 to output the corresponding deep fusion feature map.

5. The target detection method based on multi-scale feature fusion according to claim 2, characterized in that: The FFE network includes a global module; the global module is used to extract global features from the input preliminary fusion feature map to obtain richer feature semantic information in the preliminary fusion feature map.

6. The target detection method based on multi-scale feature fusion according to claim 2, characterized in that: The H-BFP network first performs the high-level feature map P output by the FPN network. h Downsample to obtain the corresponding high-level feature map P h+1 ; Then the high-level feature map P h and P h+1 The low-level fusion feature map output by the FFE network is added and averaged to obtain the feature information after fusion of each layer; then the feature information is subjected to no local operation, and the high-level feature map P is obtained by sampling the feature information. h and P h+1 High-level fusion feature map P of corresponding scale h ′ and P h +1; then the high-level fusion feature map P h ′ and P′ h+1 Perform convolution and activation function operations to calculate the corresponding attention feature maps respectively; finally, based on the high-level fusion feature map P h ′ and P′ h+1 The corresponding attention feature maps are calculated respectively h With P h ′, and P h+1 and P′ h+1 Weighted sum, and then update the high-level fusion feature map P h ′ and P′ h+1 .

7. The target detection method based on multi-scale feature fusion according to claim 6, characterized in that: Update the high-level feature map using the following formula: P h ′=torch.sigmoid(self.conv(P′ h ))*P h ′+(1-torch.sigmoid(self.conv(P h ′)))*P h ; P′ h+1 =torch.sigmoid(self.conv(P′ h+1 ))*P h +1+(1-torch.sigmoid(self.conv(P′ h+1 )))*P h+1 ; In the above formula: P h ′、P′ h+1 represents the high-level fusion feature map, P h , P h+1 Represents a high-level feature map, self.conv(P′ h ) indicates P h 'After convolution operation; self.conv(P' h+1 ) represents P′ h+1 After convolution operation; torch.sigmoid(self.conv(P h ′)) indicates that the high-level fusion feature map P is not updated h ′ when calculating the attention feature map; torch.sigmoid(self.conv(P′ h+1 )) indicates that the high-level fusion feature map P′ is not updated h+1 The attention feature map calculated at .

Images