Weighted fusion method, system and equipment for results of multiple detection models and medium

Abstract

The invention provides a multi-detection model result weighted fusion method, system and device and a medium, and relates to the technical field of computer vision, and the method comprises the steps: training a plurality of detection models, and obtaining the reasoning results of the detection models on a test set. Respectively aiming at a single detection model, constructing a training data set according to a reasoning result and a test set, and training a regression model to calibrate the confidence coefficient of the detection model. And carrying out confidence coefficient calibration on the plurality of detection models by using the regression model, and carrying out weighted fusion on the detection frames output by the detection models based on a calibration result to generate a fused reasoning result. According to the method, the problems existing in methods based on feature fusion and result fusion in the prior art are solved, and the overall detection performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, system, device and medium for weighted fusion of multi-detection model results. Background Art

[0002] The current fusion methods of multiple detection models can be divided into feature-based fusion and result-based fusion. Feature fusion is achieved through network design, which involves merging feature maps from different sources in the network in a certain way and then inputting them into the next network layer. Feature fusion runs through the training and reasoning of the network; result fusion refers to the fusion of multiple detection model reasoning results, and the methods include traditional NMS, WBF based on weighted NMS, etc. Result fusion does not require retraining the network, which is equivalent to the fusion based on all single model reasoning results. The specific existing technologies involved include a flight parking space detection method based on multi-detector fusion (patent CN109697426B), a multi-detector result fusion tracking method (patent CN114612697A), a high-dynamic multi-target drone detection method based on feature fusion (patent CN114821377A), and a target detection method based on multi-detection network fusion (patent CN116824549B).

[0003] At least the following problems still exist in the existing technology: the alignment of confidence distributions between different models is not taken into account during the result fusion process; directly fusing multiple detection results when the confidence distributions of different models are inconsistent is likely to lead to problems such as poor fusion effect or limited fusion effect; current feature fusion requires retraining of the new network, the actual effect is unknown, the trial and error cost is high, and more training resources are consumed. Summary of the invention

[0004] The first aspect of the present invention provides a weighted fusion method for multiple detection model results to solve the problems existing in feature-based fusion and result-based fusion in the prior art. The method comprises:

[0005] Training multiple detection models, and obtaining inference results of the multiple detection models for the test set respectively;

[0006] For a single detection model, calculating the target confidence of the single detection model according to the inference result of the single detection model and the test set, constructing a training data set according to the inference result of the single detection model and the target confidence, and training a regression model based on the constructed training data set;

[0007] Respectively training a plurality of regression models corresponding to the plurality of detection models one by one, applying the plurality of regression models to calibrate the confidence of the plurality of detection models, and obtaining the plurality of detection models after confidence calibration;

[0008] According to the multiple detection models after confidence calibration, weighted fusion is performed on the detection boxes output by the multiple detection models to obtain a fused reasoning result.

[0009] Furthermore, the test set includes a first test set and a second test set, and the reasoning results of a single detection model among the multiple detection models for the first test set and the second test set are respectively obtained to form the reasoning results of the multiple detection models.

[0010] Furthermore, the data ratio of the first test set to the second test set is 9:1.

[0011] Furthermore, the confidence thresholds of the multiple detection models are all set to 0.1, and the intersection-over-union ratio thresholds of the multiple detection models are all set to 0.8.

[0012] Furthermore, obtaining the inference result of the single detection model on the second test set includes:

[0013] Obtain all inference frames in the second test set that are determined to be true positives by the single detection model, and for a single inference frame determined to be a true positive:

[0014] An intersection-and-union ratio between the single inference box and the true box of the second test set is calculated, and a value of the calculated intersection-and-union ratio is used as a target confidence corresponding to the single inference box.

[0015] Furthermore, weighted fusion is performed on the detection frames output by the multiple detection models, including:

[0016] Obtaining a detection frame set of a current category of the multiple detection models;

[0017] Identifying complementary detection frames and similar detection frames in the detection frame set, wherein complementary detection frames do not need to be merged, and similar detection frames are merged;

[0018] A voting threshold is preset, and the voting threshold is used to determine whether the two current detection frames are similar detection frames. If the intersection-over-union ratio between the two current detection frames is greater than or equal to the voting threshold, the two current detection frames are considered to be similar detection frames, and the two current detection frames are weightedly fused to obtain a fused comprehensive detection frame.

[0019] The two current detection frames are weightedly fused, the method comprising:

[0020]

[0021] Among them, is the final fusion frame obtained after weighting, Represents the original reasoning results of each detection model, σ SV yes The hyperparameters in , is the confidence of the detection box.

[0022] The second aspect of the present invention provides a multi-detection model result weighted fusion system to solve the problems existing in the prior art based on feature fusion and result fusion. The system comprises:

[0023] A regression model training module is used to pre-train multiple detection models, respectively obtain the inference results of the multiple detection models for the test set, and for a single detection model: calculate the target confidence of the single detection model according to the inference result of the single detection model and the test set, construct a training data set according to the inference result of the single detection model and the target confidence, and train the regression model based on the constructed training data set;

[0024] A confidence calibration module, used to respectively train a plurality of regression models corresponding to the plurality of detection models one by one, and to apply the plurality of regression models to perform confidence calibration on the plurality of detection models to obtain the plurality of detection models after confidence calibration;

[0025] The detection result fusion module is used to perform weighted fusion on the detection boxes output by the multiple detection models according to the multiple detection models after confidence calibration to obtain a fused inference result.

[0026] A third aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements any of the above-mentioned weighted fusion methods for multi-detection model results.

[0027] A fourth aspect of the present invention provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned multi-detection model result weighted fusion methods.

[0028] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least the following: based on the existing detection results of multiple detection models, the present invention aligns the confidence distribution of the results of multiple detection models on the one hand, and performs confidence voting weighting in the weighted fusion stage of multiple detection results on the other hand, thereby achieving the effect of improving detection accuracy and reducing missed detections and false detections. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 It is a schematic diagram of the overall process of a weighted fusion method of multiple detection model results provided by an embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of the flow of a detection frame fusion method provided by an embodiment of the present invention;

[0032] Figure 3 It is a schematic diagram of the structure of a multi-detection model result weighted fusion system provided by an embodiment of the present invention;

[0033] Figure 4 is a schematic diagram of a computer device provided by an embodiment of the present invention;

[0034] Reference numerals in the figure: 300, a multi-detection model result weighted fusion system; 301, a regression model training module; 302, a confidence calibration module; 303, a detection result fusion module; 401, a memory; 402, a processor. DETAILED DESCRIPTION

[0035] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0036] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0037] This embodiment proposes a method for weighted fusion of multiple detection model results based on confidence alignment. First, a small number of detection results containing true values ​​are taken to train the confidence alignment model, and then the confidence in other prediction results is mapped to the same distribution using the trained confidence alignment model. Finally, the weighted fusion of the detection frames is performed using confidence voting weighting. Based on the existing detection results of multiple detection models, the present invention aligns the confidence distribution of multiple detection model results on the one hand, and performs confidence voting weighting in the weighted fusion stage of multiple detection results on the other hand, thereby achieving the effect of improving detection accuracy and reducing missed detections and false detections.

[0038] like Figure 1 As shown, the specific implementation steps of a weighted fusion method for multiple detection model results provided in this embodiment include:

[0039] Step S100: train multiple detection models, and obtain the inference results of the multiple detection models for the test set respectively. For a single detection model:

[0040] Calculating the target confidence of the single detection model according to the inference result of the single detection model and the test set, constructing a training data set according to the inference result of the single detection model and the target confidence, and training a regression model based on the constructed training data set, wherein the regression model is used to calibrate the confidence of the single detection model;

[0041] Preferably, in this embodiment, three models, yolov7, yolov7-e6e, and dino, are trained, and each model includes three categories: car, bus, and truck.

[0042] Furthermore, in this embodiment, the test set includes a first test set and a second test set, and the inference results of a single detection model among the multiple detection models for the first test set and the second test set are respectively obtained to form the inference results of the multiple detection models.

[0043] Preferably, the data ratio of the first test set to the second test set is 9:1.

[0044] Specifically, first obtain the inference results of multiple models for the test set. First, in this embodiment, multiple detection models are pre-trained according to actual needs, and different detection models are used in turn to infer the detection results of the test set and save the inference results. Secondly, a training data set is constructed, and the training data includes the inference results of each detection model on the second test set and the corresponding ground truth (GT) information of the second test set.

[0045] Furthermore, all the inference boxes in the second test set that are judged as true positives (TP) by the single detection model are obtained, and for the single inference box judged as true positive: the intersection over union (IOU value) between the single inference box and the real box of the second test set is calculated, and the calculated value of the intersection over union is used as the target confidence corresponding to the single inference box.

[0046] Specifically, the above method unifies the alignment standard of confidence, that is, the actual detection box confidence is defined according to the IOU value of the actual inference box and the corresponding true value box. Based on the confidence of all inference boxes before and after the second test set is updated, a regression model can be constructed to fit and train the mapping relationship between the original confidence and the target confidence.

[0047] Preferably, in this embodiment, a linear regression model is selected to calibrate the confidence of the multiple detection models respectively. Preferably, the number of samples in the second test set is 2000, and an average sample includes 20 TP boxes and 40,000 regression points, that is, the number of training samples of the linear regression model is 40,000. Each detection model corresponds to a linear regression model, and the total number of linear regression models that need to be trained is equal to the number of detection models. Each trained linear regression model can be directly used to calibrate the confidence of the corresponding detection model reasoning box.

[0048] Preferably, during the reasoning process of each detection model, the reasoning result post-processing flow of each detection model is adopted, and the confidence threshold of each detection model is set to 0.1 and the IOU threshold is set to 0.8, so as to generate more original detection frames.

[0049] Step S200: training a plurality of regression models corresponding to the plurality of detection models one by one respectively, applying the plurality of regression models to calibrate the confidence of the plurality of detection models, and obtaining the plurality of detection models after confidence calibration;

[0050] Specifically, according to the linear regression models trained in step S100, the confidence corresponding to each inference box in the test set is calibrated. Each time, the original confidence is input into the linear regression model and the calibrated confidence is output. After the calibrated confidence covers the original confidence, the next step of detection box fusion based on confidence voting weighting is performed.

[0051] Step S300: According to the multiple detection models after confidence calibration, weighted fusion is performed on the detection boxes output by the multiple detection models to obtain a fused inference result.

[0052] Specifically, if Figure 2 As shown, according to the detection frame results after the confidence check of each detection model in step S200, the detection frame fusion based on confidence voting weighting can be further performed. Specifically, the following steps are included:

[0053] Step S301, obtaining a detection box set boxes of the current category of the multiple detection models;

[0054] Step S302, identifying complementary detection frames and similar detection frames in the detection frame set boxes; wherein, complementary detection frames: detection frames provided by different detection models have no overlap or little overlap in position and do not need to be merged; similar detection frames: detection frames provided by different detection models have a large overlap in position and need to be merged to avoid redundancy;

[0055] Step S303: preset a voting threshold vote_thresh, where the voting threshold is used to determine whether two current detection frames are similar detection frames. If the intersection-over-union (IOU) between the two current detection frames is greater than or equal to the voting threshold, the two current detection frames are considered to be similar detection frames, and weighted fusion is performed on the two current detection frames.

[0056] Step S304: perform weighted fusion on the two current detection frames according to the following formula (1) to obtain a fused comprehensive detection frame, and re-evaluate the fused comprehensive detection frame to determine its accuracy and reliability.

[0057] like Figure 2 As shown in the figure, the fusion by category is considered at the same time during the detection frame fusion process. In practice, the effect of fusion by category will be better. In the loop, for each detection frame, the IOU (intersection-over-union ratio) between it and other detection frames is calculated, and whether to merge the detection frames is determined based on vote_thresh (confidence voting threshold). The method for calculating the merged detection frame is as follows: the coordinates of the merged detection frame are obtained by weighting the coordinates of all the detection frames to be merged. The merging formula is shown in the following formula (1):

[0058]

[0059] In formula (1), is defined as: is the final fusion frame obtained after weighting, Represents the original inference results of each detection model, is the confidence of the detection box. σ SV is a hyperparameter used to adjust the smoothness of the formula. In actual use, σ SV Set to 0.04.

[0060] Based on the same inventive concept, a multi-detection model result weighted fusion system is also provided in an embodiment of the present invention, as described in the following embodiments. Since the principle of solving the problem by a multi-detection model result weighted fusion system is similar to that of a multi-detection model result weighted fusion method, the implementation of a multi-detection model result weighted fusion system can refer to the implementation of a multi-detection model result weighted fusion method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0061] Figure 3 is a structural block diagram of a multi-detection model result weighted fusion system 300 according to an embodiment of the present invention. Figure 3 As shown, including:

[0062] The regression model training module 301 is used to pre-train multiple detection models, respectively obtain the inference results of the multiple detection models for the test set, and for a single detection model: calculate the target confidence of the single detection model according to the inference result of the single detection model and the test set, construct a training data set according to the inference result of the single detection model and the target confidence, train a regression model based on the constructed training data set, and the regression model is used to calibrate the confidence of the single detection model;

[0063] A confidence calibration module 302, used to respectively train a plurality of regression models corresponding to the plurality of detection models one by one, and to apply the plurality of regression models to perform confidence calibration on the plurality of detection models, so as to obtain the plurality of detection models after confidence calibration;

[0064] The detection result fusion module 303 is used to perform weighted fusion on the detection boxes output by the multiple detection models according to the multiple detection models after confidence calibration to obtain a fused inference result.

[0065] Furthermore, the test set includes a first test set and a second test set, and the reasoning results of a single detection model among the multiple detection models for the first test set and the second test set are respectively obtained to form the reasoning results of the multiple detection models.

[0066] Furthermore, the data ratio of the first test set to the second test set is 9:1.

[0067] Furthermore, the confidence thresholds of the multiple detection models are all set to 0.1, and the intersection-over-union ratio thresholds of the multiple detection models are all set to 0.8.

[0068] Furthermore, the regression model training module 301 is also used for:

[0069] Obtain all inference frames in the second test set that are determined to be true positives by the single detection model, and for a single inference frame determined to be a true positive:

[0070] An intersection-and-union ratio between the single inference box and the true box of the second test set is calculated, and a value of the calculated intersection-and-union ratio is used as a target confidence corresponding to the single inference box.

[0071] In this embodiment, a computer device is also provided, such as Figure 4 As shown, it includes a memory 401, a processor 402, and a computer program stored in the memory 401 and executable on the processor 402. When the processor 402 executes the computer program, any one of the above-mentioned weighted fusion methods for multi-detection model results is implemented.

[0072] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0073] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned multi-detection model result weighted fusion methods.

[0074] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0075] The embodiments of the present invention achieve the following technical effects:

[0076] Based on multiple detection results, this application improves the fusion effect of multiple detection models through confidence alignment and confidence voting weighted fusion. This method significantly improves the detection accuracy and reduces missed detections and false detections. It solves the problem that the new network needs to be retrained based on feature fusion, the actual effect is unknown, the trial and error cost is high, and more training resources are consumed; it also solves the problem that the alignment of confidence distributions between different models is not considered in the existing result fusion process, resulting in poor fusion effect or limited fusion effect. The F1 index of the detection frame fused and output by the technical solution of this embodiment is improved by more than 2 points compared with the detection frame after traditional fusion.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A weighted fusion method for multiple detection model results, characterized in that: The method comprises: Training multiple detection models, and obtaining inference results of the multiple detection models for the test set respectively; For a single detection model, calculating the target confidence of the single detection model according to the inference result of the single detection model and the test set, constructing a training data set according to the inference result of the single detection model and the target confidence, and training a regression model based on the constructed training data set; Respectively training a plurality of regression models corresponding to the plurality of detection models one by one, applying the plurality of regression models to calibrate the confidence of the plurality of detection models, and obtaining the plurality of detection models after confidence calibration; According to the multiple detection models after confidence calibration, weighted fusion is performed on the detection boxes output by the multiple detection models to obtain a fused reasoning result.

2. A weighted fusion method for multiple detection model results according to claim 1, characterized in that: include: The test set includes a first test set and a second test set, and the reasoning results of a single detection model among the multiple detection models for the first test set and the second test set are respectively obtained to form the reasoning results of the multiple detection models.

3. A weighted fusion method for multiple detection model results according to claim 2, characterized in that: The data ratio of the first test set to the second test set is 9:

1.

4. The method for weighted fusion of multiple detection model results according to claim 1, characterized in that: include: The confidence thresholds of the multiple detection models are all set to 0.1, and the intersection-over-union ratio thresholds of the multiple detection models are all set to 0.

8.

5. A weighted fusion method for multiple detection model results according to claim 2, characterized in that: Obtaining the inference result of the single detection model on the second test set, including: Obtain all inference frames in the second test set that are determined to be true positives by the single detection model, and for a single inference frame determined to be a true positive: An intersection-and-union ratio between the single inference box and the true box of the second test set is calculated, and a value of the calculated intersection-and-union ratio is used as a target confidence corresponding to the single inference box.

6. A weighted fusion method for multiple detection model results according to claim 1, characterized in that: The weighted fusion of the detection frames output by the multiple detection models includes: Obtaining a detection frame set of a current category of the multiple detection models; Identifying complementary detection frames and similar detection frames in the detection frame set, wherein complementary detection frames do not need to be merged, and similar detection frames are merged; A voting threshold is preset, and the voting threshold is used to determine whether the two current detection frames are similar detection frames. If the intersection-over-union ratio between the two current detection frames is greater than or equal to the voting threshold, the two current detection frames are considered to be similar detection frames, and the two current detection frames are weightedly fused to obtain a fused comprehensive detection frame.

7. A weighted fusion method for multiple detection model results according to claim 6, characterized in that: The two current detection frames are weightedly fused, the method comprising: Among them, is the final fusion frame obtained after weighting, Represents the original reasoning results of each detection model, σ SV yes The hyperparameters in , is the confidence of the detection box.

8. A multi-detection model result weighted fusion system, characterized in that: include: A regression model training module is used to pre-train multiple detection models, respectively obtain the inference results of the multiple detection models for the test set, and for a single detection model: calculate the target confidence of the single detection model according to the inference result of the single detection model and the test set, construct a training data set according to the inference result of the single detection model and the target confidence, and train the regression model based on the constructed training data set; A confidence calibration module, used to respectively train a plurality of regression models corresponding to the plurality of detection models one by one, and to apply the plurality of regression models to perform confidence calibration on the plurality of detection models to obtain the plurality of detection models after confidence calibration; The detection result fusion module is used to perform weighted fusion on the detection boxes output by the multiple detection models according to the multiple detection models after confidence calibration to obtain a fused inference result.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, a weighted fusion method of multiple detection model results as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing a multi-detection model result weighted fusion method as described in any one of claims 1 to 7.

Images