Target detection post-processing method and system based on information fusion, terminal and medium

Abstract

The application provides a target detection post-processing method and system based on information fusion, a terminal and a medium, comprising: clustering all prediction boxes in a target detection result into multiple clusters to obtain a representative prediction box of each cluster and statistical information of detection confidence in the cluster; performing category confidence calculation on the representative prediction box of each cluster to calculate the probability that the category prediction is correct as the category confidence; calculating the probability that each prediction box belongs to correct prediction according to the category confidence and the statistical information of detection confidence in each cluster; retaining the prediction box with a probability greater than a preset threshold; and discarding the prediction box with a probability less than the preset threshold. The application refers to multiple different types of confidence information, has a more robust advantage compared to a traditional post-processing method represented by non-maximum suppression, and integrates multiple prediction results for the same object, so that the positioning of the object boundary is more accurate.

Description

Technical Field

[0001] This invention relates to a target detection post-processing method, specifically, to a target detection post-processing method, system, terminal, and medium based on information fusion, belonging to the field of computer vision. Background Technology

[0002] Object detection is a crucial task in computer vision. Understanding image content is a prerequisite for technologies such as intelligent surveillance and autonomous driving. Accurate object detection is the foundation of image content understanding, and post-processing is an indispensable component of object detection systems, significantly impacting their accuracy. The object detection task involves finding objects of a specific category from a given image, calculating the bounding boxes that precisely enclose the objects, and predicting the probability that the objects within those boxes belong to a specific category. Existing object detection methods often make many redundant predictions for the same object. The goal of post-processing is to filter out these redundant predictions for the same object to improve the accuracy of the detection results.

[0003] Non-maximum suppression (NMS) is an important method in the post-processing workflow of target detection, as exemplified by patent CN108960266B. The principle of NMS is to iteratively select the predicted bounding boxes to be retained, using the intersection-union ratio (IUU) between predicted boxes as the similarity between two predictions. In each iteration, the predicted box with the highest detection confidence is selected and retained, while all other predicted boxes with a similarity greater than a specific threshold are discarded. The most commonly used NMS employs a fixed similarity threshold, which requires manual adjustment of the threshold to achieve good post-processing results and necessitates extensive experimental verification of the threshold selection's rationality. Furthermore, this method does not handle densely distributed targets well. In recent years, NMS methods using soft thresholds have been proposed, such as patent CN 108596170 B. In these methods, the similarity threshold is adaptively adjusted based on the density of the predicted box distribution, improving detection accuracy in cases of target clustering.

[0004] While nonmaximum suppression (NMS) can quickly and efficiently filter out most redundant bounding boxes and retain relatively accurate ones, it has a significant drawback: NMS only outputs the bounding box with the highest detection confidence in each iteration, completely ignoring the information of the filtered-out bounding boxes. In contrast, fusing multiple bounding boxes for the same target usually yields more accurate localization results. Besides the inherent problems with NMS itself, post-processing methods based on this algorithm require further filtering of the NMS output based on a preset detection confidence threshold, retaining the bounding boxes with higher detection confidence as the final output of the target detection system. This filtering method is not robust.

[0005] Therefore, there is an urgent need to develop a new post-processing technology for target detection to avoid the above problems. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a target detection post-processing method, system, terminal, and medium based on information fusion.

[0007] A first aspect of the present invention provides a target detection post-processing method based on information fusion, comprising:

[0008] All predicted boxes in the target detection results are clustered into multiple clusters to obtain the representative predicted box of each cluster and the statistical information of the detection confidence within the cluster;

[0009] For each cluster, the category confidence score is calculated by performing a category confidence score on the representative prediction box. The probability that the category prediction is correct is then used as the category confidence score.

[0010] Based on the category confidence and the statistical information of the detection confidence within each cluster, the probability of each prediction box belonging to the correct prediction is calculated. Prediction boxes with a probability greater than a preset threshold are retained, and prediction boxes with a probability less than the preset threshold are discarded.

[0011] Optionally, the step of clustering all predicted boxes in the target detection results into multiple clusters involves: using an iterative clustering method to assign all predicted boxes into multiple clusters.

[0012] Furthermore, the clustering method includes:

[0013] The crossover ratio (CROR) is used as a measure of similarity between two boxes. The weighted average of the predicted boxes in each cluster is calculated with the detection confidence as the weight and is used as the representative predicted box of that cluster.

[0014] The maximum value and sum of all detection confidence scores in each cluster are counted as statistical information on the detection confidence scores within the cluster.

[0015] Optionally, a category confidence score is calculated for the representative prediction box of each cluster, wherein a convolutional neural network is used to calculate the confidence score that the object in the representative prediction box belongs to the predicted category.

[0016] Furthermore, the convolutional neural network is a neural network with 18 convolutional layers and 2 fully connected layers.

[0017] Optionally, the probability of each prediction box belonging to a correct prediction is calculated based on the category confidence and the statistical information of the detection confidence within each cluster, wherein a fully connected neural network is used to calculate the probability that the prediction box is a correct prediction.

[0018] Furthermore, the probability of a representative prediction box being correctly predicted is calculated using a fully connected neural network, wherein the maximum and sum of the detection confidence scores of the representative prediction box and the category confidence score are used as inputs, and the probability of the representative prediction box being correctly predicted is calculated through a neural network containing three fully connected layers.

[0019] A second aspect of the present invention provides a target detection post-processing system based on information fusion, comprising:

[0020] Clustering module: Clusters all predicted bounding boxes in the target detection results into multiple clusters, and obtains the representative predicted bounding box of each cluster and the statistical information of the detection confidence within the cluster;

[0021] Category confidence calculation module: Calculates the category confidence of the representative prediction box of each cluster obtained by the clustering module, and calculates the probability that its category prediction is correct, which is used as the category confidence.

[0022] Prediction box filtering module: Based on the category confidence obtained by the category confidence calculation module and the statistical information of the detection confidence within each cluster obtained by the clustering module, calculate the probability that each prediction box belongs to the correct prediction, retain prediction boxes with a probability greater than a preset threshold, and discard prediction boxes with a probability less than the preset threshold.

[0023] A third aspect of the present invention provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to perform the aforementioned target detection post-processing method based on information fusion.

[0024] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is used to perform the aforementioned target detection post-processing method based on information fusion.

[0025] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0026] The target detection post-processing method and system provided by the present invention use information from all predicted bounding boxes related to the target object when estimating the bounding box of the target object, and estimate the correctness of the predicted category, thereby improving the accuracy of the prediction of the target object category and location.

[0027] The target detection post-processing method and system provided by this invention, based on multiple dimensions of information such as class confidence and statistical information on detection confidence within each cluster, filters the final output detection results, making it more robust than nonmaximum suppression methods based on a single confidence level. Therefore, this invention can better accomplish the target detection post-processing task. Attached Figure Description

[0028] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0029] Figure 1 This is a flowchart of a target detection post-processing method in one embodiment of the present invention;

[0030] Figure 2 This is a structural block diagram of the target detection post-processing module of the present invention;

[0031] Figure 3 This is a flowchart of a target detection post-processing method in a preferred embodiment of the present invention. Detailed Implementation

[0032] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0033] Reference Figure 1 As shown in the figure, this embodiment of the invention provides a target detection post-processing method based on information fusion, including the following steps:

[0034] S100, cluster all the predicted boxes in the target detection results into multiple clusters, and obtain the representative predicted box of each cluster and the statistical information of the detection confidence within the cluster;

[0035] In this step, all predicted bounding boxes can be divided into multiple clusters using an iterative clustering method. Specifically, in a preferred embodiment, the clustering method may include: using the intersection-union ratio (IU) as a measure of similarity between two boxes; calculating the weighted average of the predicted bounding boxes in each cluster as the representative predicted bounding box of that cluster, with detection confidence as the weight; and calculating the maximum value and sum of all detection confidence scores in each cluster as statistical information on the detection confidence within the cluster. The information obtained in this step is used for further processing and calculation in subsequent steps.

[0036] S200, calculate the class confidence of the representative prediction box for each cluster, and calculate the probability that its class prediction is correct, which is used as the class confidence.

[0037] In this step, a convolutional neural network can be used to calculate the confidence level representing whether the object within the predicted bounding box belongs to the predicted category. Specifically, a convolutional neural network with 18 convolutional layers and 2 fully connected layers is preferred. Of course, other neural network structures can be used in other embodiments, and it is not limited to this. In this step, estimating the correctness of the predicted category is beneficial to improving the accuracy of the prediction of the target object category and location.

[0038] S300: Based on the statistical information of category confidence and detection confidence within each cluster, calculate the probability that each prediction box belongs to the correct prediction, retain prediction boxes with a probability greater than a preset threshold, and discard prediction boxes with a probability less than the preset threshold.

[0039] In this step, a fully connected neural network is used to calculate the probability that the representative prediction box is correctly predicted. Further, using the maximum and sum of the detection confidence scores of the representative prediction box, as well as the class confidence score, as input, a neural network with three fully connected layers is used to calculate the probability that the representative prediction box is correctly predicted. This step uses multiple dimensions of information, including class confidence and statistical information on detection confidence within each cluster, to filter the final output detection results, making it more robust than non-maximum suppression methods based on a single confidence score.

[0040] The target detection post-processing method based on information fusion proposed in the above embodiments of the present invention utilizes information fusion of target detection results. Information fusion is a method that aggregates information from multiple predicted boxes for the same target to improve detection accuracy. Specifically, in the embodiments of the present invention, all predicted boxes are iteratively clustered, using the intersection-union ratio (IUU) between boxes as a measure of similarity. In each iteration, predicted boxes with a similarity higher than a certain threshold to the representative predicted box of the current cluster and without a assigned box are included in the current cluster. Then, the representative predicted box of the current cluster is recalculated. If no new predicted boxes are included, a new cluster is initialized, and the predicted box with the highest detection confidence among those without an assigned box is included. This fusion method fully utilizes all information in the detection results, making the localization of the target more accurate, while providing richer confidence information, making the selection of the final output results more robust.

[0041] Reference Figure 2 As shown, based on the same technical concept, another embodiment of the present invention also provides a target detection post-processing system based on information fusion, comprising:

[0042] Clustering module: Clusters all predicted bounding boxes in the target detection results into multiple clusters, and obtains the representative predicted bounding box of each cluster and the statistical information of the detection confidence within the cluster;

[0043] Category confidence calculation module: Calculates the category confidence of the representative prediction box of each cluster obtained by the clustering module, and calculates the probability that its category prediction is correct, which is used as the category confidence.

[0044] Predicted bounding box filtering module: Based on the category confidence obtained from the category confidence calculation module and the statistical information of the detection confidence within each cluster obtained from the clustering module, calculate the probability that each predicted bounding box belongs to the correct prediction, retain the predicted bounding boxes with a probability greater than the preset threshold, and discard the predicted bounding boxes with a probability less than the preset threshold.

[0045] The system proposed in this embodiment can be described in terms of its modules as described above. Figure 1 The techniques corresponding to each step in the target detection post-processing method based on information fusion shown are not described in detail here.

[0046] Based on the same technical concept, another embodiment of the present invention also provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, it is used to perform… Figure 1 The example shown is a post-processing method for target detection based on information fusion.

[0047] Based on the same technical concept, another embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which is used to perform actions when executed by a processor. Figure 1 The example shown is a post-processing method for target detection based on information fusion.

[0048] To better illustrate the above-described technical solutions of the present invention, detailed embodiments are described below. However, it should be understood that the present invention is not limited to the following embodiments.

[0049] like Figure 3 As shown in this embodiment, for a set of predicted boxes output by the detection algorithm, the clustering module is first used to divide the set of boxes into multiple clusters, each cluster being a prediction of a target object; the representative predicted box of each cluster is input into the category confidence calculation module to calculate the probability that its category prediction is correct, which is used as the category confidence; the statistical information of the category confidence and the detection confidence within each cluster is input into the predicted box filtering module to calculate the probability that each predicted box belongs to the correct prediction, and by comparing it with a threshold, the final output predicted box is determined.

[0050] Specifically, in this embodiment, a clustering module, a category confidence calculation module, and a predicted bounding box filtering module are used to implement post-processing for target detection based on information fusion, and the following specific operations are performed:

[0051] S1, Clustering Module: Clusters all predicted bounding boxes in the target detection results into multiple clusters, and obtains the representative predicted bounding box of each cluster and the statistical information of the detection confidence within the cluster.

[0052] For the n predicted bounding boxes {b1, b2, ... bn} output by the object detection algorithm n The clustering module divides these n predicted boxes into k clusters C = {B1, B2, ..., B}. k In each cluster, the predicted bounding boxes are considered to point to the same target object, and the value of k is unknown before the clustering process is completed.

[0053] In this embodiment, the clustering method is performed iteratively, specifically:

[0054] At the start of clustering, initialize the cluster currently being processed.

[0055] In each iteration, if Select the forecast box that has not yet been assigned. The bounding box with the highest confidence level was detected in the middle. Will Add to B c In the middle, from B u Delete and As B at this time c The representative prediction box Calculate all prediction boxes B that have not yet been assigned. u With cluster B currently being processed c The representative prediction box intersection ratio like Greater than a preset threshold T IOU Then Assigned to B c In the middle, from B u Delete

[0056] After processing all unassigned prediction boxes, if no new prediction boxes are added to B in this iteration... c In the middle, then for B c The processing of B is complete. c Add to the output C, let Proceed to the next iteration; if a new prediction box is added to B c If the current cluster B is recalculated, then the calculation will be repeated. c The representative prediction box for:

[0057]

[0058] Then proceed to the next iteration. At the start of a new iteration, if... The clustering process then ends, and the clustering result C is output.

[0059] S2, Category Confidence Calculation Module: Calculates the category confidence of the representative prediction box for each cluster, and calculates the probability that the category prediction is correct, which is used as the category confidence.

[0060] In this step, each cluster B in the clustering result C is... i The representative prediction box The corresponding rectangular region is cropped from the original image and scaled to a 128*128 image I. I is then input into an 18-layer ResNet CNN model. The output dimension of the last fully connected layer of this model is consistent with the total number of classes N in reality. The output of this CNN model is then converted into probability predictions v = (v1, v2, ... v) for N classes using a sigmoid operation. N ), where v i Reflects The probability that an object belongs to category i. The parameters of the CNN model are initialized using transfer learning. ResNet is pre-trained on the ImageNet dataset for a classification task, and the parameters of the model are initialized using the converged model parameters.

[0061] S3, Prediction Box Filtering Module: Based on the statistical information of category confidence and detection confidence within each cluster, calculate the probability that each prediction box belongs to the correct prediction, retain prediction boxes with a probability greater than a preset threshold, and discard prediction boxes with a probability less than the preset threshold.

[0062] In this step, we will analyze the data for each cluster B in C. i The maximum detection confidence of all predicted boxes. m (B i ) and total Conf s (B i Assume B i If the predicted category is c, then [Conf] m (B i ),Conf s (B i ),v c [As a description] The feature vectors are input into a fully connected neural network with two hidden layers and an output dimension of 1. The network and each hidden layer use the ReLU activation function, and the output layer uses the sigmoid function to control the value range to (0,1) to represent... This represents the probability of a correct prediction. Boxes with a probability greater than a preset threshold are added to the system's output. The preset threshold can be used as a hyperparameter of the system and obtained by tuning it on a validation dataset.

[0063] The embodiments of the present invention refer to various types of confidence information, which has a more robust advantage compared to traditional post-processing methods represented by nonmaximum suppression. At the same time, it integrates multiple prediction results for the same object, making the localization of the object boundary more accurate.

[0064] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A target detection post-processing method based on information fusion, characterized in that, include: All predicted boxes in the target detection results are clustered into multiple clusters to obtain the representative predicted box of each cluster and the statistical information of the detection confidence within the cluster; For n predicted bounding boxes {b1, b2, ..., bn} in the target detection results, these n predicted bounding boxes are divided into k clusters C = {B1, B2, ..., Bk}. The predicted bounding boxes in each cluster are considered to point to the same target object. The value of k is unknown before the clustering process is completed. The clustering method is performed iteratively, specifically including: At the start of clustering, initialize the cluster currently being processed. ; In each iteration, if Select the prediction box that has not yet been assigned. The bounding box with the highest confidence level was detected in the middle. ,Will Add to From Delete ,and As at this time The representative prediction box Calculate all prediction boxes that have not yet been assigned. With the cluster currently being processed The representative prediction box intersection ratio ,like Greater than a preset threshold Then Assigned to From Delete ; After processing all unassigned prediction boxes, if no new prediction boxes are added in this iteration... In the middle, then for The processing is complete, and Add to output results In the middle, order Proceed to the next iteration; if a new prediction box is added... If the current cluster is not found, then the calculation will be recalculated. The representative prediction box Then proceed to the next iteration; at the start of a new iteration, if Then the clustering process ends and the clustering results are output. ; For each cluster, the representative prediction box is used to calculate the class confidence score, and the probability that its class prediction is correct is used as the class confidence score; specifically, this includes: processing the clustering results... Each cluster The representative prediction box The corresponding rectangular region is cropped from the original image, and the resulting scaled image is... Given an 18-layer ResNet CNN model, the output dimension of the last fully connected layer of this model is equal to the total number of classes in reality. To maintain consistency, the output of this CNN model is processed... Operation converted to Probability prediction for each category ,in Reflects The objects in the middle belong to the category The probability of; Based on the category confidence and the statistical information of the detection confidence within each cluster, the probability of each predicted box belonging to a correct prediction is calculated. Predicted boxes with a probability greater than a preset threshold are retained, and predicted boxes with a probability less than the preset threshold are discarded. Specifically, this includes: statistical analysis. Each cluster The maximum detection confidence of all predicted boxes. and total Assuming The prediction category is Then As a description The feature vectors are input into a fully connected neural network with two hidden layers and an output dimension of 1. This network and each hidden layer use... Activation function, used in the output layer The function controls the range within , to indicate This represents the probability of a correct prediction.

2. The target detection post-processing method based on information fusion according to claim 1, characterized in that, The recalculation of the current cluster The representative prediction box for: 。 3. A target detection post-processing system based on information fusion, employing the target detection post-processing method based on information fusion as described in claim 1, characterized in that, include: Clustering module: Clusters all predicted bounding boxes in the target detection results into multiple clusters, and obtains the representative predicted bounding box of each cluster and the statistical information of the detection confidence within the cluster; Category confidence calculation module: Calculates the category confidence of the representative prediction box of each cluster obtained by the clustering module, and calculates the probability that its category prediction is correct, which is used as the category confidence. Prediction box filtering module: Based on the category confidence obtained by the category confidence calculation module and the statistical information of the detection confidence within each cluster obtained by the clustering module, calculate the probability that each prediction box belongs to the correct prediction, retain prediction boxes with a probability greater than a preset threshold, and discard prediction boxes with a probability less than the preset threshold.

4. An electronic terminal, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it is used to perform the target detection post-processing method based on information fusion as described in claim 1.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, this program is used to perform the target detection post-processing method based on information fusion as described in claim 1.

Images