A sample mining method, device and storage medium based on target detection

Abstract

The present invention discloses a sample mining method, device and storage medium based on target detection, the method comprising: using a pre-trained target detection model to predict each unlabeled sample in an unlabeled sample set to obtain a prediction result; forming a standard Gaussian heat map and a mask map according to the high confidence peak center points of all categories in the heat map in the prediction result and their width and height attributes; evaluating the heat map in the prediction result according to the mask map and the standard Gaussian heat map to obtain an evaluation result of each unlabeled sample; and screening high-value samples in the unlabeled sample set according to the evaluation result. By implementing the present invention, high-quality samples can be mined based on the existing output characteristics of the target detection model itself, while reducing the annotation cost, without adding a special scoring module or scoring loss function to the network, which greatly reduces the workload.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a sample mining method, device and storage medium based on target detection. Background Art

[0002] In recent years, deep learning technology has made rapid progress and played an indelible role in technological innovation in the fields of vision, speech, etc. However, deep learning is highly dependent on big data and requires a large amount of data supply to optimize a large number of parameters so that the model can learn how to extract high-quality features. However, most deep learning tasks require a large number of high-quality labeled data sets, and obtaining these high-quality labeled data sets requires a lot of manpower. Therefore, how to use the least amount of samples to mine "high-information" data from unlabeled data sets, while maximizing the performance benefits of the model and reducing the workload of labeling, is currently a significant research hotspot.

[0003] At present, there are some evaluation criteria for determining the amount of information contained in an image, but these methods are often applied to binary or multi-classification models, but not to target detection models. Because the target detection task focuses more on the amount of information at the target instance level rather than the classification information of the entire image, a targeted high-value sample mining method is needed to determine the information content of the image based on the value score of the target instance in the image. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a sample mining method, device and storage medium based on target detection to solve the technical problem that the existing sample mining method for binary or multi-classification models cannot be used in sample mining for target detection tasks in the prior art.

[0005] The technical solution proposed by the present invention is as follows:

[0006] A first aspect of an embodiment of the present invention provides a sample mining method based on target detection, comprising: using a pre-trained target detection model to predict each unlabeled sample in an unlabeled sample set to obtain a prediction result; forming a standard Gaussian heat map and a mask map based on the high-confidence peak center points of all categories in the heat map in the prediction result and their width and height attributes; evaluating the heat map in the prediction result based on the mask map and the standard Gaussian heat map to obtain an evaluation result for each unlabeled sample; and screening high-value samples in the unlabeled sample set based on the evaluation results.

[0007] Optionally, forming a standard Gaussian heat map and a mask map based on the high-confidence peak center points and their width and height attributes of all categories in the heat map in the prediction result includes: determining the high-confidence center point based on the peak point of the heat map in the prediction result and a preset threshold; forming a standard Gaussian heat map based on the high-confidence center point and its width and height attributes in the prediction result; and forming a mask map based on the relationship between the Gaussian value of each position in the standard Gaussian heat map and a preset Gaussian threshold.

[0008] Optionally, determining the high confidence center point based on the peak point of the heat map in the prediction result and a preset threshold includes: obtaining the peak point on the heat map of each category based on a pooling operation; filtering the peak points according to the preset threshold to obtain the high confidence center point.

[0009] Optionally, the heat map in the prediction result is evaluated according to the mask map and the standard Gaussian heat map to obtain an evaluation result of each unlabeled sample, including: calculating a first distribution according to the mask map and the heat map of each unlabeled sample; calculating a second distribution according to the mask map and the standard Gaussian heat map; and calculating the evaluation result of each unlabeled sample according to the distance between the first distribution and the second distribution.

[0010] Optionally, screening the high-value samples in the unlabeled sample set according to the evaluation results includes: calculating the average of the difference between the first probability and the second probability of all high-confidence center points to obtain an edge evaluation result; and screening the high-value samples in the unlabeled sample set according to the evaluation results and the edge evaluation results.

[0011] Optionally, after screening the high-value samples in the unlabeled sample set according to the evaluation results, the method further includes: updating the unlabeled sample set and the labeled sample set according to the screened high-value samples; re-executing the training of the target detection model, the prediction of the unlabeled sample set, the formation of the standard Gaussian heat map and the mask map, the calculation of the evaluation results, and the sorting and selection of the unlabeled samples in sequence according to the updated labeled sample set, until it is determined that the stopping condition is met, and then stopping the training of the target detection model.

[0012] Optionally, the target detection model is a CenterNet target detection model; and the evaluation result is calculated by KL divergence, JS divergence or Wasserstein distance.

[0013] The second aspect of an embodiment of the present invention provides a sample mining device based on target detection, including: a prediction module, used to use a pre-trained target detection model to predict each unlabeled sample in an unlabeled sample set to obtain a prediction result; a branch calculation module, used to form a standard Gaussian heat map and a mask map based on the high confidence peak center points of all categories in the heat map in the prediction result and their width and height attributes; an evaluation module, used to evaluate the heat map in the prediction result based on the mask map and the standard Gaussian heat map to obtain an evaluation result for each unlabeled sample; a screening module, used to screen high-value samples in the unlabeled sample set according to the evaluation result.

[0014] A third aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the sample mining method based on target detection as described in the first aspect of the embodiment of the present invention and any one of the first aspects.

[0015] A fourth aspect of an embodiment of the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the sample mining method based on target detection as described in the first aspect of the embodiment of the present invention and any one of the first aspects.

[0016] The technical solution provided by the present invention has the following effects:

[0017] The sample mining method, device and storage medium based on target detection provided by the embodiments of the present invention form a standard Gaussian heat map and a mask map based on the high confidence peak center points and their width and height attributes of all categories in the heat map in the prediction results of the target detection model, and evaluate the heat map based on the mask map and the standard Gaussian heat map to obtain an evaluation result; thus, the sample mining method mines high-quality unlabeled samples based on the existing output characteristics of the target detection model itself, while reducing the number of labeled samples and thus reducing the data labeling cost, without the need to add a special scoring module or scoring loss function to the network, greatly reducing the workload.

[0018] The target detection-based sample mining method provided by the embodiment of the present invention calculates the evaluation result of each unlabeled sample through the Gaussian heat map evaluation strategy. At the same time, the edge evaluation result of each unlabeled sample is calculated by calculating the average value of the difference between the first probability and the second probability of all high-confidence center points. When screening samples, high-value samples in the unlabeled sample set are comprehensively screened based on the evaluation results and the edge evaluation results. The accuracy of judging the amount of sample information is greatly improved. In addition, when screening samples, a larger number of high-quality samples are obtained by cyclically updating the calculation method, meeting the demand for the number of samples. At the same time, the cyclic process gradually improves the performance of the CenterNet detection network, minimizing the data workload while maximizing the performance benefits of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 is a flow chart of a sample mining method based on target detection according to an embodiment of the present invention;

[0021] Figure 2 is a structural block diagram of a target detection model in a sample mining method based on target detection according to an embodiment of the present invention;

[0022] Figure 3 is a flow chart of a sample mining method based on target detection according to another embodiment of the present invention;

[0023] Figure 4 is a Gaussian thermodynamic diagram of a target center point of a sample mining method based on target detection according to an embodiment of the present invention;

[0024] Figure 5 is a flow chart of a sample mining method based on target detection according to another embodiment of the present invention;

[0025] Figure 6 is a flow chart of a sample mining method based on target detection according to another embodiment of the present invention;

[0026] Figure 7 is a structural block diagram of a sample mining device based on target detection according to an embodiment of the present invention;

[0027] Figure 8 is a schematic diagram of the structure of a computer-readable storage medium provided according to an embodiment of the present invention;

[0028] Fig. 9 is a schematic structural diagram of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0030] As shown in the background technology, the labeling operation for training samples is particularly important in the implementation process of target detection technology. However, in existing target detection technology, the labeled resources are often limited, and there is a large amount of unlabeled data that is not utilized. Usually, it takes a lot of time and manual methods to annotate each target in the image, which makes the whole process very time-consuming, difficult and costly. At the same time, existing sample mining methods are often applied to binary or multi-classification models, but target detection models are not applicable. Because the target detection task pays more attention to the amount of information at the target instance level rather than the classification information of the entire image.

[0031] In view of this, an embodiment of the present invention provides a sample mining method based on target detection to solve the technical problems in the prior art that manual labeling of samples is difficult and costly and the existing sample mining methods for binary or multi-classification models cannot be used.

[0032] According to an embodiment of the present invention, a sample mining method based on target detection is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] In this embodiment, a sample mining method based on target detection is provided, which can be used in electronic devices such as computers, mobile phones, tablet computers, etc. Figure 1 is a flow chart of a sample mining method based on target detection according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0034] Step S101: using a pre-trained target detection model to predict each unlabeled sample in an unlabeled sample set to obtain a prediction result.

[0035] Since the sample mining method is mainly aimed at sample mining of target detection tasks, it is necessary to obtain the information of sample instance level. Therefore, a target detection model can be used to mine the sample value in the unlabeled sample set to obtain high-value samples in the unlabeled sample set.

[0036] In the target detection model, the CenterNet target detection model has a heat map prediction branch, which can output the probability distribution of the center point of each category of targets. By observing these center point heat maps, it can be found that the lower the center energy of the center point heat map, the lower the reliability of target classification and positioning. This means that there are many targets in the sample or the amount of information contained in the sample is large, making it impossible for the target detection model to accurately determine the category to which the sample belongs. Therefore, the information value of the sample is high and it is more mineable. Using this feature of the CentenNet target detection model, the information content of unlabeled image data can be scored, and more unlabeled data with high information value can be mined. While reducing the amount of data annotation, the detection performance of CenterNet can be improved simultaneously.

[0037] Specifically, when using the target detection model for sample mining, the CenterNet target detection model can be trained with the labeled sample set to obtain a pre-trained target detection model, and then the pre-trained target detection model is used to predict the unlabeled sample set, that is, each unlabeled sample in the unlabeled sample set is input into the pre-trained target detection model to obtain the prediction result of each unlabeled sample. Specifically, Figure 2 As shown in the figure, for the CenterNet target detection model, when predicting the unlabeled sample set, the prediction result is obtained mainly through the feature extraction module, the multi-scale feature fusion model and the prediction module, among which the feature extraction module is used to extract features from the input data, the multi-scale fusion module fuses the multi-scale features extracted from the feature extraction module, and the prediction module predicts according to the fused multi-scale features to obtain the prediction result.

[0038] Step S102: forming a standard Gaussian heat map and a mask map according to the high confidence peak center points of all categories in the heat map in the prediction result and their width and height attributes.

[0039] When the pre-trained target detection model is used to predict each unlabeled sample in the unlabeled sample set, the prediction results containing three branches will be output. Specifically, Figure 2 As shown, the prediction results of the three branches include the center point class heatmap prediction branch (heatmap, hm), the center point width and height attribute prediction branch (wh), and the center point downsampling offset error prediction branch (offset); after determining the prediction results of the three branches, the position of the object detection box can be determined.

[0040] Among them, the center point class heat map prediction branch is the core output module in the target detection model. The function of this module is to output the center point of the object and the class of the object. In the prediction of this module, the center of the object is predicted as a point (that is, the point is set to 1 and the rest are 0). However, in the actual training process, a single point is difficult to learn. Therefore, in the actual output, a two-dimensional Gaussian distribution is calculated with the center point of the object as the center of the circle, so that the model changes from learning the position of a point to regressing an entire circular area. Setting a Gaussian distribution, in addition to making the algorithm easier to optimize, can also expand the area of ​​positive samples and reduce the problem of imbalance.

[0041] The center point width and height attribute prediction branch can output the width and height of the object frame. The center point downsampling offset error prediction branch is used to further correct the position of the center point of the object. The output size of the heatmap branch is only 1 / 16 of the original image, and the length and width are only 1 / 4 respectively, that is, one pixel on the feature map can correspond to 16 points in the original image. Therefore, in order to further "correct" the center point position of the object, the output of this branch is the same as the center point width and height attribute prediction branch, and continues to output the X offset and Y offset at each pixel.

[0042] Based on the above content, it can be known that the hm branch is used to predict the center heat map of targets of different categories, and the predictions of targets of different categories are distributed in different feature channel layers. The wh branch is used to predict the width and height attributes of the center point of the potential target. Therefore, when the pre-trained target detection model is used to predict each unlabeled sample, a heat map of the category to which each target on the unlabeled sample image belongs will be output in the hm branch. For example, in this embodiment, a total of five categories are predicted for the unlabeled sample, and the unlabeled sample includes three targets, then heat maps of five categories will be output, and the heat map of each category will show that each target in the unlabeled sample is predicted to be a Gaussian distribution of the category, that is, three Gaussian distributions or three center points will be displayed on the heat map of each category. At the same time, in the wh branch, the width and height attributes of each center point on the heat map of each category will be output.

[0043] Based on the high confidence peak center points of all categories in the heat map output by the hm branch and the width and height attributes of each center point output by the wh branch, a standard Gaussian heat map of each category can be formed, and the standard Gaussian heat map can be calculated based on each width and height attribute. The formed standard Gaussian heat map can be used as a standard for subsequent evaluation of the heat map of unlabeled samples. At the same time, based on the relationship between the standard Gaussian heat map and the preset Gaussian threshold, a mask map can also be formed as a mask for subsequent evaluation of the heat map.

[0044] Step S103: Evaluate the heat map in the prediction result according to the mask image and the standard Gaussian heat map to obtain an evaluation result of each unlabeled sample.

[0045] In one embodiment, when the heat map of unlabeled samples is evaluated using a standard Gaussian heat map and a mask map, the heat maps of all categories predicted for the unlabeled samples can be taken as a whole, the corresponding standard Gaussian heat maps of all categories can be taken as a whole, and the corresponding mask maps of all categories can be taken as a whole; then the mask map and the heat map of each unlabeled sample are multiplied to obtain a first distribution, the mask map and the standard Gaussian heat map are multiplied to obtain a second distribution, and the evaluation result is obtained by calculating the distance between the first distribution and the second distribution.

[0046] When calculating the distance between the first distribution and the second distribution, any one of KL divergence, JS divergence or Wasserstein distance can be used. The calculation formula of KL divergence, JS divergence or Wasserstein distance can refer to the calculation formula currently used, which will not be repeated here.

[0047] Step S104: Filter high-value samples in the unlabeled sample set according to the evaluation results. Since the evaluation result calculated by any one of KL divergence, JS divergence or Wasserstein distance is a specific numerical value, the specific numerical values ​​corresponding to the evaluation results calculated for each unlabeled sample can be sorted to filter out a certain number of high-value samples with high scores.

[0048] In addition, it should be noted that although the high-value samples are mainly obtained based on the prediction mining of the pre-trained target detection model, the high-value samples can also be applied to other models different from the target detection model for target detection. That is, the high-value samples can be used not only in the CenterNet target detection model, but also in other target detection methods.

[0049] The sample mining method based on target detection provided by the embodiment of the present invention forms a standard Gaussian heat map and a mask map based on the high confidence peak center points and their width and height attributes of all categories in the heat map in the prediction results of the target detection model, and evaluates the heat map based on the mask map and the standard Gaussian heat map to obtain an evaluation result; thus, the sample mining method mines high-quality unlabeled samples based on the existing output characteristics of the target detection model itself, while reducing the number of labeled samples and thus reducing the data labeling cost, without the need to add a special scoring module or scoring loss function to the network, greatly reducing the workload.

[0050] In one embodiment, if Figure 3As shown, a standard Gaussian heat map and a mask map are formed according to the high confidence peak center points of all categories in the heat map in the prediction result, including the following steps:

[0051] Step S201: Determine a high confidence center point according to the peak point of the heat map in the prediction result and a preset threshold. Specifically, based on the above content, it can be known that for each unlabeled sample in the unlabeled sample set, after the prediction of the pre-trained target detection model, a heat map of multiple categories corresponding to each unlabeled sample will be output in the hm branch. At the same time, the heat map includes the center points predicted for multiple targets. Since the energy of some center points is particularly low, they may be pseudo center points and can be removed by filtering.

[0052] In one embodiment, a pooling operation may be used to obtain the peak point of each center point in each category heat map; then a preset confidence threshold is used to filter out all pseudo center points below the threshold, and retain high confidence center points. For example, a maxpool operation may be used to obtain the peak point of each category heat map to obtain multiple peak points. When the confidence threshold is preset, the confidence threshold may be lower than the conventional confidence threshold set by CenterNet to ensure more low-quality detection outputs.

[0053] Step S202: A standard Gaussian heat map is formed based on the high confidence center point and its width and height attributes in the prediction result. Through the filtering operation of the above step S201, multiple high confidence center points can be obtained for each category of heat map. At this time, the width and height attributes of multiple high confidence center points are obtained using the width and height attribute branch in the prediction result, and a standard Gaussian heat map is produced through the width and height attributes.

[0054] Among them, when making a standard Gaussian heat map, the following formula can be used to generate it:

[0055]

[0056] Among them, Y xyc represents the Gaussian kernel of the central heat map, exp represents the exponential function with the natural constant e as the base in higher mathematics, (x, y) is the pixel position of any point in the hm map (heat map of each category), is the coordinate of the center point of the true label (ground-truth) in the hm image. When the pixel position and the ground-truth center point coordinates are close to coincide, the Gaussian kernel output value is close to 1; when the pixel position and the ground-truth center point are very different, the Gaussian kernel output value is close to 0. is the “adaptive variance of target scale”, and its value is closely related to the size of the target, affecting the size and distribution of each point in the generated heat map. When the value is small, the Gaussian kernel output distribution is concentrated at the category peak, making the "bright spots" in the generated heat map smaller. When the value is larger, the "bright spots" in the generated heat map spread outward. In this formula, the adaptive variance of the target scale It can be calculated by converting the width and height attributes of the high confidence center point. Therefore, the standard Gaussian heat map corresponding to each category can be calculated by the above formula.

[0057] Moreover, based on the above formula, we can know that the hm branch outputs a heat map of all the potential target centers. The higher the energy of the heat map center, the higher the confidence of the target classification and location. Therefore, under the condition of a fixed target size, if the energy is more concentrated, it often indicates that the confidence of the target classification and location is higher. Figure 4 As shown, target 1 and target 2 have the same width and height, so Similarly, the standard Gaussian heat map generated by the above formula is shown in GT. It can be seen that compared with target 2, the heat map of target 1 is more divergent, indicating that the current detection model has a higher uncertainty in detecting the target, so images with more target 1 have a higher value for optimizing the current detection model.

[0058] Therefore, the standard Gaussian heat map generated above can be used as an evaluation standard for the heat map of each category. By comparing the two, the amount of information contained in the sample to be labeled can be judged, thereby screening out samples with higher value.

[0059] Step S203: A mask image is formed according to the relationship between the Gaussian value of each position in the standard Gaussian heat map and the preset Gaussian threshold. After the standard Gaussian heat map is formed, the relationship between the Gaussian value of each position and the preset Gaussian threshold is judged based on the standard Gaussian heat map. When it is greater than the preset Gaussian threshold, the value of the corresponding position is set to 1, otherwise it is set to 0. Among them, the preset Gaussian threshold can be 0 or a smaller value. After comparing and resetting each position, a mask image can be produced through the standard Gaussian heat map. Through the mask image, it is possible to focus only on the Gaussian area near the high confidence center point in each category heat map, while ignoring other low confidence background outputs.

[0060] In one embodiment, if Figure 5 As shown, screening high-value samples in the unlabeled sample set according to the evaluation results includes the following steps:

[0061] Step S301: Calculate the average value of the difference between the first probability and the second probability of all high confidence center points to obtain an edge evaluation result.

[0062] Specifically, in addition to the evaluation of the center point heat map sampling, considering the difference in confidence values ​​of each category of the predicted target at the same center point, the edge sampling strategy can also be comprehensively considered to mine the information content of the sample. Among them, for the edge sampling strategy, if the difference in the probability values ​​of the first two categories at the center point is smaller, it means that the current detection model is less clear about the target category classification at the center point, so the potential target contains more information. Therefore, the edge sampling strategy can also be used to determine the edge evaluation result.

[0063] For the edge evaluation result, it can be obtained by calculating the average value of the difference between the first probability and the second probability of all high confidence center points. The first probability is represented by the highest probability in the predicted category probability, and the second probability is represented by the second highest probability in the predicted category probability. Among them, when calculating the edge evaluation result, in order to reduce the amount of calculation, the average value of the difference between the first two predicted category probabilities (first probability and second probability) of all high confidence center points can be used to calculate the specific calculation formula of the edge evaluation result using the following formula:

[0064]

[0065] Among them, p1 represents the maximum category probability of the current center point, p2 represents the second largest category probability of the current center point, and N represents the number of all high confidence center points. The maximum category probability and the second largest category probability are specifically expressed as the probabilities of different categories predicted for the same target in the unlabeled sample; therefore, when calculating the edge evaluation result of an unlabeled sample, it is necessary to calculate the maximum category probability and the second largest category probability predicted for all targets in the sample to be labeled, and then calculate the difference through the above formula and average it to obtain the edge evaluation result of the unlabeled sample.

[0066] Step S302: Filter high-value samples in the unlabeled sample set according to the evaluation result and the edge evaluation result. Specifically, the evaluation result of each unlabeled sample is calculated by the Gaussian heat map evaluation strategy through the above step S103. At the same time, the edge evaluation result of each unlabeled sample is calculated through the above step 301. Therefore, the evaluation result and the edge evaluation result can be combined to obtain a total evaluation result, and then the total evaluation result is used to filter high-value samples in the unlabeled sample set.

[0067] Specifically, the overall evaluation result of the unlabeled samples can be expressed by the following formula:

[0068] v=α·v1+β·v2

[0069] Among them, v1 represents the evaluation result, v2 represents the edge evaluation result, α is the Gaussian heat map evaluation strategy, that is, the weight of the evaluation result, and β is the edge sampling strategy, that is, the weight of the edge evaluation result. The two weights can be determined in advance based on actual conditions.

[0070] In one embodiment, the unlabeled samples in the unlabeled sample set are sorted by the total evaluation results, and a preset number of samples, such as the first K samples, are taken as high-value samples. Although a certain number of high-value samples can be determined at this time, the number of samples obtained may be small. Therefore, more high-value samples can be obtained by cyclic updating.

[0071] In one embodiment, the cyclic process specifically includes: updating the unlabeled sample set and the labeled sample set according to the screened high-value samples; re-executing the training of the target detection model, the prediction of the unlabeled sample set, the formation of the standard Gaussian heat map and the mask map, the calculation of the evaluation results, and the sorting and selection of the unlabeled samples in sequence according to the updated labeled sample set, until it is determined that the stop condition is met, and then stopping the training of the target detection model

[0072] Specifically, after determining k high-value samples, they can be labeled and added to the labeled sample set, and deleted from the unlabeled sample set to update the unlabeled sample set and the labeled sample set. Then, the CenterNet target detection model is retrained using the updated labeled sample set, and the prediction of the unlabeled sample set, the formation of the standard Gaussian heat map and mask map, the calculation of the evaluation results and the edge evaluation results, and the sorting and selection of the unlabeled samples are re-performed to mine a new batch of high-quality samples to be labeled, and then the updated sample set is updated. Repeat the above steps until the detection performance of CenterNet reaches saturation on the data.

[0073] In one embodiment, if Figure 6As shown in the figure, the sample mining method based on target detection is implemented by the following process: a CenterNet target detection network model is trained based on the existing labeled sample set. The unlabeled samples of the unlabeled sample set are predicted based on the trained CenterNet detection model to obtain the prediction results; based on the prediction results, the maxpool operation is performed on the hm prediction branch to obtain the peak points of the heat map of each category. Through the preset confidence threshold, all pseudo center points below the threshold are filtered out, and the high confidence center points are retained; the width and height attributes of all high confidence center points are obtained on the output of the wh prediction branch, and a standard Gaussian heat map is produced through the width and height attributes. A mask map is produced through the standard Gaussian heat map. When producing the mask map, if the Gaussian value of a certain position is greater than 0, the value of the corresponding position is set to 1, otherwise it is set to 0. Based on the produced mask map, the JS divergence between the hm heat map and the standard Gaussian heat map is calculated as the score based on the Gaussian heat map evaluation strategy. The average value of the difference between the probability of the first two predicted categories of all high confidence center points is obtained as the edge evaluation result. The score based on the Gaussian heat map evaluation strategy and the edge evaluation result are weighted to obtain the total evaluation result. The first K unlabeled samples are selected from all the samples in the unlabeled sample set in descending order of the total evaluation result v value as high-value samples to be labeled. The K high-value samples to be labeled are manually labeled and added to the labeled sample set, while being removed from the unlabeled data set. The CenterNet target detection model is retrained using the updated labeled sample set, and the above steps are repeated to mine a new batch of high-quality samples to be labeled, and the sample set is updated again. This continues until the detection performance of CenterNet reaches saturation on the data.

[0074] The sample mining method based on target detection provided by the embodiment of the present invention calculates the evaluation result of each unlabeled sample through the Gaussian heat map evaluation strategy. At the same time, the edge evaluation result of each unlabeled sample is calculated by calculating the average value of the difference between the first probability and the second probability of all high-confidence center points. When screening samples, high-value samples in the unlabeled sample set are comprehensively screened based on the evaluation results and the edge evaluation results. The accuracy of judging the amount of sample information is greatly improved. In addition, when screening samples, a larger number of high-quality samples are obtained by cyclically updating the calculation method, meeting the demand for the number of samples. At the same time, the cyclic process gradually improves the performance of the CenterNet detection network, minimizing the data workload while maximizing the performance benefits of the model.

[0075] The target detection-based sample mining method provided by the embodiment of the present invention iteratively mines high-quality unlabeled samples based on the existing output characteristics of the CenterNet network itself, while reducing the number of labeled samples, thereby reducing the data labeling cost, without adding a special scoring module or scoring loss function to the network, greatly reducing the workload. Through this sample mining method, the performance of the CenterNet detection network will also be gradually improved during this iterative process, minimizing the data workload while maximizing the performance benefits of the model. In addition, the high-quality labeled data set mined by this sample mining method can also be applied to other target detection methods, and is not limited to the CenterNet detection algorithm.

[0076] The embodiment of the present invention also provides a sample mining device based on target detection, such as Figure 7 As shown, the device comprises:

[0077] The prediction module is used to use a pre-trained target detection model to predict each unlabeled sample in the unlabeled sample set to obtain a prediction result; the specific content can be found in the corresponding part of the above method embodiment, which will not be repeated here.

[0078] The branch calculation module is used to form a standard Gaussian heat map and a mask map according to the high confidence peak center points and their width and height attributes of all categories in the heat map in the prediction result; for specific contents, please refer to the corresponding part of the above method embodiment, which will not be repeated here.

[0079] The evaluation module is used to evaluate the heat map in the prediction result according to the mask map and the standard Gaussian heat map to obtain the evaluation result of each unlabeled sample; the specific content refers to the corresponding part of the above method embodiment, which will not be repeated here.

[0080] The screening module is used to screen the high-value samples in the unlabeled sample set according to the evaluation results. For details, please refer to the corresponding part of the above method embodiment, which will not be repeated here.

[0081] The target detection-based sample mining device provided by the embodiment of the present invention forms a standard Gaussian heat map and a mask map based on the high-confidence peak center points and their width and height attributes of all categories in the heat map in the prediction results of the target detection model, and evaluates the heat map based on the mask map and the standard Gaussian heat map to obtain an evaluation result; thus, the sample mining method mines high-quality unlabeled samples based on the existing output characteristics of the target detection model itself, while reducing the number of labeled samples and thus reducing the data labeling cost, without the need to add a special scoring module or scoring loss function to the network, greatly reducing the workload.

[0082] For a detailed description of the functions of the target detection-based sample mining device provided in an embodiment of the present invention, please refer to the description of the target detection-based sample mining method in the above embodiment.

[0083] The embodiment of the present invention also provides a storage medium, such as Figure 8 As shown, a computer program 601 is stored thereon, and when the instruction is executed by the processor, the steps of the sample mining method based on target detection in the above embodiment are implemented. The storage medium also stores audio and video stream data, feature frame data, interaction request signaling, encrypted data, and preset data size, etc. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0084] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0085] The embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, the electronic device may include a processor 51 and a memory 52, wherein the processor 51 and the memory 52 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0086] The processor 51 may be a central processing unit (CPU). The processor 51 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0087] The memory 52 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the corresponding program instructions / modules in the embodiment of the present invention. The processor 51 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 52, that is, to implement the sample mining method based on target detection in the above method embodiment.

[0088] The memory 52 may include a program storage area and a data storage area, wherein the program storage area may store an application required for operating the device and at least one function; the data storage area may store data created by the processor 51, etc. In addition, the memory 52 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 52 may optionally include a memory remotely arranged relative to the processor 51, and these remote memories may be connected to the processor 51 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0089] The one or more modules are stored in the memory 52 and when executed by the processor 51, perform the following steps: Figure 1 -6 is a sample mining method based on target detection in the embodiment shown in FIG.

[0090] For details of the above electronic equipment, please refer to Figures 1 to 6 The corresponding related descriptions and effects in the illustrated embodiments can be understood and will not be repeated here.

[0091] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A sample mining method based on target detection, characterized in that: include: Use the pre-trained target detection model to predict each unlabeled sample in the unlabeled sample set to obtain the prediction result; Forming a standard Gaussian heat map and a mask map according to the high confidence peak center points and center point width and height attributes of all categories in the heat map in the prediction result; Evaluating the heat map in the prediction result according to the mask image and the standard Gaussian heat map to obtain an evaluation result for each unlabeled sample; Filter high-value samples in the unlabeled sample set according to the evaluation results; The forming of a standard Gaussian heat map and a mask map according to the high confidence peak center points and center point width and height attributes of all categories in the heat map in the prediction result includes: Determining a high confidence center point according to a peak point of the heat map in the prediction result and a preset threshold; Forming a standard Gaussian heat map according to the high confidence center point and its width and height attributes in the prediction result; A mask image is formed according to the relationship between the Gaussian value at each position in the standard Gaussian heat map and a preset Gaussian threshold.

2. The sample mining method based on target detection according to claim 1, characterized in that: The step of determining a high confidence center point according to a peak point of a heat map in the prediction result and a preset threshold value includes: Obtain the peak points on the heat map of each category based on the pooling operation; The peak points are filtered according to the preset threshold to obtain a high confidence center point.

3. The sample mining method based on target detection according to claim 1, characterized in that: The step of evaluating the heat map in the prediction result according to the mask image and the standard Gaussian heat map to obtain the evaluation result of each unlabeled sample includes: Calculating a first distribution according to the mask image and a heat map of each unlabeled sample; Calculating a second distribution according to the mask image and the standard Gaussian heat map; The evaluation result of each unlabeled sample is obtained according to the distance calculation between the first distribution and the second distribution.

4. The sample mining method based on target detection according to claim 2, characterized in that: The screening of high-value samples in the unlabeled sample set according to the evaluation results includes: Calculate the average of the difference between the first probability and the second probability of all high confidence center points to obtain an edge evaluation result, wherein the first probability is represented as the highest probability in the predicted category probability, and the second probability is represented as the second highest probability in the predicted category probability; High-value samples in the unlabeled sample set are screened according to the evaluation result and the edge evaluation result.

5. The sample mining method based on target detection according to claim 1, characterized in that: After screening the high-value samples in the unlabeled sample set according to the evaluation results, the method further includes: Update the unlabeled sample set and the labeled sample set based on the screened high-value samples; According to the updated labeled sample set, the training of the target detection model, the prediction of the unlabeled sample set, the formation of the standard Gaussian heat map and the mask map, the calculation of the evaluation results, and the sorting and selection of the unlabeled samples are re-executed in sequence until the stopping condition is determined to be met, and the training of the target detection model is stopped.

6. The sample mining method based on target detection according to claim 1, characterized in that: The target detection model is a CenterNet target detection model; The evaluation result is calculated by KL divergence, JS divergence or Wasserstein distance.

7. A sample mining device based on target detection, characterized in that: include: A prediction module, used to use a pre-trained target detection model to predict each unlabeled sample in the unlabeled sample set to obtain a prediction result; A branch calculation module, used for forming a standard Gaussian heat map and a mask map according to the high confidence peak center points and center point width and height attributes of all categories in the heat map in the prediction result; An evaluation module, used to evaluate the heat map in the prediction result according to the mask map and the standard Gaussian heat map, and obtain an evaluation result of each unlabeled sample; A screening module, used to screen high-value samples in the unlabeled sample set according to the evaluation results; The forming of a standard Gaussian heat map and a mask map according to the high confidence peak center points and center point width and height attributes of all categories in the heat map in the prediction result includes: Determining a high confidence center point according to a peak point of the heat map in the prediction result and a preset threshold; Forming a standard Gaussian heat map according to the high confidence center point and its width and height attributes in the prediction result; A mask image is formed according to the relationship between the Gaussian value at each position in the standard Gaussian heat map and a preset Gaussian threshold.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the target detection-based sample mining method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the target detection-based sample mining method as described in any one of claims 1 to 6 by executing the computer instructions.

Images