An abnormality marking content repairing method, device and electronic equipment
By combining overlap evaluation metrics and confidence levels, abnormal annotations are automatically filtered and repaired, solving the problem of training data quality degradation caused by annotation anomalies and improving the model's predictive performance and consistency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG GEELY HLDG GRP CO LTD
- Filing Date
- 2024-11-15
- Publication Date
- 2026-07-07
AI Technical Summary
In existing technologies, abnormal annotations lead to a decline in training data quality, affecting model performance and generalization ability. Furthermore, manual review is time-consuming, labor-intensive, and inconsistent, while automatic verification methods are ineffective.
By acquiring the original and predicted annotation content, and using multiple overlap evaluation indicators and confidence levels, the system automatically filters out annotation boxes with low confidence levels and retains those with high confidence levels, thus forming a repair result for mixed annotation content.
It enables automatic and accurate repair of abnormal labeled content, improves the quality of training data, and enhances the predictive performance and consistency of the model.
Smart Images
Figure CN119599913B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of machine learning, and in particular to a method, apparatus, and electronic device for repairing anomalously labeled content. Background Technology
[0002] With the rapid development of artificial intelligence and deep learning, building high-performance models increasingly relies on rich and accurate training data. In supervised learning frameworks, especially in semi-supervised and fully supervised learning scenarios, training datasets not only contain the raw input data but also include finely labeled annotations. These annotations serve as crucial guidance for model learning, enabling the model to understand data features and make predictions.
[0003] However, whether the annotations are added manually or by an automated annotation system, omissions and errors are inevitable. These abnormal annotations directly reduce the quality of the training data, thereby affecting the model's performance and generalization ability, and may lead to poor performance of the model in real-world applications. Summary of the Invention
[0004] In view of this, this application provides a method, apparatus, and electronic device for repairing abnormally labeled content, in order to solve the defects existing in related technologies. The technical solution of this application is as follows:
[0005] According to an embodiment of the first aspect of this application, a method for repairing abnormally labeled content is provided, comprising:
[0006] The original annotation content of the original image and the predicted annotation content output by the target model when predicting the original image are obtained. The original annotation content and the predicted annotation content are then superimposed to obtain mixed annotation content. The target model is trained based on the original image and the original annotation content. The original annotation content and the predicted annotation content each include annotation information for objects in the original image. The annotation information of the original annotation content includes at least the original bounding box and its confidence score. The annotation information of the predicted annotation content includes at least the predicted bounding box and its confidence score. The original annotation content contains anomalies.
[0007] For multiple overlap evaluation indicators corresponding to the original and predicted bounding boxes, an overlap threshold corresponding to each overlap evaluation indicator is determined, wherein the overlap threshold corresponding to any overlap evaluation indicator is determined based on the original and predicted annotation content.
[0008] Take any two annotation boxes from all the annotation boxes in the mixed annotation content as a group of annotation boxes, and if any group of annotation boxes meets at least one overlap threshold, delete the annotation box with the relatively low confidence in any group of annotation boxes, and determine the mixed annotation content after deletion as the repair result of the original annotation content.
[0009] According to an embodiment of the second aspect of this application, a device for repairing abnormally labeled content is provided, comprising:
[0010] An acquisition unit is used to acquire the original annotation content of the original image and the predicted annotation content output by the target model when predicting the original image, and to superimpose the original annotation content and the predicted annotation content to obtain mixed annotation content. The target model is trained based on the original image and the original annotation content. The original annotation content and the predicted annotation content respectively include annotation information for objects in the original image. The annotation information of the original annotation content includes at least the original bounding box and its confidence score. The annotation information of the predicted annotation content includes at least the predicted bounding box and its confidence score. The original annotation content contains anomalies.
[0011] The threshold determination unit is used to determine the overlap threshold corresponding to each overlap evaluation index for multiple overlap evaluation indices corresponding to the original annotation box and the predicted annotation box, wherein the overlap threshold corresponding to any overlap evaluation index is determined based on the original annotation content and the predicted annotation content.
[0012] The repair unit is used to combine any two annotation boxes in all the annotation boxes of the mixed annotation content as a group of annotation boxes, and delete the annotation boxes with relatively low confidence in any annotation box group if any annotation box group meets at least one overlap threshold, and determine the deleted mixed annotation content as the repair result of the original annotation content.
[0013] According to an embodiment of a third aspect of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in the first aspect.
[0014] According to an embodiment of the fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method as described in the first aspect.
[0015] According to an embodiment of the fifth aspect of this application, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the method described in the first aspect above.
[0016] In the technical solution provided in this application, a target model trained based on the original image and original annotation content can predict the original image and output predicted annotation content. The original annotation content includes the original bounding boxes and their confidence scores for objects in the original image, and the predicted annotation content includes the predicted bounding boxes and their confidence scores for objects in the original image. When abnormal original and predicted annotation content is obtained, the overlap thresholds of multiple overlap evaluation metrics corresponding to the bounding boxes contained in the annotation content can be determined. Simultaneously, superimposing the abnormal original and predicted annotation content yields hybrid annotation content, where each bounding box can form a bounding box combination with other bounding boxes. If any bounding box combination satisfies at least one determined overlap threshold, it indicates a high degree of overlap between the two positions. In this case, the bounding box with the lower confidence score in the combination is deleted. Correspondingly, after performing the above processing on each bounding box combination, the hybrid annotation content formed by the remaining bounding boxes is the repair result for the abnormal original annotation content.
[0017] As can be seen, this solution can automatically and accurately repair original annotation content with anomalies. Specifically, it can use multiple overlap evaluation indicators to judge the combination of annotation boxes contained in mixed annotation content, thereby accurately filtering out the combination of annotation boxes that meet the overlap requirements; at the same time, for each accurately filtered annotation box combination, the confidence level can be used to select the annotation boxes included therein, retaining the annotation boxes with higher confidence levels, further improving the accuracy of the repair results.
[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit the embodiments of this application. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.
[0020] Figure 1 This is a schematic diagram of training data shown in an exemplary embodiment of this application;
[0021] Figure 2 This is a flowchart illustrating an exemplary embodiment of the present application of a method for repairing abnormally labeled content;
[0022] Figure 3This is a schematic diagram of a mixed Gaussian distribution shown in an exemplary embodiment of this application;
[0023] Figure 4 This is a schematic diagram illustrating a hybrid annotation content according to an exemplary embodiment of this application;
[0024] Figure 5 This is an overall flowchart illustrating an exemplary embodiment of this application;
[0025] Figure 6 This is a schematic diagram of an exemplary embodiment of this application illustrating a device for repairing abnormally labeled content;
[0026] Figure 7 This is a schematic diagram of an electronic device illustrated in an exemplary embodiment of this application. Detailed Implementation
[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this application as detailed in the appended claims.
[0028] It should be noted that the steps of the corresponding methods in other embodiments are not necessarily performed in the order shown and described in this application. In some other embodiments, the methods may include more or fewer steps than those described in this application. Furthermore, a single step described in this application may be broken down into multiple steps in other embodiments; and multiple steps described in this application may be combined into a single step in other embodiments.
[0029] With the rapid development of artificial intelligence and deep learning technologies, building high-performance models increasingly relies on rich and accurate training data. In supervised learning frameworks, especially in semi-supervised and fully supervised learning scenarios, training datasets not only contain the original input data but also include finely labeled annotations. These annotations serve as crucial guidance for model learning, enabling the model to understand data features and make predictions.
[0030] However, whether the annotations are added manually or by an automated annotation system, omissions and errors are inevitable. These abnormal annotations directly reduce the quality of the training data, thereby affecting the model's performance and generalization ability, and may lead to poor performance of the model in real-world applications.
[0031] Currently, the detection and correction of data annotation anomalies mainly rely on manual review. This method is not only time-consuming and labor-intensive, but also easily influenced by subjective judgment, failing to guarantee consistency. Furthermore, as the size of training datasets continues to increase, relying entirely on manual checks becomes increasingly impractical. While some research has proposed automatic verification mechanisms based on rules or statistical models, these methods typically cannot guarantee effective correction of annotations containing anomalies.
[0032] To address the aforementioned issues, this application proposes an automated repair scheme for anomalous labeled content. This scheme involves filtering out overlapping label boxes with low confidence levels based on the degree of overlap and confidence scores of each box in the original and predicted label files, retaining only the boxes with higher confidence scores as the final repair result. The embodiments of this application will be described in detail below.
[0033] Figure 1 This is a schematic diagram of training data illustrated in an exemplary embodiment of this application. For example... Figure 1 As shown, the training data includes image 11 and labeled content 12.
[0034] Image 11 can be a real image captured by an image acquisition device or a synthetic image generated by a simulation system. Image 11 typically contains multiple labeled objects, and the selection and definition of these labeled objects are closely related to the actual scenario in which the model will be deployed, and should be important elements that reflect the scene information. For example, in the field of autonomous driving, labeled objects may include pedestrians, vehicles, and roads; while in medical image analysis, labeled objects may include lesions, organs, and cell tissues.
[0035] The annotation content 12 includes annotation information for each object in image 11. The annotation information for any object typically includes a bounding box 121, an annotation category 122, and a confidence level (which is usually not visually represented in the image).
[0036] like Figure 1 As shown, bounding box 121 is a rectangular boundary used to locate and define objects in an image, typically represented using coordinates. In object detection tasks, the model attempts to identify multiple objects in an image and draws a bounding box for each detected object to indicate its approximate location in the image. Label category 122 is classification information for objects in the image, used to indicate the object's content. In object detection tasks, each bounding box not only needs to locate the object but also indicate its category. For example, the bounding box might contain "person," "car," or "truck," etc.
[0037] When preparing training data for model training, bounding boxes and corresponding category labels can be created for each object in the image using manual or automatic annotation systems to aid model learning. For a given image, the corresponding annotations are typically added from the same annotation source; for example, the annotation information for all objects may be added entirely by an automatic annotation system or entirely by manual annotation. In this approach, the bounding boxes contained in the training data are referred to as the original bounding boxes.
[0038] During the model prediction phase, the model not only predicts and outputs bounding boxes and category labels for each detected object based on image 11, but also provides a confidence score for each predicted bounding box. In this scheme, the predicted bounding boxes are called predicted bounding boxes. The confidence score reflects the probability that the model believes the bounding box actually contains an object of the predicted category; it can also be understood as a quantitative indicator of the model's certainty about the prediction result. It is usually a value between 0 and 1, where 1 indicates that the model is very confident in the prediction result, and 0 indicates that it has no confidence at all. For bounding boxes added manually or by an automatic annotation system, their confidence scores are uniformly added, and the specific details will be explained in detail in subsequent embodiments.
[0039] It should also be noted that in practical applications, confidence can be further divided into classification confidence and detection confidence. Classification confidence represents the model's certainty about the class labels, while detection confidence comprehensively considers the accuracy of the class labels and bounding box positions, reflecting the model's confidence in the overall detection results. The confidence used in the technical solution of this application can be either classification confidence or detection confidence, and this application does not impose any limitation on this.
[0040] Figure 2 This is a flowchart illustrating an exemplary embodiment of this application of a method for repairing abnormally labeled content. See also: Figure 2 The method includes:
[0041] Step 201: Obtain the original annotation content of the original image and the predicted annotation content output by the target model on the original image, and superimpose the original annotation content and the predicted annotation content to obtain mixed annotation content. The target model is trained based on the original image and the original annotation content. The original annotation content and the predicted annotation content respectively include annotation information for objects in the original image. The annotation information of the original annotation content includes at least the original bounding box and its confidence score. The annotation information of the predicted annotation content includes at least the predicted bounding box and its confidence score. The original annotation content contains anomalies.
[0042] In one embodiment, the original image is an image that can be directly acquired, such as... Figure 1Image 11 in the training data shown can be either a real image captured by an image acquisition device or a synthetic image generated by a simulation system. The original annotation content refers to the annotations added manually or automatically to the corresponding original images when preparing the training data for model training. Specifically, it includes annotation information for objects in the original images, and the annotation information includes at least the original bounding boxes and their corresponding confidence scores. Related concepts can be found in [link to relevant documentation]. Figure 1 Example.
[0043] Given the original image and its original annotations, the target model can be trained to learn data features and make predictions based on the current original data and annotations. After a preset number of training iterations, the target model will possess a certain predictive ability and can then predict the original image, outputting predicted annotations. These predicted annotations also include annotation information for objects in the original image, at least including predicted bounding boxes and their corresponding confidence scores. The confidence scores of the bounding boxes included in the predicted annotations are the confidence scores directly output by the target model; related concepts can be found in [link to relevant documentation]. Figure 1 Example.
[0044] Here, the target model can be a model trained entirely based on the original image and the original annotation content. Of course, in order to save training resources and improve the model's prediction effect, the target model can also be a model obtained by transfer learning based on an existing model, and then trained on the original data and the original annotation content. This application does not impose any restrictions on this.
[0045] It is understandable that, since the model itself has a certain generalization ability and is trained on multiple datasets, although the target model is trained on the original image and the original annotation content, the predicted annotation content output by the model for the original image may not be the same as the original annotation content corresponding to the original image.
[0046] Since the original annotations were added manually or by an automated annotation system, various anomalies are inevitable, such as missing annotations, incorrect annotations, and inaccurate annotation boxes. Therefore, it is necessary to repair the original annotations that have anomalies.
[0047] In one embodiment, anomalies in the original labeled content can be determined as follows: The labeling quality score of the original labeled content can be determined based on the degree of deviation between the original and predicted labeled content; the higher the deviation, the lower the labeling quality score. After obtaining the labeling quality scores of multiple original labeled contents, a Gaussian mixture distribution is fitted to these labeling quality scores using a Gaussian mixture algorithm. The Gaussian mixture distribution typically contains multiple data clusters, each cluster usually modeled by a Gaussian distribution, such as... Figure 3As shown, this includes clusters 31, 32, and 33. The horizontal axis represents the annotation quality score, and the vertical axis represents the number of images with the corresponding annotation quality score. Then, the upper limit of a preset cluster in this Gaussian mixture distribution can be used as the quality score threshold corresponding to the annotation quality score, and it can be determined that each original annotation with a quality score lower than the quality score threshold is abnormal. For example, it can be... Figure 3 The upper limit of 0.1 for the first cluster X1 is used as the quality score threshold, and all original annotations with a quality score below 0.1 are considered to be anomaly. Specifically, the upper limit of the preset cluster can be determined based on the upper boundary of the cluster. First, each cluster has its corresponding parameters, including the mean. And variance, where mean This is the center point of the cluster. Then, it can be determined based on the mean. with standard deviation The boundary of the cluster can be calculated using the square root of the variance, for example, by using the mean. Add or subtract a certain number of standard deviations When using the mean plus or minus three standard deviations, the lower bound (lower limit) of the cluster is... -3 The upper bound (upper limit) of this cluster is +3 Within this boundary, approximately 99.7% of the data points within the cluster can be covered. This is the upper limit of the cluster. +3 This can be used as a quality score threshold; less than +3 The original annotation content corresponding to the annotation quality score is the original annotation content with anomalies.
[0048] It is understandable that different original annotations have different annotation quality scores. When fitting a Gaussian mixture distribution based on multiple annotation quality scores, using different original annotations will also cause corresponding changes in the Gaussian mixture distribution. Therefore, the quality score threshold determined based on the Gaussian mixture distribution is a threshold that matches the original annotations corresponding to the quality scores currently participating in the distribution fitting. Compared to the traditional method of using a fixed value as the quality score threshold, the solution in this embodiment can dynamically determine the quality score threshold that matches the original annotations, thereby more accurately filtering out original annotations with anomalies.
[0049] In one embodiment, the degree of deviation between the original labeled content and the predicted labeled content can be determined based on dimensional scores of multiple preset dimensions. The preset dimensions include at least one of the following: whether the object labeling is complete, whether the object content is correct, and whether the label box position is accurate. Incomplete object labeling in the original labeled content can be understood as a missing labeling issue, such as failing to identify a "person" in the original image. Incorrect object content in the original labeled content can be understood as a labeling error issue, such as labeling a "person" as a "car" in the original image. Inaccurate label box position in the original labeled content can be understood as the label box in the original labeled content failing to accurately mark the position of the object in the original image, such as the bounding area of the label box being much larger than the actual area of the object in the original image.
[0050] For any given original image, a scoring method from relevant technologies can be used to calculate sub-scores for the original image from the three preset dimensions mentioned above. These three sub-scores are then weighted to obtain the overall annotation quality score for the original image. During the weighted calculation process, the weight corresponding to each sub-score can be adjusted by technical personnel based on the actual situation and needs. For example, if technical personnel place greater emphasis on the completeness of object annotations, or if the main anomaly in the original annotation content is incomplete object annotations, then the weight corresponding to that preset dimension can be appropriately increased, allowing the corresponding sub-score to play a greater role in the weighted calculation, so that the overall annotation quality score can better reflect this issue.
[0051] Given both the original and predicted annotations, the two can be overlaid to obtain mixed annotations. Figure 4 This is a schematic diagram illustrating a hybrid annotation content as an exemplary embodiment of this application. For example... Figure 4 As shown, for object A in the original image, bounding box 41 is the bounding box for object A in the original annotation content, and its corresponding confidence score is the uniformly added confidence score; bounding box 42 is the bounding box for object A in the predicted annotation content, and its corresponding confidence score is the confidence score directly output by the target model. For object B in the original image, bounding box 43 is the bounding box for object B in the original annotation content, and its corresponding confidence score is the uniformly added confidence score; bounding box 44 is the bounding box for object B in the predicted annotation content, and its corresponding confidence score is the confidence score directly output by the target model.
[0052] Step 202: For the multiple overlap evaluation indicators corresponding to the original annotation box and the predicted annotation box, determine the overlap threshold corresponding to each overlap evaluation indicator, wherein the overlap threshold corresponding to any overlap evaluation indicator is determined based on the original annotation content and the predicted annotation content.
[0053] Step 203: Take any two annotation boxes from all the annotation boxes in the mixed annotation content as a group of annotation boxes, and delete the annotation boxes with relatively low confidence in any group of annotation boxes if any group of annotation boxes meets at least one overlap threshold. Then, determine the deleted mixed annotation content as the repair result of the original annotation content.
[0054] In one embodiment, the mixed-annotation content contains many redundant annotation boxes, for example, for... Figure 3 Object A in the diagram has both bounding boxes 41 and 42. In this case, this application can determine the degree of overlap between any two bounding boxes based on multiple overlap evaluation indicators, and further determine whether these two bounding boxes are a combination of bounding boxes targeting the same object. Each overlap evaluation indicator has a corresponding overlap threshold, and each overlap threshold is dynamically determined based on the original annotation content and the predicted annotation content.
[0055] Specifically, multiple overlap evaluation indicators can include the area intersection-over-union ratio (IoU) of different bounding boxes and the ratio of edge distances between different bounding boxes. The IoU is the ratio of the area of the overlapping portion of any two bounding boxes to the total area covered by those two bounding boxes. For example... Figure 3 The area of annotation box 41 is 1, the area of annotation box 42 is 1, and the area of their overlapping part is 0.8. Therefore, their intersection-union ratio is 0.8 ÷ 2 = 0.4. The edge distance ratio between different annotation boxes is determined based on the boundaries of the two annotation boxes. Specifically, for any annotation box, the distance between any two opposite edges of that annotation box can be calculated, and the corresponding distances between the two opposite edges of the other annotation box to be evaluated can be calculated accordingly. Then, based on the two distance values obtained and the minimum of the two distance values, the edge distance ratio between the two annotation boxes is calculated using the following formula:
[0056]
[0057] When the opposite sides are selected as the top and bottom edges of the annotation box, the distance between the opposite sides can be understood as the height of the annotation box; when the opposite sides are selected as the left and right edges of the annotation box, the distance between the opposite sides can be understood as the width of the annotation box. For example, if Figure 3 The width of annotation box 43 is 1.5, and the width of annotation box 44 is 1.2. Therefore, the ratio of the edge distance between these two annotation boxes can be determined as (1.5-1.2) / 1.2=0.25. Of course, technicians can define the calculation method for the edge distance ratio in other ways based on the performance of the target model; this application does not impose any restrictions on this.
[0058] If any two bounding boxes form a bounding box combination that satisfies at least one overlap threshold, then these two bounding boxes can be considered a bounding box combination for the same object. Specifically, bounding box combinations whose area intersection-union ratio is not less than (i.e., greater than) the intersection-union ratio threshold can be considered bounding box combinations for the same object, and bounding box combinations whose edge distance ratio is not greater than (i.e., less than) the distance ratio threshold can be considered bounding box combinations for the same object.
[0059] Understandably, using a single overlap evaluation index is equivalent to judging the overlap between two bounding boxes solely through a threshold comparison method, which cannot guarantee the accuracy of the judgment result. For example, when using only the area intersection-union ratio (IUGR) as the overlap evaluation index, and classifying two bounding boxes with an IUGR greater than a threshold as a combination of bounding boxes targeting the same object, if the IUGR threshold is set too high, even if the two bounding boxes are indeed a combination of bounding boxes targeting the same object, the inaccurate position of the bounding boxes contained in the original annotation content will result in a small area IUGR, thus failing to determine that the two bounding boxes are a combination of bounding boxes targeting the same object. If the IUGR threshold is set too low, even if the two bounding boxes are not a combination of bounding boxes targeting the same object, the partial overlap between the two bounding boxes will incorrectly identify them as a combination of bounding boxes targeting the same object. Therefore, the technical solution of this application uses multiple overlap evaluation indices to assess the overlap of bounding box combinations in different dimensions, which can more accurately identify whether a combination of bounding boxes targets the same object, making the judgment result more reliable.
[0060] As mentioned in the previous embodiments, when calculating the annotation quality score, sub-scores are calculated based on three dimensions: whether the object annotation is complete, whether the object content is correct, and whether the annotation box position is accurate. In the following embodiments, abnormal original annotation content will also be repaired to address the problems that may occur in these three dimensions.
[0061] In one embodiment, if any combination of annotation boxes satisfies at least one overlap threshold, the combination of annotation boxes can be considered as a combination of annotation boxes for the same object. At this time, the annotation boxes with relatively low confidence in the combination of annotation boxes can be deleted, and the deleted mixed annotation content is determined as the repair result of the original annotation content.
[0062] Specifically, when the original annotation content has incomplete object annotations, the mixed annotation content only includes the bounding boxes from the predicted annotation content. For example, if the original annotation content does not annotate object A in the original image, but the predicted annotation content output by the target model includes annotations for object A, then for object A, the mixed annotation content only includes the bounding boxes from the predicted annotation content. In this case, based on the method described in the above embodiments that judges based on multiple overlap evaluation indicators, the bounding box combination formed by this bounding box and any other bounding box included in the mixed annotation content is judged. It should not meet any of the various overlap thresholds. Therefore, it is determined that there is no bounding box combination for the same object in the mixed annotation content, and the subsequent steps of deleting bounding boxes with low confidence will not be executed. The bounding box for object A in the predicted annotation content is retained in the mixed annotation content.
[0063] Specifically, when the original annotation content has issues such as incorrect object content or inaccurate annotation box positions, the confidence level corresponding to the annotation boxes in the original annotation content can be set as the first confidence level, and the confidence level corresponding to the annotation boxes in the predicted annotation content can be set as the second confidence level. In this application's solution, technicians can choose whether to believe the predicted annotation content output by the target model based on the performance of the target model. Generally, after the target model has undergone multiple iterations of training and the prediction results have reached a good level, it is possible to unconditionally believe the predicted annotation content output by the target model. Therefore, when the mixed annotation content includes two annotation boxes for the same object, choosing to unconditionally believe the prediction results of the target model means deleting the annotation boxes included in the original annotation content. In this case, technicians can uniformly set the first and second confidence levels, ensuring that the first confidence level is less than the second confidence level. For example, the first confidence level can be set to 0.99999, and the second confidence level can be set to 1. Although the first confidence level is a high value, it is still less than the second confidence level. When the second confidence level is greater than the first confidence level, the label box with the lower confidence level will be deleted, which is the label box contained in the original label content. The label boxes in the predicted label content are retained in the mixed label content.
[0064] Of course, the target model may not perform well in the early stages of iterative training (especially in judging whether the object content is correct and whether the bounding box position is accurate). In this case, the first confidence level can be set slightly lower, such as 0.8, and the second confidence level can be directly adopted from the target model's output. In this situation, if the second confidence level given by the target model is greater than 0.8, which is a high value, it means that the target model is very confident in the judgment of the annotation results. In this case, when deleting the bounding box with lower confidence, the bounding box corresponding to the first confidence level will also be deleted, that is, the predicted annotation results output by the target model are chosen to be trusted. On the other hand, if the second confidence level given by the target model is not greater than 0.8, it means that the target model is not very confident in the judgment of the annotation results. In this case, when deleting the bounding box with lower confidence, the bounding box corresponding to the second confidence level will also be deleted, that is, the predicted annotation results output by the target model are chosen not to be trusted.
[0065] It should be noted that when determining whether the original labeled content has a corresponding problem in a preset dimension, a Gaussian mixture model can be used to fit the sub-scores corresponding to multiple original labeled content in each dimension to obtain the Gaussian mixture distribution for that dimension. The upper limit of the preset cluster is then used as the sub-score threshold for that dimension. If the score of any original labeled content in any dimension is less than the sub-score threshold, then that original labeled content is considered to have a corresponding problem in that dimension.
[0066] As can be seen from the above embodiments, the technical solution of this application can automatically and accurately repair the original annotation content with anomalies. Specifically, multiple overlap evaluation indicators can be used to judge the combination of annotation boxes contained in the mixed annotation content, thereby accurately selecting the combination of annotation boxes that meet the overlap requirements; at the same time, for each accurately selected combination of annotation boxes, the confidence level can be used to select the annotation boxes included therein, retaining the annotation boxes with higher confidence levels, further improving the accuracy of the repair results.
[0067] In one embodiment, it is understood that when multiple overlap evaluation indicators are used, and each overlap evaluation indicator has a corresponding overlap threshold, it is necessary to find the optimal solution for the combination of multiple overlap thresholds so that it is possible to more accurately identify whether the combination of bounding boxes is a combination of bounding boxes for the same object.
[0068] Specifically, a candidate threshold set containing multiple candidate threshold groups can be determined first, where each candidate threshold group includes candidate overlap thresholds corresponding to the multiple overlap evaluation indicators. When the overlap evaluation indicators include the area intersection-union ratio (IU) and the edge distance ratio, each candidate threshold group includes a candidate IU threshold and a candidate edge distance ratio threshold. For any candidate threshold group, the bounding box combinations are judged based on the candidate overlap thresholds included in the candidate threshold group. A trial repair is performed on threshold groups that meet at least one candidate overlap threshold in the candidate threshold group; that is, the bounding boxes with relatively low confidence are deleted from each bounding box combination, and the resulting mixed annotations are determined as the repair result for the original annotations. The repair result obtained here is the trial repair result using the candidate threshold group. Then, the annotation quality score of the trial repair result can be determined based on the deviation between the trial repair result corresponding to the candidate threshold group and the predicted annotation content, serving as the fitness score for the candidate threshold group. Since the candidate threshold set includes multiple candidate threshold groups, trial repair can be performed on each candidate threshold group, and its corresponding fitness score can be obtained. Based on this, the candidate overlap thresholds in the candidate threshold group with the highest fitness score can be used as the overlap thresholds corresponding to the corresponding overlap evaluation indexes. In other words, the candidate crossover and union ratio thresholds in the candidate threshold group with the highest fitness score can be used as the final crossover and union ratio thresholds, and the candidate distance ratio thresholds in the candidate threshold group with the highest fitness score can be used as the final distance ratio thresholds.
[0069] In one embodiment, the candidate threshold sets can be provided in advance by technicians based on experience and actual conditions, or they can be automatically calculated and output by a genetic algorithm. First, the genetic algorithm randomly generates an initial population, with each individual in the population representing a candidate threshold set. Then, the genetic algorithm measures the performance of individuals based on the fitness scores corresponding to each candidate threshold set and selects individuals with excellent performance (i.e., higher fitness scores) (i.e., candidate threshold sets) to enter the next generation. Subsequently, parent individuals are randomly selected for crossover operations to generate new offspring individuals, and the candidate overlap thresholds for some individuals are randomly changed. For example, a small random number is added to or subtracted from the candidate intersection-union threshold and / or candidate distance ratio threshold. The performance of the current offspring individual is then measured based on the fitness scores corresponding to each offspring individual. The genetic algorithm will repeat the above steps until a preset number of iterations is reached, or the fitness scores converge, meaning that no candidate threshold sets with higher fitness scores can be found. By employing a genetic algorithm to find the optimal candidate threshold set, the automation level of repairing abnormal original annotation content is further improved. This allows for more efficient determination of the overlap threshold, which accurately judges the degree of overlap between annotation boxes, thus resulting in more accurate repair results. Of course, in addition to genetic algorithms, other optimization algorithms can also be used to determine the overlap threshold, such as particle swarm optimization, differential evolution, etc. This application does not impose any restrictions on this approach.
[0070] In one embodiment, the original annotations can be updated to the repaired version of the original annotations, and the target model can be retrained based on the original image and the updated original annotations. After repairing the abnormal original annotations using the method described in the above embodiments, more accurate annotations can be obtained compared to before the repair. Retraining the model based on the original image and these more accurate annotations allows the model to learn more correct features, further improving the model's predictive performance.
[0071] In one embodiment, by updating the original annotation content corresponding to the original image multiple times and iteratively training the target model based on the original image and the updated original annotation content, the predictive performance of the target model after each round of iterative training can be evaluated. If the improvement in the predictive performance of the target model meets the stopping requirement in a preset number of iterative training rounds, the iterative training is stopped. Here, the stopping requirement can be set by a technician according to the actual situation and needs. For example, it can be determined whether the predictive performance of the target model has not improved within a consecutive preset number of rounds, and the iterative training can be stopped if there is no improvement within a consecutive preset number of rounds; it can also be determined whether the improvement in the predictive performance of the target model shows convergence, and the iterative training can be stopped if a preset convergence requirement is met. This application does not limit this.
[0072] Specifically, the dataset for each iteration includes the original image and its original annotations, which may be the original annotations obtained after restoration. A search space containing different hyperparameter combinations can be defined. The dataset can be divided into k subsets. Each time, k-1 subsets are used as the training set for training, and the remaining subset is used as the validation set for validation. This process is repeated k times, with each subset used as the validation set and the remaining k-1 subsets used as the training set for training. For each hyperparameter combination, its average performance index is calculated over k training iterations, thereby identifying hyperparameter combinations that perform well across different data subsets.
[0073] When training with any k-1 subsets as the training set, Dropout and early stopping techniques are used simultaneously to avoid overfitting. Dropout randomly ignores certain neurons in the target model at each training step, making the model less dependent on specific neurons and improving its generalization ability. Early stopping monitors the target model's performance on the validation set and stops training early when the model's performance on the validation set no longer shows improvement, ensuring the target model has optimal performance on the current validation set.
[0074] Then, using the selected optimal combination of hyperparameters, the model can be retrained on the entire training set, and Dropout and early stopping techniques can be applied again to consolidate the model's stability and improve the final performance. This series of steps not only optimizes the configuration of the target model but also ensures that the target model performs better when facing unknown data, thereby improving the overall efficiency and reliability of the model.
[0075] In one embodiment, after stopping iterative training of the target model, the original image corresponding to the current model and its original annotation content can be used as the target annotation data. At this point, the original annotation content included in the target annotation data has undergone multiple rounds of anomaly filtering and corresponding repairs, and compared to the initial original annotation content, it can more accurately reflect the location and category of each object in the original image. This target annotation data can be widely used for training other related models, and each related model can learn data features based on this target annotation data.
[0076] It should be noted that if only a small number of original annotations are abnormal, or if the repair results do not meet the preset requirements, these abnormal original annotations and their corresponding original images can be directly deleted. Even after directly removing the abnormal original annotations and their corresponding original images, the remaining original images and annotations will still be relatively accurate target annotation data.
[0077] The following will combine Figure 5 The overall process of this solution is described.
[0078] First, step 501 is executed to train the target model. The training data consists of a dataset composed of the original images and original annotations. This can be understood as the first major round of training for the model. In this step, the target model will be trained in multiple smaller rounds to obtain a target model with excellent performance for the current training data.
[0079] Step 502 is executed to evaluate the target model, which evaluates the predictive performance of the target model in the current large round.
[0080] Step 503 is executed to determine whether the prediction performance of the target model meets the stopping requirements. This step may specifically include steps 5031 and 5032. Step 5031 is used to determine whether the prediction performance of the target model is better than the prediction performance after the previous large training round (if the current large training round is the first large training round, this step is skipped), and if the current prediction performance of the target model is not better than the prediction performance after the previous large training round, step 5032 determines whether the prediction performance of the target model has not improved within consecutive preset rounds.
[0081] If the prediction performance of the target model does not improve within a consecutive preset number of rounds, proceed to step 504, using the original image and original annotation content (i.e., the training data corresponding to the current target model) as the target annotation data. Of course, at the same time, a target model with better prediction performance for the target annotation data can also be obtained.
[0082] If the prediction performance of the target model improves within consecutive preset rounds, proceed to step 505, use the current target model to predict the original image, and output the predicted annotation content.
[0083] By performing the above steps, the training and performance evaluation of the target model within a large round are completed, and the output of the current target model also provides the operational basis for the next steps.
[0084] Next, step 506 is executed to determine the annotation quality score. In this step, the annotation quality score of the current original annotation content is determined based on the sub-scores of the original annotation content and the predicted annotation content in three dimensions: whether the object annotation is complete, whether the object content is correct, and whether the annotation box position is accurate.
[0085] Step 507 involves fitting a Gaussian mixture distribution of the annotation quality scores and determining the quality score threshold. In this step, a Gaussian mixture algorithm is used to fit a Gaussian mixture distribution of the annotation quality scores corresponding to multiple original annotation contents, and the upper limit of the preset cluster in the obtained Gaussian mixture distribution is used as the quality score threshold corresponding to the annotation quality scores.
[0086] Execute step 508 to identify original annotation content with anomalies. After obtaining the quality score threshold, the original annotation content corresponding to annotation quality scores lower than the quality score threshold can be identified as original annotation content with anomalies.
[0087] By performing the above steps, the original annotation content with anomalies has been filtered out. The next step is to process this original annotation content with anomalies.
[0088] Perform step 509 to repair any abnormal original annotations, or delete the original annotations and their corresponding original images. The repair method can be found in the aforementioned embodiments and will not be repeated here.
[0089] Perform step 510 to update the training data. Here, the training data is the same as the original image and original annotations used in step 501. After repairing any abnormal original annotations, the abnormal original annotations are updated with the corresponding repair results, thus updating the training data. Similarly, directly deleting the abnormal original annotations also updates the training data.
[0090] Next, step 501 is executed again to train the target model for the next large round based on the updated training data. The above process is repeated until the target labeled data can be obtained through step 504.
[0091] Based on the method for repairing abnormally labeled content provided in this application, this application also provides a device for repairing abnormally labeled content, see [link to device]. Figure 6 ,include:
[0092] The acquisition unit 601 is used to acquire the original annotation content of the original image and the predicted annotation content output by the target model when predicting the original image, and to superimpose the original annotation content and the predicted annotation content to obtain mixed annotation content. The target model is trained based on the original image and the original annotation content. The original annotation content and the predicted annotation content respectively include annotation information for objects in the original image. The annotation information of the original annotation content includes at least the original bounding box and its confidence score. The annotation information of the predicted annotation content includes at least the predicted bounding box and its confidence score. The original annotation content has anomalies.
[0093] The threshold determination unit 602 is used to determine the overlap threshold corresponding to each overlap evaluation index for multiple overlap evaluation indices corresponding to the original annotation box and the predicted annotation box, wherein the overlap threshold corresponding to any overlap evaluation index is determined based on the original annotation content and the predicted annotation content.
[0094] The repair unit 603 is used to combine any two annotation boxes in all the annotation boxes of the mixed annotation content as a group of annotation boxes, and delete the annotation boxes with relatively low confidence in any annotation box group if any annotation box group meets at least one overlap threshold, and determine the deleted mixed annotation content as the repair result of the original annotation content.
[0095] Optionally, determining that the original labeled content is abnormal includes:
[0096] The annotation quality score of the original annotation content is determined based on the degree of deviation between the original annotation content and the predicted annotation content, and the annotation quality score is negatively correlated with the degree of deviation.
[0097] For each of the original annotation content's annotation quality scores, a Gaussian mixture algorithm is used to fit the Gaussian mixture distribution of the annotation quality scores, and the upper limit of a preset cluster in the Gaussian mixture distribution is used as the quality score threshold corresponding to the annotation quality score.
[0098] It was determined that each original annotation content with a quality score lower than the aforementioned quality score threshold was abnormal.
[0099] Optionally, the threshold determination unit 602 is specifically used for:
[0100] Determine a candidate threshold set containing multiple candidate threshold groups, wherein any candidate threshold group includes candidate overlap thresholds corresponding to the multiple overlap evaluation indicators respectively;
[0101] For any candidate threshold group in the candidate threshold set, delete the label boxes with relatively low confidence in any combination of label boxes, and determine the deleted mixed label content as the repair result of the original label content; and determine the label quality score of the repair result based on the degree of deviation between the repair result of the original label content and the predicted label content, and use the determined label quality score as the fitness score of any candidate threshold group;
[0102] Each candidate overlap threshold in the candidate threshold group with the highest fitness score is used as the overlap threshold corresponding to the corresponding overlap evaluation index.
[0103] Optionally, determining the candidate threshold set, which includes multiple candidate threshold groups, includes: calculating and outputting the candidate threshold set using a genetic algorithm.
[0104] Optionally, determining the annotation quality score of the original annotation content based on the degree of deviation between the original annotation content and the predicted annotation content includes: determining the degree of deviation between the original annotation content and the predicted annotation content according to the dimensional scores of multiple preset dimensions, wherein the preset dimensions include at least one of the following: whether the object annotation is complete, whether the object content is correct, and whether the annotation box position is accurate.
[0105] Optionally, for any original image,
[0106] The confidence level in the original labeled content is the first confidence level, and the confidence level in the predicted labeled content is the second confidence level, wherein the first confidence level is less than the second confidence level.
[0107] Optionally, the multiple overlap evaluation indicators include the area intersection-union ratio of different bounding boxes and the edge distance ratio of different bounding boxes, wherein the overlap threshold corresponding to the area intersection-union ratio is the intersection-union ratio threshold, and the overlap threshold corresponding to the edge distance ratio is the distance ratio threshold.
[0108] Optionally, deleting the label boxes with relatively low confidence in any combination of label boxes if the combination of label boxes satisfies at least one overlap threshold includes:
[0109] If the area intersection-union ratio of any combination of bounding boxes is not less than the intersection-union ratio threshold, delete the bounding boxes with relatively low confidence in any combination of bounding boxes.
[0110] If the edge distance ratio of any combination of annotation boxes is not greater than the distance ratio threshold, delete the annotation boxes with relatively low confidence in any combination of annotation boxes.
[0111] Optionally, the device further includes:
[0112] The update unit 604 is used to update the original annotation content to the repair result of the original annotation content;
[0113] The retraining unit 605 is used to retrain the target model based on the original image and the updated original annotation content.
[0114] Optionally, the device further includes:
[0115] Evaluation unit 606 is used to evaluate the prediction performance of the target model after each round of iterative training, when the original annotation content corresponding to the original image is updated multiple times and the target model is iteratively trained based on the original image and the original annotation content after each update.
[0116] The stop judgment unit 607 is used to stop iterative training and use the original image and its original annotation content corresponding to the current model as target annotation data if the improvement of the prediction performance in the preset round of iterative training meets the stopping requirements.
[0117] Optionally, the original image includes at least one of the following: a real image captured by an image acquisition device, a synthetic image generated by a simulation system; and / or,
[0118] The original annotations for each original image include either manually annotated or automatically annotated content.
[0119] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0120] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0121] Accordingly, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method for repairing abnormal annotation content as described in any of the above embodiments.
[0122] See Figure 7At the hardware level, the electronic device includes a processor 702, an internal bus 704, a network interface 706, memory 708, and non-volatile memory 710, and may also include other hardware required for business operations. The processor 702 reads the corresponding computer program from the non-volatile memory 710 into the memory 708 and then runs it, forming a speed adjustment device at the logical level. Of course, in addition to software implementation, this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.
[0123] Accordingly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for repairing abnormally labeled content as described in any of the above embodiments.
[0124] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.
[0125] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0126] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0127] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0128] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.
[0129] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0130] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0131] The above description is merely a preferred embodiment of one or more embodiments of this application and is not intended to limit the scope of one or more embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this application should be included within the scope of protection of one or more embodiments of this application.
Claims
1. A method of repairing an abnormality-labeled content, characterized by, include: The original annotation content of the original image and the predicted annotation content output by the target model when predicting the original image are obtained. The original annotation content and the predicted annotation content are then superimposed to obtain mixed annotation content. The target model is trained based on the original image and the original annotation content. The original annotation content and the predicted annotation content each include annotation information for objects in the original image. The annotation information of the original annotation content includes at least the original bounding box and its confidence score. The annotation information of the predicted annotation content includes at least the predicted bounding box and its confidence score. The original annotation content contains anomalies. For multiple overlap evaluation indicators corresponding to the original and predicted bounding boxes, an overlap threshold corresponding to each overlap evaluation indicator is determined, wherein the overlap threshold corresponding to any overlap evaluation indicator is determined based on the original and predicted annotation content. Any two annotation boxes in all the annotation boxes of the mixed annotation content are combined into one annotation box group. If any annotation box group meets at least one overlap threshold, the annotation box with the relatively low confidence in any annotation box group is deleted, and the mixed annotation content after deletion is determined as the repair result of the original annotation content. Different overlap evaluation indicators are used to evaluate the overlap of the annotation box group in different dimensions.
2. The method of claim 1, wherein, The original annotation content was determined to be abnormal, including: The annotation quality score of the original annotation content is determined based on the degree of deviation between the original annotation content and the predicted annotation content, and the annotation quality score is negatively correlated with the degree of deviation. For each of the original annotation content's annotation quality scores, a Gaussian mixture algorithm is used to fit the Gaussian mixture distribution of the annotation quality scores, and the upper limit of a preset cluster in the Gaussian mixture distribution is used as the quality score threshold corresponding to the annotation quality score. It was determined that each original annotation content with a quality score lower than the aforementioned quality score threshold was abnormal.
3. The method of claim 2, wherein, The determination of the overlap threshold corresponding to each overlap evaluation index includes: Determine a candidate threshold set containing multiple candidate threshold groups, wherein any candidate threshold group includes candidate overlap thresholds corresponding to the multiple overlap evaluation indicators respectively; For any candidate threshold group in the candidate threshold set, delete the label boxes with relatively low confidence in any combination of label boxes, and determine the deleted mixed label content as the repair result of the original label content; and determine the label quality score of the repair result based on the degree of deviation between the repair result of the original label content and the predicted label content, and use the determined label quality score as the fitness score of any candidate threshold group. Each candidate overlap threshold in the candidate threshold group with the highest fitness score is used as the overlap threshold corresponding to the corresponding overlap evaluation index.
4. The method according to claim 3, characterized in that, Determining a candidate threshold set containing multiple candidate threshold groups includes: calculating and outputting the candidate threshold set using a genetic algorithm.
5. The method according to claim 2, characterized in that, The step of determining the annotation quality score of the original annotation content based on the degree of deviation between the original annotation content and the predicted annotation content includes: determining the degree of deviation between the original annotation content and the predicted annotation content according to the dimensional scores of multiple preset dimensions, wherein the preset dimensions include at least one of the following: whether the object annotation is complete, whether the object content is correct, and whether the annotation box position is accurate.
6. The method according to claim 1, characterized in that, For any original image, The confidence level in the original labeled content is the first confidence level, and the confidence level in the predicted labeled content is the second confidence level, wherein the first confidence level is less than the second confidence level.
7. The method according to claim 1, characterized in that, The multiple overlap evaluation indicators include the area intersection-union ratio of different bounding boxes and the edge distance ratio of different bounding boxes, wherein the overlap threshold corresponding to the area intersection-union ratio is the intersection-union ratio threshold, and the overlap threshold corresponding to the edge distance ratio is the distance ratio threshold.
8. The method according to claim 7, characterized in that, The step of deleting the label boxes with relatively low confidence in any combination of label boxes, provided that any combination of label boxes meets at least one overlap threshold, includes: If the area intersection-union ratio of any combination of bounding boxes is not less than the intersection-union ratio threshold, delete the bounding boxes with relatively low confidence in any combination of bounding boxes. If the edge distance ratio of any combination of annotation boxes is not greater than the distance ratio threshold, delete the annotation boxes with relatively low confidence in any combination of annotation boxes.
9. The method according to claim 1, characterized in that, Also includes: Update the original annotation content to the repaired result of the original annotation content; The target model is retrained based on the original image and the updated original annotations.
10. The method according to claim 9, characterized in that, Also includes: The prediction performance of the target model after each round of iterative training is evaluated by updating the original annotation content corresponding to the original image multiple times and iterating the target model based on the original image and the original annotation content after each update. If the improvement in prediction performance in the preset rounds of iterative training meets the stopping requirement, then the iterative training is stopped and the original image and its original annotation content corresponding to the current model are used as the target annotation data.
11. The method according to claim 1, characterized in that, The original image includes at least one of the following: a real image captured by an image acquisition device, a synthetic image generated by a simulation system; and / or, The original annotations for each original image include either manually annotated or automatically annotated content.
12. A device for repairing abnormally labeled content, characterized in that, include: An acquisition unit is used to acquire the original annotation content of the original image and the predicted annotation content output by the target model when predicting the original image, and to superimpose the original annotation content and the predicted annotation content to obtain mixed annotation content. The target model is trained based on the original image and the original annotation content. The original annotation content and the predicted annotation content respectively include annotation information for objects in the original image. The annotation information of the original annotation content includes at least the original bounding box and its confidence score. The annotation information of the predicted annotation content includes at least the predicted bounding box and its confidence score. The original annotation content contains anomalies. The threshold determination unit is used to determine the overlap threshold corresponding to each overlap evaluation index for multiple overlap evaluation indices corresponding to the original annotation box and the predicted annotation box, wherein the overlap threshold corresponding to any overlap evaluation index is determined based on the original annotation content and the predicted annotation content. The repair unit is used to combine any two annotation boxes in all the annotation boxes of the mixed annotation content as a single annotation box combination, and delete the annotation box with relatively low confidence in any annotation box combination if any annotation box combination meets at least one overlap threshold, and determine the deleted mixed annotation content as the repair result of the original annotation content; wherein, different overlap evaluation indicators are used to evaluate the overlap of the annotation box combination in different dimensions.
13. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-11 by executing the executable instructions.
14. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-11.
15. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-11.