Abnormal annotation content repairing method and device and electronic equipment

Abstract

The invention provides an abnormal annotation content repairing method and device and electronic equipment, and the method comprises the steps: obtaining the original annotation content of an original image and the predicted annotation content outputted by the prediction of the original image through a target model, and carrying out the superposition processing of the original annotation content and the predicted annotation content, and obtaining the mixed annotation content, the original annotation content comprises an original annotation box and a confidence coefficient thereof, the predicted annotation content comprises a predicted annotation box and a confidence coefficient thereof, and the original annotation content is abnormal; aiming at the plurality of coincidence degree evaluation indexes, determining a coincidence degree threshold value corresponding to each coincidence degree evaluation index; taking any two labeling boxes in all the labeling boxes of the mixed labeling content as a labeling box combination, and deleting the labeling boxes with relatively low confidence in any labeling box combination under the condition that any labeling box combination meets at least one overlap ratio threshold, and determining the deleted mixed annotation content as a repair result of the original annotation content.

Description

Technical Field

[0001] The present application relates to the field of machine learning, and in particular to a method, device and electronic device for repairing abnormal annotated content. Background Art

[0002] With the rapid development of artificial intelligence and deep learning, building high-performance models increasingly relies on rich and accurate training data. In the supervised learning framework, especially in semi-supervised and fully supervised learning scenarios, the training data set contains not only the original input data, but also the detailed annotations of these data. The annotation information serves as a key guide for model learning, enabling the model to understand the data features and make predictions.

[0003] However, whether the annotations are added manually or by the automatic annotation system, there will inevitably be anomalies such as missing annotations and annotation errors. These abnormal annotations directly reduce the quality of training data, which in turn affects the performance and generalization ability of the model, and may cause the model to perform poorly in practical applications. Summary of the invention

[0004] In view of this, the present application provides a method, device and electronic device for repairing abnormal annotation content to solve the defects existing in the relevant technology. The technical solution of the present application is as follows:

[0005] According to an embodiment of the first aspect of the present application, a method for repairing abnormally marked content is provided, comprising:

[0006] Acquire original annotation content of an original image and predicted annotation content output by a target model through prediction of the original image, and superimpose the original annotation content and the predicted annotation content to obtain mixed annotation content, wherein the target model is trained according to the original image and the original annotation content, the original annotation content and the predicted annotation content respectively include annotation information for objects in the original image, the annotation information of the original annotation content at least includes an original annotation box and its confidence, the annotation information of the predicted annotation content at least includes a predicted annotation box and its confidence, and the original annotation content has anomalies;

[0007] Determining, for a plurality of overlap evaluation indicators corresponding to the original annotation frame and the predicted annotation frame, an overlap threshold value corresponding to each overlap evaluation indicator, wherein the overlap threshold value corresponding to any overlap evaluation indicator is determined based on the original annotation content and the predicted annotation content;

[0008] Any two annotation boxes among all the annotation boxes of the mixed annotation content are taken as a annotation box combination, and when any annotation box combination satisfies at least one overlap threshold, the annotation boxes with relatively low confidence in any annotation box combination are deleted, and the deleted mixed annotation content is determined as a repair result of the original annotation content.

[0009] According to an embodiment of the second aspect of the present application, a device for repairing abnormally marked content is provided, including:

[0010] an acquisition unit, used to acquire original annotation content of an original image and predicted annotation content output by a target model through prediction of the original image, and to perform superposition processing on the original annotation content and the predicted annotation content to obtain mixed annotation content, wherein the target model is trained according to the original image and the original annotation content, the original annotation content and the predicted annotation content respectively include annotation information for objects in the original image, the annotation information of the original annotation content at least includes an original annotation box and its confidence, the annotation information of the predicted annotation content at least includes a predicted annotation box and its confidence, and the original annotation content has anomalies;

[0011] a threshold determination unit, configured to determine, for a plurality of overlap degree evaluation indicators corresponding to the original annotation frame and the predicted annotation frame, an overlap degree threshold value corresponding to each overlap degree evaluation indicator, wherein the overlap degree threshold value corresponding to any overlap degree evaluation indicator is determined based on the original annotation content and the predicted annotation content;

[0012] A repairing unit is used to treat any two annotation boxes in all the annotation boxes of the mixed annotation content as a combination of annotation boxes, and when any annotation box combination satisfies at least one overlap threshold, delete the annotation box with relatively low confidence in any annotation box combination, and determine the deleted mixed annotation content as a repair result of the original annotation content.

[0013] According to an embodiment of the third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0014] According to an embodiment of the fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] According to an embodiment of the fifth aspect of the present application, a computer program product is proposed, including a computer program / instruction, and when the computer program / instruction is executed by a processor, the method described in the first aspect is implemented.

[0016] In the technical solution provided by the present application, according to the target model obtained by training the original image and the original annotation content, the original image can be predicted and the predicted annotation content can be output. The original annotation content includes the original annotation box and its confidence for the object in the original image, and the predicted annotation content includes the predicted annotation box and confidence for the object in the original image. When the original annotation content and the predicted annotation content with abnormalities are obtained, the overlap threshold of multiple overlap evaluation indicators corresponding to the annotation boxes contained in the annotation content can be determined. At the same time, the original annotation content with abnormalities and the predicted annotation content are superimposed to obtain mixed annotation content, and each annotation box in the mixed annotation content can form a combination of annotation boxes with other annotation boxes. If any combination of annotation boxes meets at least one determined overlap threshold, it means that the overlap degree of the two positions is high. At this time, the annotation box with lower confidence in the combination is deleted. Correspondingly, after each combination of annotation boxes is processed as above, the mixed annotation content composed of the remaining annotation boxes is the repair result of the original annotation content with abnormalities.

[0017] It can be seen that this solution can automatically and accurately repair the original annotation content with abnormalities. Specifically, multiple overlap evaluation indicators can be used to judge the annotation box combinations contained in the mixed annotation content, so as to accurately screen out the annotation box combinations with the overlap degree meeting the requirements; at the same time, for each accurately screened annotation box combination, the confidence level can be used to select the annotation boxes included in it, and the annotation boxes with higher confidence levels can be retained, further improving the accuracy of the repair results.

[0018] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory and do not limit the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0020] Figure 1 is a schematic diagram of training data shown in an exemplary embodiment of the present application;

[0021] Figure 2 It is a flow chart of a method for repairing abnormal annotation content shown in an exemplary embodiment of the present application;

[0022] Figure 3is a schematic diagram of a mixed Gaussian distribution shown in an exemplary embodiment of the present application;

[0023] Figure 4 It is a schematic diagram of mixed annotation content shown in an exemplary embodiment of the present application;

[0024] Figure 5 It is an overall flow chart showing an exemplary embodiment of the present application;

[0025] Figure 6 is a schematic diagram of a device for repairing abnormally marked content shown in an exemplary embodiment of the present application;

[0026] Figure 7 It is a schematic diagram of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of the present application. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of the present application as detailed in the appended claims.

[0028] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this application. In some other embodiments, the steps included in the method may be more or less than those described in this application. In addition, a single step described in this application may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this application may also be combined into a single step for description in other embodiments.

[0029] With the rapid development of artificial intelligence and deep learning technologies, building high-performance models increasingly relies on rich and accurate training data. In the supervised learning framework, especially in semi-supervised and fully supervised learning scenarios, the training data set contains not only the original input data, but also the detailed annotations of these data. Annotation information serves as a key guide for model learning, enabling the model to understand data features and make predictions.

[0030] However, whether the annotations are added manually or by the automatic annotation system, there will inevitably be anomalies such as missing annotations and annotation errors. These abnormal annotations directly reduce the quality of training data, which in turn affects the performance and generalization ability of the model, and may cause the model to perform poorly in practical applications.

[0031] At present, the detection and correction of data annotation anomalies mainly rely on manual review, which is not only time-consuming and labor-intensive, but also easily affected by personal subjective judgment and cannot guarantee consistency. In addition, as the size of training data sets continues to increase, it becomes increasingly unfeasible to rely solely on manual inspection. In related technologies, although some studies have proposed some automatic verification mechanisms based on rules or statistical models, these methods usually cannot guarantee the repair effect of anomalies in annotations.

[0032] In order to solve the above problems, the present application proposes an automated repair solution for abnormal annotation content, that is, based on the overlap degree and confidence of each annotation box in the original annotation file and the predicted annotation file, the overlapping annotation boxes with lower confidence are screened out to retain the annotation boxes with higher confidence as the final repair result. The embodiments of the present application are described in detail below.

[0033] Figure 1 FIG. 1 is a schematic diagram of training data shown in an exemplary embodiment of the present application. Figure 1 As shown, the training data includes an image 11 and annotated content 12 .

[0034] Image 11 can be a real image taken by an image acquisition device or a synthetic image generated by a simulation system. Image 11 usually contains multiple annotated objects, and the selection and definition of the annotated objects are closely related to the actual scene where the model will be deployed, and should be able to reflect the important elements in the scene information. For example, in the field of autonomous driving, the annotated objects may include pedestrians, vehicles, and roads; while in medical image analysis, the annotated objects may include lesions, organs, and cell tissues.

[0035] The annotation content 12 includes annotation information for each object in the image 11. The annotation information for any object generally includes an annotation box 121, an annotation category 122, and a confidence level (generally not displayed in the image in a visual form).

[0036] like Figure 1 As shown, the annotation box 121 is a rectangular boundary used to locate and frame objects in an image, usually represented by coordinates. In the target detection task, the model will try to identify multiple objects in the image and draw a bounding box for each detected object to indicate the approximate location of the object in the image. The annotation category 122 is the classification information for the object in the image, which is used to indicate the content of the object. In the target detection task, each annotation box not only needs to locate the object, but also needs to indicate the category of the object. For example, the annotation box may contain "person", "car" or "truck", etc.

[0037] When preparing training data for model training, you can create annotation boxes for each object in the image and add corresponding category labels through manual or automatic annotation systems to help the model learn. For a certain image, its corresponding annotation content is usually added by the same annotation source, such as the annotation information of each object is all added by the automatic annotation system or all added manually. In this solution, the annotation box contained in the training data is called the original annotation box.

[0038] In the model prediction stage, the model will not only predict and output the annotation box and category label of each detected object based on image 11, but also provide a confidence score for each predicted annotation box. In this solution, the predicted annotation box is called the predicted annotation box. The confidence reflects the probability that the model believes that the annotation box does contain the object of the predicted category. It can also be understood as a quantitative indicator of the degree of certainty of the model's prediction result. It is usually a value between 0 and 1, where 1 means that the model is very confident about the prediction result and 0 means no confidence at all. For the annotation boxes added manually or by the automatic annotation system, their confidence is obtained by unified addition, and the specific content will be described in detail in the subsequent embodiments.

[0039] It should also be noted that in practical applications, confidence can be further divided into classification confidence and detection confidence. Classification confidence indicates the degree of certainty of the model for the category label, while detection confidence comprehensively considers the accuracy of the category label and the position of the annotation box, reflecting the model's confidence in the entire detection result. The confidence used in the technical solution of this application can be either classification confidence or detection confidence, and this application does not limit this.

[0040] Figure 2 FIG. 1 is a flow chart of a method for repairing abnormal annotation content shown in an exemplary embodiment of the present application. Figure 2 , the method comprising:

[0041] Step 201, obtaining original annotation content of an original image and predicted annotation content output by a target model through prediction of the original image, and superimposing the original annotation content and the predicted annotation content to obtain mixed annotation content, wherein the target model is trained based on the original image and the original annotation content, the original annotation content and the predicted annotation content respectively include annotation information for objects in the original image, the annotation information of the original annotation content at least includes an original annotation box and its confidence, the annotation information of the predicted annotation content at least includes a predicted annotation box and its confidence, and the original annotation content has anomalies.

[0042] In one embodiment, the original image is an image that can be directly obtained, such as Figure 1The image 11 in the training data shown can be a real image taken by an image acquisition device or a synthetic image generated by a simulation system. The original annotation content is the annotation content added by a manual or automatic annotation system to the corresponding original image when preparing the training data for model training. Specifically, it includes the annotation information for the object in the original image. The annotation information at least includes the original annotation box and its corresponding confidence level. The related concepts can be seen in Figure 1 Example.

[0043] When the original image and the original annotation content are obtained, the target model can be trained so that it can learn data features and make predictions based on the current original data and the current original annotation content. After the target model is trained for a preset number of rounds, the target model has a certain prediction ability and can predict the original image and output the predicted annotation content. The predicted annotation content also includes annotation information for the object in the original image, at least including the predicted annotation box and its corresponding confidence. The confidence of the annotation box included in the predicted annotation content is the confidence directly output by the target model. The relevant concepts can be seen in Figure 1 Example.

[0044] Here, the target model can be a model trained completely based on the original image and the original annotation content. Of course, in order to save training resources and improve the model prediction effect, the target model can also be a model obtained by transfer learning based on an existing model, and then trained based on the original data and the original annotation content. This application does not impose any restrictions on this.

[0045] It is understandable that since the model itself has a certain generalization ability and is trained based on multiple data, although the target model is trained based on the original image and the original annotation content, the predicted annotation content output for the original image is not necessarily the same as the original annotation content corresponding to the original image.

[0046] Here, since the original annotation content is added by manual or automatic annotation systems, various anomalies are inevitable, such as missing annotations, annotation errors, inaccurate annotation boxes, etc., so the original annotation content with anomalies needs to be repaired.

[0047] In one embodiment, the presence of anomalies in the original annotation content may be determined in the following manner. The annotation quality score of the original annotation content may be determined based on the degree of deviation between the original annotation content and the predicted annotation content. The higher the degree of deviation, the lower the annotation quality score. After obtaining the annotation quality scores of the multiple original annotation contents, a mixed Gaussian distribution of these annotation quality scores is fitted using a mixed Gaussian algorithm. A mixed Gaussian distribution usually contains multiple data clusters, each cluster is usually modeled by a Gaussian distribution, such as Figure 3As shown in FIG. 1 , it includes clusters 31, 32, and 33, the horizontal axis represents the annotation quality score, and the vertical axis represents the number of images with the corresponding annotation quality score. Afterwards, the upper limit value of the preset cluster in the mixed Gaussian distribution can be used as the quality score threshold corresponding to the annotation quality score, and it is determined that each original annotation content with an annotation quality score lower than the quality score threshold is abnormal. For example, Figure 3 The upper limit value 0.1 of the first cluster X1 in is used as the quality score threshold, and all original annotation contents with an annotation quality score lower than 0.1 are identified as abnormal. Specifically, the upper limit value of the preset cluster can be determined based on the upper boundary of the cluster. First, each cluster has its corresponding parameters, including mean μ and variance, where mean μ is the center point of the cluster. Afterwards, the boundary of the cluster can be calculated based on mean μ and standard deviation σ (square root of variance), for example, using mean μ plus or minus a certain number of standard deviations σ. In the case of using mean plus or minus 3 times the standard deviation, the lower boundary (lower limit) of the cluster is μ-3σ, and the upper boundary (upper limit) of the cluster is μ+3σ. Within this boundary, about 99.7% of the data points in the cluster can be covered. At this time, the upper limit value μ++3σ of the cluster can be used as the quality score threshold, and the original annotation content corresponding to the annotation quality score less than μ++3σ is the original annotation content with abnormalities.

[0048] It is understandable that different original annotation contents have different annotation quality scores, and when fitting a mixed Gaussian distribution based on multiple annotation quality scores, using different multiple original annotation contents will also cause corresponding changes in the mixed Gaussian distribution. Therefore, the quality score threshold determined based on the mixed Gaussian distribution is a threshold that matches the original annotation content corresponding to the annotation quality score currently participating in the distribution fitting. Compared to the traditional method of using a fixed value as the quality score threshold, the scheme of this embodiment can dynamically determine the quality score threshold that matches the original annotation content, and thus can more accurately screen out the original annotation content with abnormalities.

[0049] In one embodiment, the degree of deviation between the original annotation content and the predicted annotation content can be determined based on the dimensional scores of multiple preset dimensions, and the preset dimensions include at least one of the following: whether the object annotation is complete, whether the object content is correct, and whether the position of the annotation box is accurate. If the object annotation of the original annotation content is incomplete, it can be understood that the original annotation content has a problem of missing annotations, such as not recognizing the "person" in the original image; if the object content of the original annotation content is incorrect, it can be understood that the original annotation content has a problem of incorrect annotations, such as labeling the "person" in the original image as "car"; if the position of the annotation box of the original annotation content is inaccurate, it can be understood that the marking box in the original annotation content fails to accurately mark the position of the object in the original image, such as the frame range of the annotation box is much larger than the actual range of the object in the original image.

[0050] For any original image, the scoring calculation method in the relevant technology can be used to calculate the sub-scores of the original image from the above three preset dimensions, and the three sub-scores are weighted to obtain the overall annotation quality score of the original image. In the weighted calculation process, the weight corresponding to each sub-score can be adjusted by the technician according to the actual situation and needs. For example, if the technician pays more attention to whether the object annotation is complete, or there is an abnormal original annotation content and the main abnormality is that the object annotation is incomplete, then the weight corresponding to the preset dimension can be appropriately increased, so that the corresponding sub-score plays a greater role in the weighted calculation, so that the overall annotation quality score can reflect the problem more.

[0051] When the original annotation content and the predicted annotation content are obtained, the two can be superimposed to obtain mixed annotation content. Figure 4 FIG. 1 is a schematic diagram of mixed annotation content shown in an exemplary embodiment of the present application. Figure 4 As shown, for object A in the original image, annotation box 41 is the annotation box for object A in the original annotation content, and its corresponding confidence is the confidence added uniformly; annotation box 42 is the annotation box for object A in the predicted annotation content, and its corresponding confidence is the confidence directly output by the target model. For object B in the original image, annotation box 43 is the annotation box for object B in the original annotation content, and its corresponding confidence is the confidence added uniformly; annotation box 44 is the annotation box for object B in the predicted annotation content, and its corresponding confidence is the confidence directly output by the target model.

[0052] Step 202, for a plurality of overlap evaluation indicators corresponding to the original annotation frame and the predicted annotation frame, determine an overlap threshold value corresponding to each overlap evaluation indicator, wherein the overlap threshold value corresponding to any overlap evaluation indicator is determined based on the original annotation content and the predicted annotation content.

[0053] Step 203: any two annotation boxes among all the annotation boxes of the mixed annotation content are regarded as a combination of annotation boxes, and when any combination of annotation boxes satisfies at least one overlap threshold, the annotation boxes with relatively low confidence in any combination of annotation boxes are deleted, and the deleted mixed annotation content is determined as a repair result of the original annotation content.

[0054] In one embodiment, there are many redundant annotation boxes in the mixed annotation content, for example, Figure 3For an object A in the image, there are both annotation boxes 41 and 42. At this time, the present application can determine the degree of overlap of any two annotation boxes based on multiple overlap evaluation indicators, and further determine whether the two annotation boxes are a combination of annotation boxes for the same object. Each overlap evaluation indicator has a corresponding overlap threshold, and each overlap threshold is dynamically determined based on the original annotation content and the predicted annotation content.

[0055] Specifically, multiple overlap evaluation indicators may include the intersection-over-union ratio of the areas of different annotation boxes and the ratio of the edge distances of different annotation boxes. The intersection-over-union ratio (IoU) is the ratio of the area of ​​the overlapping part of any two annotation boxes to the total coverage area of ​​the any two annotation boxes. Figure 3 The area of ​​the annotation box 41 is 1, the area of ​​the annotation box 42 is 1, and the area of ​​the overlapping part of the two is 0.8, then the intersection ratio of the two is 0.8÷2=0.4. The edge distance ratio between different annotation boxes is determined according to the boundaries of the two annotation boxes. Specifically, for any annotation box, you can select any two opposite sides of the annotation box to calculate their distance, and calculate the distance of the two corresponding sides in the other annotation box to be evaluated accordingly, and then calculate the edge distance ratio of the two annotation boxes according to the two distance values ​​obtained and the minimum value of the two distance values ​​by the following formula:

[0056]

[0057] When the opposite sides are selected as the upper and lower sides of the annotation box, the distance between the opposite sides can be understood as the height of the annotation box; when the opposite sides are selected as the left and right sides of the annotation box, the distance between the opposite sides can be understood as the width of the annotation box. Figure 3 The width of the middle annotation box 43 is 1.5, and the width of the annotation box 44 is 1.2. At this time, it can be determined that the edge distance ratio of the two annotation boxes is (1.5-1.2) / 1.2=0.25. Of course, the technician can also define the calculation method of the edge distance ratio in other ways according to the performance of the target model, and this application does not limit this.

[0058] In the case where a combination of any two annotation boxes meets at least one overlap threshold, the two annotation boxes can be considered as a combination of annotation boxes for the same object. Specifically, a combination of annotation boxes whose area intersection-over-union ratio is not less than (i.e., greater than) the intersection-over-union ratio threshold can be considered as a combination of annotation boxes for the same object, and a combination of annotation boxes whose edge distance ratio is not greater than (i.e., less than) the distance ratio threshold can be considered as a combination of annotation boxes for the same object.

[0059] It is understandable that when a single overlap evaluation index is used, it is equivalent to judging the overlap of two annotation frames only by a type of threshold comparison method, and the accuracy of the judgment result cannot be guaranteed. For example, when only the area intersection ratio is used as the overlap evaluation index, and two annotation frames whose area intersection ratio is greater than the intersection ratio threshold are judged as the combination of annotation frames for the same object, if the intersection ratio threshold is set too large, even if the two annotation frames are indeed the combination of annotation frames for the same object, due to the inaccurate position of the annotation frames contained in the original annotation content, the area intersection ratio of the two annotation frames is relatively small, which leads to the failure to judge that the two annotation frames are the combination of annotation frames for the same object; if the intersection ratio threshold is set too small, even if the two annotation frames are not the combination of annotation frames for the same object, because the two annotation frames have some overlapping areas, the two annotation frames will be mistakenly identified as the combination of annotation frames for the same object. Therefore, the technical solution of the present application adopts multiple overlap evaluation indexes to evaluate the overlap of the annotation frame combination in different dimensions, which can more accurately identify whether the annotation frame combination is the combination of annotation frames for the same object, making the judgment result more reliable.

[0060] As mentioned in the above-mentioned embodiment, when calculating the annotation quality score, sub-scores will be calculated based on three dimensions: whether the object annotation is complete, whether the object content is correct, and whether the annotation box position is accurate. In the following embodiment, the original annotation content with abnormalities will also be repaired according to the problems that may arise in these three dimensions.

[0061] In one embodiment, when any combination of annotation boxes meets at least one overlap threshold, the combination of annotation boxes can be considered to be a combination of annotation boxes for the same object. At this time, the annotation boxes with relatively low confidence in the combination of annotation boxes can be deleted, and the mixed annotation content after deletion can be determined as the repair result of the original annotation content.

[0062] Specifically, when the original annotation content has the problem of incomplete object annotation, the mixed annotation content only includes the annotation box in the predicted annotation content. For example, if the original annotation content does not include annotations for object A in the original image, and the predicted annotation content output by the target model includes annotations for object A, then for object A, the mixed annotation content only includes the annotation box in the predicted annotation content. At this time, based on the method of judging by multiple overlap evaluation indicators recorded in the above embodiment, the combination of the annotation box composed of the annotation box and any other annotation box included in the mixed annotation content is judged, and none of them should meet the overlap thresholds. Therefore, it will be determined that there is no combination of annotation boxes for the same object in the mixed annotation content, and the subsequent steps of deleting the annotation boxes with lower confidence will not be performed. The annotation box for object A in the predicted annotation content is retained in the mixed annotation content.

[0063] Specifically, when there is a problem that the object content is incorrect or the position of the annotation box is inaccurate in the original annotation content, the confidence corresponding to the annotation box in the original annotation content can be set to the first confidence, and the confidence corresponding to the annotation box in the predicted annotation content can be set to the second confidence. In the present application scheme, the technician can choose whether to believe the predicted annotation content predicted by the target model according to the performance of the target model. Generally speaking, after the target model has been trained for multiple iterations and the prediction results can reach a good level, you can choose to unconditionally believe the predicted annotation content predicted by the target model. Therefore, in the case where the mixed annotation content includes two annotation boxes for the same object, choose to unconditionally believe the prediction results of the target model, that is, it is necessary to delete the annotation box contained in the original annotation content. In this case, the technician can uniformly set the first confidence and the second confidence, and make the first confidence less than the second confidence. For example, the first confidence can be set to 0.99999 and the second confidence can be set to 1. At this time, although the first confidence is a higher value, it is still less than the second confidence. When the second confidence is greater than the first confidence, when deleting the annotation box with lower confidence, the annotation box corresponding to the first confidence will be deleted, that is, the annotation box included in the original annotation content, and the annotation box in the predicted annotation content will be retained in the mixed annotation content.

[0064] Of course, the target model may not perform well in the early stage of iterative training (especially in judging whether the object content is correct and whether the position of the annotation box is accurate). In this case, the first confidence can be set slightly lower, such as 0.8, and the second confidence directly uses the confidence output by the target model. In this case, if the second confidence given by the target model is greater than 0.8, which is a higher value, it means that the target model is very confident in the judgment of the annotation result. At this time, when deleting the annotation box with a lower confidence, the annotation box corresponding to the first confidence will be deleted, that is, the predicted annotation result output by the target model is chosen to be believed; and if the second confidence given by the target model is not greater than 0.8, it means that the target model is not very confident in the judgment of the annotation result. At this time, when deleting the annotation box with a lower confidence, the annotation box corresponding to the second confidence will be deleted, that is, the predicted annotation result output by the target model is chosen not to be believed.

[0065] It should be noted that when judging whether the original annotation content has corresponding problems in the preset dimension, the sub-scores corresponding to the multiple original annotation contents can also be fitted through the mixed Gaussian model in each dimension to obtain the mixed Gaussian distribution of the dimension, and the upper limit value of the preset cluster is used as the sub-score threshold of the dimension. When the score of any dimension corresponding to any original annotation content is less than the sub-score threshold, it is considered that any original annotation content has corresponding problems in the dimension.

[0066] It can be seen from the above embodiments that the technical solution of the present application can automatically and accurately repair the original annotation content with abnormalities. Specifically, multiple overlap evaluation indicators can be used to judge the annotation box combinations contained in the mixed annotation content, so as to accurately screen out the annotation box combinations whose overlap meets the requirements; at the same time, for each accurately screened annotation box combination, the confidence level can be used to select the annotation boxes included therein, and the annotation boxes with higher confidence levels can be retained, thereby further improving the accuracy of the repair results.

[0067] In one embodiment, it can be understood that when multiple overlap evaluation indicators are used and each overlap evaluation indicator has a corresponding overlap threshold, it is necessary to find the optimal solution for matching the multiple overlap thresholds so as to more accurately identify whether the labeling box combination is a labeling box combination for the same object.

[0068] Specifically, a candidate threshold set including multiple candidate threshold groups can be determined first, wherein any candidate threshold group includes candidate overlap thresholds corresponding to the multiple overlap evaluation indicators. In the case where the overlap evaluation indicators include the area intersection ratio and the edge distance ratio, any candidate threshold group includes a candidate intersection ratio threshold and a candidate edge distance ratio threshold. For any candidate threshold group, the annotation box combination will be judged according to the candidate overlap threshold contained in the candidate threshold group, and the threshold group that meets at least one candidate overlap threshold in the candidate threshold group will be tentatively repaired, that is, the annotation box with relatively low confidence in each annotation box combination is deleted, and the mixed annotation content after deletion is determined as the repair result of the original annotation content. The repair result obtained here is the tentative repair result using the candidate threshold group. Afterwards, the annotation quality score of the tentative repair result can be determined based on the deviation degree between the tentative repair result corresponding to the candidate threshold group and the predicted annotation content, as the fitness score of the candidate threshold group. Since the candidate threshold set includes multiple candidate threshold groups, tentative repair can be performed for each candidate threshold group, and its corresponding fitness score can be further obtained. On this basis, each candidate overlap threshold in the candidate threshold group with the highest fitness score can be used as the overlap threshold corresponding to the corresponding overlap degree evaluation index, that is, the candidate intersection-union ratio threshold in the candidate threshold group with the highest fitness can be used as the final intersection-union ratio threshold, and the candidate distance ratio threshold in the candidate threshold group with the highest fitness can be used as the final distance ratio threshold.

[0069] In one embodiment, the candidate threshold set can be provided in advance by the technician based on experience and actual conditions, or the candidate threshold set can be automatically calculated and output by a genetic algorithm. First, the genetic algorithm randomly generates an initial population, and each individual in the population is a candidate threshold set. After that, the genetic algorithm measures the performance of the individual according to the fitness score corresponding to each candidate threshold set, and selects individuals with excellent performance (i.e., high fitness score) (i.e., candidate threshold sets) to enter the next generation. Thereafter, the parent generation individuals are randomly selected for crossover operation to generate new offspring individuals, and the candidate overlap thresholds contained in some individuals are randomly changed, for example, a small random number is added or subtracted from the candidate intersection ratio threshold and / or the candidate distance ratio threshold, and then the performance of the current offspring individual is measured according to the fitness score corresponding to each offspring individual. The genetic algorithm will repeat the above steps until the preset number of iterations is reached, or the fitness score converges, that is, the candidate threshold set with a higher fitness score cannot be found. When a genetic algorithm is used to find the optimal candidate threshold group, the automation level of repairing the original annotation content with abnormalities is further improved, and the overlap threshold that can accurately determine the overlap degree of the annotation frame can be determined more efficiently, thereby making the repair result more accurate. Of course, in addition to using a genetic algorithm, other optimization algorithms can also be used to determine the overlap threshold, such as a particle swarm optimization algorithm, a differential evolution algorithm, etc., and this application does not limit this.

[0070] In one embodiment, the original annotation content can be updated to the repaired result of the original annotation content, and the target model can be retrained based on the original image and the updated original annotation content. After repairing the original annotation content with abnormalities by the method described in the above embodiment, more accurate annotation content can be obtained compared to before the repair. Retraining the model based on the original image and these more accurate annotation contents can enable the model to learn more correct features and further improve the prediction performance of the model.

[0071] In one embodiment, when the original annotation content corresponding to the original image is updated multiple times, and the target model is iteratively trained based on the original image and the original annotation content after each update, the prediction performance of the target model after each round of iterative training can be evaluated. If the improvement degree of the prediction performance of the target model meets the stopping requirement in the preset rounds of iterative training, the iterative training is stopped. Here, the stopping requirement can be set by the technician according to the actual situation and needs. For example, it can be determined whether the prediction performance of the target model has not improved in consecutive preset rounds, and the iterative training can be stopped if it has not improved in consecutive preset rounds; it can also be determined whether the improvement range of the prediction performance of the target model shows convergence, and the iterative training can be stopped when the preset convergence requirement is met. This application does not limit this.

[0072] Specifically, the data set corresponding to each round of iteration includes the original image and its original annotation content, which may be the original annotation content obtained after restoration. At this time, a search space containing different hyperparameter combinations can be defined, and the above data set can be divided into k subsets. Each time, k-1 subsets are used as training sets for training, and the remaining subset is used as a validation set for verification. The above process is repeated k times, and each subset is used as a validation set for verification, and the remaining k-1 subsets are used as training sets for training accordingly. For each hyperparameter combination, its average performance index in k trainings is calculated, and then the hyperparameter combination that performs well on different data subsets is identified.

[0073] When using any k-1 subsets as training sets for training, Dropout and early stopping techniques will be used simultaneously to avoid overfitting. Dropout will randomly ignore certain neurons of the target model in each training step, so that the target model no longer relies on certain specific neurons during training, thereby improving the generalization ability of the model. The early stopping technique will monitor the performance of the target model on the validation set, and stop training in advance when the current validation set indicates that the performance of the target model is no longer improving, ensuring that the target model has the best performance on the current validation set.

[0074] After that, the model can be retrained on the entire training set using the selected optimal hyperparameter combination, and Dropout and early stopping techniques can be applied again to consolidate the stability of the model and improve the final performance. This series of steps not only optimizes the configuration of the target model, but also ensures that the target model can perform better when facing unknown data, thereby improving the overall performance and reliability of the model.

[0075] In one embodiment, after stopping the iterative training of the target model, the original image corresponding to the current model and its original annotation content can be used as the target annotation data. At this time, the original annotation content contained in the target annotation data is the annotation content that has undergone multiple rounds of abnormal screening and corresponding multiple repairs. Compared with the original original annotation content, it can more accurately reflect the position and category of each object in the original image. The target annotation data can be widely used in the training of other related models, and each related model can learn data features based on the target annotation data.

[0076] It should be noted that, when only a small amount of original annotation content is abnormal, or when the restoration result does not meet the preset requirements, the abnormal original annotation content and the corresponding original image can be directly deleted. When the abnormal original annotation content and the corresponding original image are directly removed, the remaining original image and original annotation content are still relatively accurate target annotation data.

[0077] The following will be combined Figure 5 Describe the overall process of this solution.

[0078] First, step 501 is executed to train the target model, and the training data is a data set consisting of original images and original annotation content. This can be understood as the first large round of training for the model. In this step, the target model will be trained in multiple small rounds to obtain a target model with excellent performance for the current training data.

[0079] Execute step 502 to evaluate the target model, where the evaluation object is the prediction performance of the target model in the current round.

[0080] Execute step 503 to determine whether the prediction performance of the target model meets the stopping requirement. This step may specifically include step 5031 and step 5032. Step 5031 is used to determine whether the prediction performance of the target model is better than the prediction performance after the last large round of training (if the current large round is the first large round, this step is skipped), and if the current prediction performance of the target model is not better than the prediction performance after the last large round of training, step 5032 is used to determine whether the prediction performance of the target model has not been improved in consecutive preset rounds.

[0081] If the prediction performance of the target model has not improved within consecutive preset rounds, execute step 504 and use the original image and original annotation content corresponding to the current target model (that is, the training data corresponding to the current target model) as the target annotation data. Of course, a target model with better prediction performance for the target annotation data can also be obtained at the same time.

[0082] If the prediction performance of the target model is improved in consecutive preset rounds, step 505 is executed to use the current target model to predict the original image and output the predicted annotation content.

[0083] By executing the above steps, the training and performance evaluation of the target model in one large round are completed, and the output of the current target model also provides an operational basis for the next steps.

[0084] Then, step 506 is executed to determine the annotation quality score. In this step, the annotation quality score of the current original annotation content is determined based on the sub-scores of the original annotation content and the predicted annotation content in three dimensions: whether the object annotation is complete, whether the object content is correct, and whether the annotation box position is accurate.

[0085] Execute step 507 to fit the mixed Gaussian distribution of the annotation quality scores and determine the quality score threshold. In this step, the mixed Gaussian algorithm is used to fit the mixed Gaussian distribution of the annotation quality scores corresponding to the multiple original annotation contents, and the upper limit value of the preset cluster in the obtained mixed Gaussian distribution is used as the quality score threshold corresponding to the annotation quality score.

[0086] Execute step 508 to determine whether there is abnormal original annotation content. After obtaining the quality score threshold, the original annotation content corresponding to the annotation quality score less than the quality score threshold can be determined as the original annotation content with abnormality.

[0087] By executing the above steps, the screening of the original annotation contents with abnormalities is completed, and the original annotation contents with abnormalities will be processed next.

[0088] Step 509 is executed to repair the original annotation content with abnormalities, or to delete the original annotation content and its corresponding original image. The repair method can be seen in the above embodiment, and will not be repeated here.

[0089] Execute step 510 to update the training data. The training data here refers to the original image and the original annotation content in step 501. After repairing the original annotation content with abnormalities, the original annotation content with abnormalities is updated to the corresponding repair results, so that the training data is updated. After directly deleting the original annotation content with abnormalities, the training data is also updated.

[0090] Afterwards, step 501 is executed again, and the target model is trained for the next large round based on the updated training data. The above process is repeated until the target annotation data can be obtained through step 504.

[0091] Based on the method for repairing abnormal annotated content provided by the present application, the present application also provides a device for repairing abnormal annotated content, see Figure 6 ,include:

[0092] An acquisition unit 601 is used to acquire original annotation content of an original image and predicted annotation content output by a target model through prediction of the original image, and to perform superposition processing on the original annotation content and the predicted annotation content to obtain mixed annotation content, wherein the target model is trained according to the original image and the original annotation content, the original annotation content and the predicted annotation content respectively include annotation information for objects in the original image, the annotation information of the original annotation content at least includes an original annotation box and its confidence, the annotation information of the predicted annotation content at least includes a predicted annotation box and its confidence, and the original annotation content has an abnormality;

[0093] A threshold determination unit 602 is used to determine a coincidence threshold value corresponding to each coincidence evaluation index for a plurality of coincidence evaluation indexes corresponding to the original annotation frame and the predicted annotation frame, wherein the coincidence threshold value corresponding to any coincidence evaluation index is determined based on the original annotation content and the predicted annotation content;

[0094] The repairing unit 603 is used to treat any two annotation boxes in all the annotation boxes of the mixed annotation content as a combination of annotation boxes, and when any annotation box combination satisfies at least one overlap threshold, delete the annotation boxes with relatively low confidence in any annotation box combination, and determine the deleted mixed annotation content as a repair result of the original annotation content.

[0095] Optionally, determining that the original annotation content is abnormal includes:

[0096] Determining a labeling quality score of the original labeling content based on a degree of deviation between the original labeling content and the predicted labeling content, wherein the labeling quality score is negatively correlated to the degree of deviation;

[0097] For the annotation quality scores of the respective original annotation contents, a mixed Gaussian algorithm is used to fit a mixed Gaussian distribution of the annotation quality scores, and an upper limit value of a preset cluster in the mixed Gaussian distribution is used as a quality score threshold corresponding to the annotation quality score;

[0098] It is determined that each original annotation content whose annotation quality score is lower than the quality score threshold has anomalies.

[0099] Optionally, the threshold determination unit 602 is specifically configured to:

[0100] Determine a candidate threshold set including a plurality of candidate threshold groups, wherein any candidate threshold group includes candidate coincidence degree thresholds corresponding to the plurality of coincidence degree evaluation indicators respectively;

[0101] For any candidate threshold group in the candidate threshold set, deleting a relatively low confidence annotation box in any annotation box combination, and determining the deleted mixed annotation content as a repair result of the original annotation content; and determining a annotation quality score of the repair result based on the deviation degree between the repair result of the original annotation content and the predicted annotation content, and using the determined annotation quality score as the fitness score of any candidate threshold group;

[0102] Each candidate coincidence threshold in the candidate threshold group with the highest fitness score is used as the coincidence threshold corresponding to the corresponding coincidence degree evaluation index.

[0103] Optionally, the determining of a candidate threshold set including a plurality of candidate threshold groups includes: calculating and outputting the candidate threshold set through a genetic algorithm.

[0104] Optionally, determining the annotation quality score of the original annotation content based on the degree of deviation between the original annotation content and the predicted annotation content includes: determining the degree of deviation between the original annotation content and the predicted annotation content according to dimensional scores of multiple preset dimensions, and the preset dimensions include at least one of the following: whether the object annotation is complete, whether the object content is correct, and whether the annotation box position is accurate.

[0105] Optionally, for any original image,

[0106] The confidence in the original annotation content is a first confidence, the confidence in the predicted annotation content is a second confidence, and the first confidence is smaller than the second confidence.

[0107] Optionally, the multiple overlap evaluation indicators include the area intersection-and-union ratio of different annotation boxes and the edge distance ratio of different annotation boxes, wherein the overlap threshold corresponding to the area intersection-and-union ratio is the intersection-and-union ratio threshold, and the overlap threshold corresponding to the edge distance ratio is the distance ratio threshold.

[0108] Optionally, when any combination of annotation boxes satisfies at least one overlap threshold, deleting an annotation box with a relatively low confidence in any combination of annotation boxes includes:

[0109] When the intersection-and-union ratio of the areas of any combination of annotation boxes is not less than the intersection-and-union ratio threshold, deleting the annotation boxes with relatively low confidence in any combination of annotation boxes;

[0110] When the edge distance ratio of any combination of annotation boxes is not greater than the distance ratio threshold, the annotation boxes with relatively low confidence in any combination of annotation boxes are deleted.

[0111] Optionally, the device further comprises:

[0112] An updating unit 604, configured to update the original annotation content to a repair result of the original annotation content;

[0113] The retraining unit 605 is used to retrain the target model based on the original image and the updated original annotation content.

[0114] Optionally, the device further comprises:

[0115] An evaluation unit 606, configured to evaluate the prediction performance of the target model after each round of iterative training, when the original annotation content corresponding to the original image is updated multiple times and the target model is iteratively trained based on the original image and the original annotation content after each update;

[0116] The stop judgment unit 607 is used to stop the iterative training and use the original image and its original annotation content corresponding to the current model as the target annotation data if the improvement degree of the prediction performance in the preset rounds of iterative training meets the stopping requirement.

[0117] Optionally, the original image includes at least one of the following: a real image captured by an image acquisition device, a synthetic image generated by a simulation system; and / or,

[0118] The original annotation content of each original image includes manual annotation content or automatic annotation content.

[0119] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0120] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0121] Correspondingly, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for repairing abnormal annotation content as described in any of the above embodiments is implemented.

[0122] See also Figure 7At the hardware level, the electronic device includes a processor 702, an internal bus +704, a network interface 706, a memory 708 and a non-volatile memory 710, and may also include hardware required for other services. The processor 702 reads the corresponding computer program from the non-volatile memory 710 into the memory 708 and then runs it, forming a vehicle speed adjustment device at the logical level. Of course, in addition to the software implementation, this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0123] Correspondingly, the present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method for repairing abnormal annotation content as described in any of the above embodiments is implemented.

[0124] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0125] In a typical configuration, a computer includes one or more processors + (CPU), input / output interfaces, network interfaces, and memory.

[0126] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory (flash+RAM). Memory is an example of a computer-readable medium.

[0127] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory + (PRAM), static random access memory + (SRAM), dynamic random access memory + (DRAM), other types of random access memory + (RAM), read-only memory + (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory + (CD-ROM), digital versatile disk + (DVD) + or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media + (transitory + media), such as modulated data signals and carrier waves.

[0128] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0129] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0130] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0131] The above description is merely a preferred embodiment of one or more embodiments of the present application and is not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included in the scope of protection of one or more embodiments of the present application.

Claims

1. A method for repairing abnormal annotation content, characterized in that: include: Acquire original annotation content of an original image and predicted annotation content output by a target model through prediction of the original image, and superimpose the original annotation content and the predicted annotation content to obtain mixed annotation content, wherein the target model is trained according to the original image and the original annotation content, the original annotation content and the predicted annotation content respectively include annotation information for objects in the original image, the annotation information of the original annotation content at least includes an original annotation box and its confidence, the annotation information of the predicted annotation content at least includes a predicted annotation box and its confidence, and the original annotation content has anomalies; Determining, for a plurality of overlap evaluation indicators corresponding to the original annotation frame and the predicted annotation frame, an overlap threshold value corresponding to each overlap evaluation indicator, wherein the overlap threshold value corresponding to any overlap evaluation indicator is determined based on the original annotation content and the predicted annotation content; Any two annotation boxes among all the annotation boxes of the mixed annotation content are taken as a annotation box combination, and when any annotation box combination satisfies at least one overlap threshold, the annotation boxes with relatively low confidence in any annotation box combination are deleted, and the deleted mixed annotation content is determined as a repair result of the original annotation content.

2. The method according to claim 1, characterized in that Determining that the original annotation content is abnormal includes: Determining a labeling quality score of the original labeling content based on a degree of deviation between the original labeling content and the predicted labeling content, wherein the labeling quality score is negatively correlated to the degree of deviation; For the annotation quality scores of the respective original annotation contents, a mixed Gaussian algorithm is used to fit a mixed Gaussian distribution of the annotation quality scores, and an upper limit value of a preset cluster in the mixed Gaussian distribution is used as a quality score threshold corresponding to the annotation quality score; It is determined that each original annotation content whose annotation quality score is lower than the quality score threshold has anomalies.

3. The method according to claim 2, characterized in that Determining the overlap threshold corresponding to each overlap evaluation index includes: Determine a candidate threshold set including a plurality of candidate threshold groups, wherein any candidate threshold group includes candidate coincidence degree thresholds corresponding to the plurality of coincidence degree evaluation indicators respectively; For any candidate threshold group in the candidate threshold set, deleting a relatively low confidence annotation box in any annotation box combination, and determining the deleted mixed annotation content as a repair result of the original annotation content; and determining a annotation quality score of the repair result based on the deviation degree between the repair result of the original annotation content and the predicted annotation content, and using the determined annotation quality score as the fitness score of any candidate threshold group; Each candidate coincidence threshold in the candidate threshold group with the highest fitness score is used as the coincidence threshold corresponding to the corresponding coincidence degree evaluation index.

4. The method according to claim 3, characterized in that The determining of a candidate threshold set including a plurality of candidate threshold groups includes: calculating and outputting the candidate threshold set by a genetic algorithm.

5. The method according to claim 2, characterized in that: The determining of the annotation quality score of the original annotation content based on the degree of deviation between the original annotation content and the predicted annotation content includes: determining the degree of deviation between the original annotation content and the predicted annotation content according to dimensional scores of multiple preset dimensions, wherein the preset dimensions include at least one of the following: whether the object annotation is complete, whether the object content is correct, and whether the annotation box position is accurate.

6. The method according to claim 1, characterized in that For any original image, The confidence in the original annotation content is a first confidence, the confidence in the predicted annotation content is a second confidence, and the first confidence is smaller than the second confidence.

7. The method according to claim 1, characterized in that The multiple overlap evaluation indicators include the area intersection-and-union ratio of different annotation boxes and the edge distance ratio of different annotation boxes, wherein the overlap threshold corresponding to the area intersection-and-union ratio is the intersection-and-union ratio threshold, and the overlap threshold corresponding to the edge distance ratio is the distance ratio threshold.

8. The method according to claim 7, characterized in that When any combination of annotation boxes meets at least one overlap threshold, deleting an annotation box with a relatively low confidence in any combination of annotation boxes includes: When the intersection-and-union ratio of the areas of any combination of annotation boxes is not less than the intersection-and-union ratio threshold, deleting the annotation boxes with relatively low confidence in any combination of annotation boxes; When the edge distance ratio of any combination of annotation boxes is not greater than the distance ratio threshold, the annotation boxes with relatively low confidence in any combination of annotation boxes are deleted.

9. The method according to claim 1, characterized in that: Also includes: Updating the original annotation content to a repair result of the original annotation content; The target model is retrained based on the original image and the updated original annotation content.

10. The method according to claim 9, characterized in that Also includes: When the original annotation content corresponding to the original image is updated multiple times and the target model is iteratively trained based on the original image and the original annotation content after each update, the prediction performance of the target model after each round of iterative training is evaluated; If the improvement of the prediction performance in the preset rounds of iterative training meets the stopping requirement, the iterative training is stopped and the original image and its original annotation content corresponding to the current model are used as the target annotation data.

11. The method according to claim 1, characterized in that: The original image includes at least one of the following: a real image captured by an image acquisition device, a synthetic image generated by a simulation system; and / or, The original annotation content of each original image includes manual annotation content or automatic annotation content.

12. A device for repairing abnormally marked content, characterized in that: include: an acquisition unit, used to acquire original annotation content of an original image and predicted annotation content output by a target model through prediction of the original image, and to perform superposition processing on the original annotation content and the predicted annotation content to obtain mixed annotation content, wherein the target model is trained according to the original image and the original annotation content, the original annotation content and the predicted annotation content respectively include annotation information for objects in the original image, the annotation information of the original annotation content at least includes an original annotation box and its confidence, the annotation information of the predicted annotation content at least includes a predicted annotation box and its confidence, and the original annotation content has anomalies; a threshold determination unit, configured to determine, for a plurality of overlap degree evaluation indicators corresponding to the original annotation frame and the predicted annotation frame, an overlap degree threshold value corresponding to each overlap degree evaluation indicator, wherein the overlap degree threshold value corresponding to any overlap degree evaluation indicator is determined based on the original annotation content and the predicted annotation content; A repairing unit is used to treat any two annotation boxes in all the annotation boxes of the mixed annotation content as a combination of annotation boxes, and when any annotation box combination satisfies at least one overlap threshold, delete the annotation box with relatively low confidence in any annotation box combination, and determine the deleted mixed annotation content as a repair result of the original annotation content.

13. An electronic device, characterized in that: include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 11 by executing the executable instructions.

14. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 11.

Images