Data processing method, intelligent device and storage medium

Abstract

The invention relates to the technical field of data processing, particularly provides a data processing method, intelligent equipment and a storage medium, and aims to solve the problem that the accuracy and reliability of a business model are affected due to poor annotation data quality. In order to achieve the purpose, the method comprises the steps of obtaining truth value data corresponding to an original image sequence; obtaining annotation data of the original image sequence, and obtaining annotation quality detection data based on the annotation quality detection model and the annotation data; determining a data learning difficulty attribute of the annotated data based on the truth value data; processing the labeled quality detection data to obtain detection data meeting a preset condition; and obtaining a processed data set based on the data learning difficulty attribute and the detection data meeting the preset condition. Through the above implementation mode, automatic detection can be carried out on large-scale annotation data, the quality inspection efficiency and quality are improved, difficult samples can be mined according to the difficulty degree of data learning, and the quality of the annotation data is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, an intelligent device and a storage medium. Background Art

[0002] In the field of autonomous driving, the quality of data annotation is crucial to the performance of business models (such as target detection models), and autonomous driving systems rely on massive amounts of labeled data for model training.

[0003] However, as the scale of data expands, the quality of labeled data is difficult to guarantee. The existing quality inspection method is mainly manual spot checks, which is not only time-consuming and labor-intensive, but also difficult to adapt to the needs of large-scale autonomous driving data. In addition, incorrect labeling or noise will seriously affect the accuracy and reliability of the model, causing deviations in the judgment of the autonomous driving system and having a negative impact on the user's driving experience.

[0004] Therefore, there is an urgent need for an automated data processing solution that can save manual quality inspection costs while improving the quality of labeled data, thereby improving the accuracy and efficiency of the autonomous driving business model and optimizing the user's intelligent driving experience. Summary of the invention

[0005] In order to overcome the above-mentioned defects, the present application is proposed to solve or at least partially solve the technical problem that the quality of labeled data is poor, which affects the accuracy and reliability of the business model.

[0006] In a first aspect, a data processing method is provided, wherein the method comprises:

[0007] Obtain the true value data corresponding to the original image sequence;

[0008] Acquire annotation data of the original image sequence, and acquire annotation quality detection data based on an annotation quality detection model and the annotation data;

[0009] Determining a data learning difficulty attribute of the labeled data based on the true value data;

[0010] Processing the labeling quality inspection data to obtain inspection data that meets preset conditions;

[0011] A processed data set is obtained based on the data learning difficulty attribute and the detection data that meets the preset conditions.

[0012] In a technical solution of the above data processing method, obtaining the annotation data of the original image sequence includes:

[0013] Inputting multiple frames of original images in the original image sequence into the target detection model to obtain multiple frames of model annotated images corresponding to the multiple frames of original images; or obtaining multiple frames of manually annotated images corresponding to the multiple frames of original images;

[0014] The annotation data is obtained based on the multiple frames of model annotation images or the multiple frames of manually annotated images.

[0015] In a technical solution of the above data processing method, the obtaining of the annotation quality detection data based on the annotation quality detection model and the annotation data includes:

[0016] The annotation data is input into the annotation quality detection model to obtain annotation quality detection data of each target in the original image sequence.

[0017] In a technical solution of the above data processing method, the processing of the labeling quality detection data to obtain detection data that meets preset conditions includes:

[0018] Aggregating the label quality detection data of each of the targets in the original image sequence;

[0019] Obtaining the number of occurrences of each of the targets in the multiple frames of original images;

[0020] The target whose occurrence number is greater than or equal to a preset threshold is determined, and the labeling quality detection data corresponding to the target whose occurrence number is greater than or equal to the preset threshold is used as the detection data that meets the preset condition.

[0021] In a technical solution of the above data processing method, the preset threshold includes a first preset threshold and a second preset threshold; determining the target whose number of occurrences is greater than or equal to the preset threshold includes:

[0022] Determine targets whose consecutive appearance times in the multiple frames of original images are greater than or equal to the first preset threshold;

[0023] Or, determining targets whose non-continuous appearance times in the multiple frames of original images are greater than or equal to the second preset threshold;

[0024] The first preset threshold is smaller than the second preset threshold.

[0025] In a technical solution of the above data processing method, the data learning difficulty attribute includes first sample data and second sample data, and the data learning difficulty of the first sample data is higher than that of the second sample data; and determining the data learning difficulty attribute of the labeled data based on the true value data includes:

[0026] Matching the true value data with the labeled data;

[0027] The labeled data is determined to be the first sample data or the second sample data based on the matching result.

[0028] In a technical solution of the above data processing method, matching the true value data with the labeled data includes:

[0029] Respectively obtain the category information and the position information of the center point of the annotation box corresponding to each of the targets in the true value data and the annotation data;

[0030] Based on the position information, obtaining the center point distance of each target in the true value data and the labeled data;

[0031] The matching result is obtained based on the category information and the center point distance.

[0032] In a technical solution of the above data processing method, obtaining the matching result based on the category information and the center point distance includes:

[0033] When the categories are the same and the center point distance is less than a preset distance, it is determined that the target is matched successfully;

[0034] Otherwise, it is determined that the target matching fails;

[0035] The matching failure includes any of the following:

[0036] The categories of the target in the true value data and the labeled data are different;

[0037] The target of the same category is only in the true value data and not in the labeled data; or

[0038] Or the target with the same category is only in the labeled data but not in the true value data.

[0039] In a technical solution of the above data processing method, determining that the labeled data is the first sample data or the second sample data based on the matching result includes:

[0040] Taking the labeled data of the target that has the same category and is only in the true value data and not in the labeled data as the first sample data;

[0041] The labeled data of the target that is successfully matched is used as the second sample data.

[0042] In a technical solution of the above data processing method, the labeling quality detection data includes a labeling quality detection score; the data set processed based on the data learning difficulty attribute and the detection data meeting the preset conditions includes:

[0043] Among the detection data meeting the preset conditions, the target labeled data whose labeling quality detection score is higher than the preset score is used as high-quality labeled data;

[0044] According to a preset ratio, the first sample data and the second sample data in the high-quality labeled data are obtained to obtain the processed data set.

[0045] In a technical solution of the above data processing method, the method further includes:

[0046] Among the detection data meeting the preset conditions, the target labeled data whose labeling quality detection score is lower than or equal to the preset score is regarded as low-quality labeled data;

[0047] The low-quality labeled data is removed.

[0048] In a technical solution of the above data processing method, the method further includes:

[0049] The target detection model is iteratively trained based on the processed data set until the target detection model converges to a preset error value.

[0050] In a second aspect, a smart device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any one of the technical solutions of the above-mentioned data processing method is implemented.

[0051] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, wherein the program codes are suitable for being loaded and run by a processor to execute the method described in any one of the technical solutions of the above-mentioned data processing method.

[0052] Solution 1. A data processing method, characterized in that the method comprises:

[0053] Obtain the true value data corresponding to the original image sequence;

[0054] Acquire annotation data of the original image sequence, and acquire annotation quality detection data based on an annotation quality detection model and the annotation data;

[0055] Determining a data learning difficulty attribute of the labeled data based on the true value data;

[0056] Processing the labeling quality inspection data to obtain inspection data that meets preset conditions;

[0057] A processed data set is obtained based on the data learning difficulty attribute and the detection data that meets the preset conditions.

[0058] Solution 2. The data processing method according to Solution 1 is characterized in that the step of obtaining the annotation data of the original image sequence comprises:

[0059] Inputting multiple frames of original images in the original image sequence into the target detection model to obtain multiple frames of model annotated images corresponding to the multiple frames of original images; or obtaining multiple frames of manually annotated images corresponding to the multiple frames of original images;

[0060] The annotation data is obtained based on the multiple frames of model annotation images or the multiple frames of manually annotated images.

[0061] Solution 3. The data processing method according to Solution 2 is characterized in that the step of obtaining the labeling quality detection data based on the labeling quality detection model and the labeling data comprises:

[0062] The annotation data is input into the annotation quality detection model to obtain annotation quality detection data of each target in the original image sequence.

[0063] Solution 4. The data processing method according to Solution 3 is characterized in that the processing of the labeling quality detection data to obtain detection data that meets the preset conditions includes:

[0064] Aggregating the label quality detection data of each of the targets in the original image sequence;

[0065] Obtaining the number of occurrences of each of the targets in the multiple frames of original images;

[0066] The target whose occurrence number is greater than or equal to a preset threshold is determined, and the labeling quality detection data corresponding to the target whose occurrence number is greater than or equal to the preset threshold is used as the detection data that meets the preset condition.

[0067] Solution 5. The data processing method according to Solution 4 is characterized in that the preset threshold includes a first preset threshold and a second preset threshold; and determining the target whose number of occurrences is greater than or equal to the preset threshold includes:

[0068] Determine targets whose consecutive appearance times in the multiple frames of original images are greater than or equal to the first preset threshold;

[0069] Or, determining targets whose total number of occurrences in multiple frames of original images is greater than or equal to the second preset threshold;

[0070] The first preset threshold is smaller than the second preset threshold.

[0071] Solution 6. The data processing method according to Solution 1 is characterized in that the data learning difficulty attribute includes first sample data and second sample data, and the data learning difficulty of the first sample data is higher than that of the second sample data; and determining the data learning difficulty attribute of the labeled data based on the true value data includes:

[0072] Matching the true value data with the labeled data;

[0073] The labeled data is determined to be the first sample data or the second sample data based on the matching result.

[0074] Solution 7. The data processing method according to Solution 6 is characterized in that the matching of the true value data and the labeled data comprises:

[0075] Respectively obtain the category information and the position information of the center point of the annotation box corresponding to each of the targets in the true value data and the annotation data;

[0076] Based on the position information, obtaining the center point distance of each target in the true value data and the labeled data;

[0077] The matching result is obtained based on the category information and the center point distance.

[0078] Solution 8. The data processing method according to Solution 7 is characterized in that obtaining the matching result based on the category information and the center point distance comprises:

[0079] When the categories are the same and the center point distance is less than a preset distance, it is determined that the target is matched successfully;

[0080] Otherwise, it is determined that the target matching fails;

[0081] The matching failure includes any of the following:

[0082] The categories of the target in the true value data and the labeled data are different;

[0083] The target of the same category is only in the true value data and not in the labeled data; or

[0084] The objects of the same category are only in the labeled data but not in the true value data.

[0085] Solution 9. The data processing method according to Solution 8 is characterized in that determining the labeled data as the first sample data or the second sample data based on the matching result comprises:

[0086] Taking the labeled data of the target that has the same category and is only in the true value data and not in the labeled data as the first sample data;

[0087] The labeled data of the target that is successfully matched is used as the second sample data.

[0088] Solution 10. The data processing method according to Solution 9 is characterized in that the labeling quality detection data includes a labeling quality detection score; and the data set processed based on the data learning difficulty attribute and the detection data that meets the preset conditions includes:

[0089] Among the detection data meeting the preset conditions, the target labeled data whose labeling quality detection score is higher than the preset score is used as high-quality labeled data;

[0090] According to a preset ratio, the first sample data and the second sample data in the high-quality labeled data are obtained to obtain the processed data set.

[0091] Solution 11. The data processing method according to Solution 10, characterized in that the method further comprises:

[0092] Among the detection data meeting the preset conditions, the target labeled data whose labeling quality detection score is lower than or equal to the preset score is regarded as low-quality labeled data;

[0093] The low-quality labeled data is removed.

[0094] Solution 12. The data processing method according to Solution 10, characterized in that the method further comprises:

[0095] The target detection model is iteratively trained based on the processed data set until the target detection model converges to a preset error value.

[0096] Solution 13. A smart device, comprising:

[0097] at least one processor;

[0098] and, a memory communicatively coupled to the at least one processor;

[0099] Wherein, a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the data processing method described in any one of Schemes 1 to 12 is implemented.

[0100] Solution 14. A computer-readable storage medium storing a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the data processing method described in any one of Solutions 1 to 12.

[0101] The above one or more technical solutions of the present application have at least one or more of the following beneficial effects:

[0102] In implementing the technical solution provided in this application, the labeling quality detection model and the data learning difficulty attribute are combined, which can not only automatically detect large-scale labeled data, effectively identify incorrect labels or noise, reduce manual quality inspection costs, and improve quality inspection efficiency and quality, but also mine difficult samples according to the difficulty of data learning and improve the quality of labeled data, thereby improving the accuracy and robustness of the autonomous driving business model and optimizing the user's driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] The disclosure of the present application will become easier to understand with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. Among them:

[0104] Figure 1 It is a flowchart of the main steps of a data processing method according to an embodiment of the present application;

[0105] Figure 2 is a schematic diagram of a data processing link according to an embodiment of the present application;

[0106] Figure 3 This is a flow chart of the main steps of determining the data learning difficulty attribute of labeled data based on true value data according to an embodiment of the present application;

[0107] Figure 4 It is a flowchart of the main steps of processing the labeling quality inspection data to obtain the inspection data meeting the preset conditions according to an embodiment of the present application;

[0108] Figure 5 It is a flowchart of the main steps of obtaining a processed data set based on the data learning difficulty attribute and the detection data meeting the preset conditions according to an embodiment of the present application;

[0109] Figure 6 is a schematic diagram of data learning difficulty attributes and labeled data quality according to an embodiment of the present application;

[0110] Figure 7 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application;

[0111] Figure 8 is a schematic diagram of a smart device according to an embodiment of the present application.

[0112] Reference numerals:

[0113] 81: memory; 82: processor; 83: sensor. DETAILED DESCRIPTION

[0114] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0115] In the description of the present application, the term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The term "at least one of A or B" or "at least one of A and B" has a similar meaning to "A and / or B" and may include only A, only B, or A and B. The singular terms "a", "the" and "the" may also include plural forms.

[0116] As described in the background technology, in the field of autonomous driving, the quality of data annotation is crucial to the performance of business models (such as target detection models), and autonomous driving systems rely on massive amounts of labeled data for model training.

[0117] However, as the scale of data expands, the quality of labeled data is difficult to guarantee. The existing quality inspection method is mainly manual spot checks, which is not only time-consuming and labor-intensive, but also difficult to adapt to the needs of large-scale autonomous driving data. In addition, incorrect labeling or noise will seriously affect the accuracy and reliability of the model, causing deviations in the judgment of the autonomous driving system and having a negative impact on the user's driving experience.

[0118] In order to solve the above problems, the present application provides a data processing method, an intelligent device and a storage medium.

[0119] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart of the main steps of a data processing method according to an embodiment of the present application. Figure 1 As shown, the data processing method in the embodiment of the present application mainly includes the following steps S101 to S105.

[0120] Step S101: obtaining true value data corresponding to the original image sequence;

[0121] The original image sequence includes multiple frames of original images, and the ground truth data (GT) corresponding to the original image sequence is the real label data of the multiple frames of original images.

[0122] Step S102: obtaining annotation data of the original image sequence, and obtaining annotation quality detection data based on the annotation quality detection model and the annotation data;

[0123] The annotation data of the original image sequence is a multi-frame annotated image obtained by annotating multiple frames of original images, and the images are annotated with information such as the three-dimensional annotation box, two-dimensional annotation box, orientation, category and attributes of each target.

[0124] Among them, in the field of autonomous driving, the target refers to various objects in the surrounding environment that need to be identified and located during the driving process of the autonomous driving vehicle, such as vehicles, pedestrians, obstacles, traffic signs and signal lights.

[0125] The annotation quality detection data includes the annotation quality detection results of each object in the original image sequence.

[0126] Step S103: determining the data learning difficulty attribute of the labeled data based on the true value data;

[0127] The data learning difficulty attribute includes first sample data and second sample data; the data learning difficulty of the first sample data is higher than that of the second sample data.

[0128] Specifically, the first sample data may be difficult sample data, whose features are complex and difficult to process. When the model is trained with the first sample data, the generalization ability of the model can be improved. The second sample data may be simple sample data, whose features are obvious and easy to process. When the model is trained with the second sample data, the model can converge quickly, but is prone to overfitting.

[0129] Step S104: Processing the labeling quality inspection data to obtain inspection data that meets preset conditions;

[0130] Step S105: obtaining a processed data set based on the data learning difficulty attribute and the detection data meeting the preset conditions.

[0131] Based on the method described in steps S101 to S105 above, the labeling quality detection model and the data learning difficulty attribute are combined, which can not only automatically detect large-scale labeled data, effectively identify incorrect labels or noise, reduce manual quality inspection costs, and improve quality inspection efficiency and quality, but also mine difficult samples according to the difficulty of data learning and improve the quality of labeled data, thereby improving the accuracy and robustness of the autonomous driving business model and optimizing the user's driving experience.

[0132] The above steps S101 to S105 are further explained below.

[0133] In some implementations of the above step S101, the original image sequence may include a plurality of frames of original images arranged in chronological order within a period of time (eg, 10s-20s).

[0134] Furthermore, the GT data corresponding to the original image sequence can be the real label data of the original image sequence used in the process of training business models (such as target detection models), which contains the accurate information of the target in each frame of the original image, which is the "standard answer" in the model learning process. In simple terms, it can be the data such as the real position, category and attributes of the target marked manually or by high-precision means, or the data obtained by processing the data collected by the sensor.

[0135] The target real position is the actual position information of the marked target object. The target real position can be marked with a two-dimensional annotation box or a three-dimensional annotation box. The two-dimensional annotation box can include the coordinates of the upper left corner and the lower right corner, or the center point coordinates and the width and height. The three-dimensional annotation box can include the three-dimensional coordinates (x, y, z) of the target and its size (length, width, height) in three-dimensional space.

[0136] The category of the target can clearly identify the category to which each target belongs. For example, in common traffic scenarios, the category of the target can be "vehicle", "pedestrian", "traffic sign" and "motorcycle".

[0137] The attributes of a target are characteristic descriptions of the target object itself. Taking a vehicle as an example, its attributes may include vehicle model, body size, vehicle speed, etc.

[0138] The acquired GT data can be manually annotated, that is, the annotator uses an annotation tool to annotate the location, category, and attributes of the target on the image. Manual annotation can ensure high accuracy, but the annotation efficiency is relatively low and may be affected by the subjective factors of the annotator, resulting in differences in annotation quality. GT data can also be processed data collected by sensors. In some scenarios, such as autonomous driving scenarios, a variety of sensors installed on the vehicle can be used to obtain target information. By processing these sensor data, high-precision target location and category information can be obtained as GT data.

[0139] Furthermore, the GT data corresponding to the original image sequence may be obtained and stored in a corresponding annotation file in a preset data type.

[0140] It should be pointed out that the above examples of GT data and methods for obtaining the same are only illustrative. In practical applications, those skilled in the art may make selections based on specific scenarios, and no limitation is made here.

[0141] The above is a further description of step S101 , and the following is a further description of step S102 .

[0142] In some implementations of the above step S102, multiple frames of original images in the original image sequence may be input into the target detection model to obtain multiple frames of model annotated images corresponding to the multiple frames of original images.

[0143] By inputting multiple frames of original images into a target detection model, such as the YOLO model, the Faster R-CNN model, etc., each frame of the original image can be analyzed and processed by the target detection model to identify the location, category and other information of target objects such as vehicles, pedestrians, and traffic signs, and mark them on the image by annotation, thereby obtaining a multi-frame model annotated image that corresponds one-to-one to the multiple frames of original images.

[0144] In some implementations, multiple frames of original images in the original image sequence may also be manually annotated to obtain multiple frames of manually annotated images corresponding to the multiple frames of original images.

[0145] Furthermore, the annotation data of the original image sequence can be obtained based on multiple frames of annotated images (including multiple frames of model annotated images or multiple frames of manually annotated images), and the annotation data of the original image sequence can be stored in a corresponding annotation file in a preset data type. The annotation data includes multiple frames of original images and their annotation information.

[0146] In some implementations of the above step S102, after obtaining the annotation data of the original image sequence, the annotation data may be input into an annotation quality detection model to obtain annotation quality detection data of each object in the original image sequence.

[0147] In order to improve the efficiency and accuracy of annotation data quality detection and reduce the cost of data processing, the annotation quality detection model can be a model with a simple structure and a small number of parameters. Specifically, it can be a quality detection model including a feature extraction module, an orientation detection module and a three-dimensional annotation box detection module.

[0148] Among them, the feature extraction module can be a convolutional neural network (CNN), such as Resnet50, etc. The feature extraction module is used to extract various feature information that can characterize the target from the input data, such as mining the texture, shape, color and other aspects of the target, and combine multiple related features together to form an ordered feature vector.

[0149] The orientation detection module and the 3D annotation box detection module can be fully connected layers (FC). The orientation detection module uses the feature information provided by the feature extraction module to accurately detect the orientation of the target. The 3D annotation box detection module determines the accuracy of the 3D annotation box of the target based on the feature information extracted by the feature extraction module.

[0150] Furthermore, in order to perform quality inspection on the annotation data through an annotation quality inspection model with a simple structure and a small number of parameters, each frame of the annotation image in the annotation data can be cropped into sub-images corresponding to each target according to the annotation information, and each sub-image and the corresponding annotation information can be input into the annotation quality inspection model to obtain the annotation quality inspection data of each target in the original image sequence.

[0151] In some embodiments, see Appendix Figure 2 , Figure 2 FIG. 1 is an engineering schematic diagram of a data processing link according to an embodiment of the present application. Figure 2 As shown, the data processing link may include mining rules 21, data reflow 22, 4DGT data automatic labeling 23, manual labeling 24, labeling quality detection model 25, manual quality detection 26 and post-processing 27.

[0152] Among them, mining rules 21 is to mine the laws and features in the data; data reflow 22 returns business-related data to the production chain; 4DGT data automatic labeling 23 is to automatically label the collected original image sequence through the target detection model to obtain four-dimensional GT data containing time information and spatial information. Furthermore, the automatically labeled data is manually labeled 24, and the manually labeled data is manually quality inspected 26. The quality inspectors will check the quality of the labeled data. The data that has been manually quality inspected can be post-processed through the Ocearnus platform 27, including data cleaning and optimization, to ensure that the data can be used by the downstream autonomous driving system to achieve control of the vehicle's driving status.

[0153] In addition, if Figure 2 As shown, it also includes a feedback mechanism of the annotation quality detection model. After the 4DGT data is automatically labeled 23 or manually labeled 24, an automatic quality detection request can be sent to the annotation quality detection model 25. The annotation quality detection model 25 performs quality detection on the automatically labeled or manually labeled data, filters out possible erroneous labeled data, and sends the quality detection results to the manual quality detection 26 to help quality inspectors perform quality assessment, save the original cost of manual quality inspection, and make the spot check of manual quality inspection more targeted.

[0154] The annotation quality detection model can automatically detect large-scale annotation data, effectively identify incorrect annotations or noise, reduce manual quality inspection costs, and improve quality inspection efficiency and quality.

[0155] The above is a further description of step S102.

[0156] Furthermore, the true value data, annotation data, and annotation quality detection data corresponding to the original image sequence can be processed through a data processing platform, such as the Oceanus platform, to obtain a processed data set.

[0157] In some implementations of the above step S103, see the attached Figure 3 , Figure 3 1 is a flow chart of the main steps of determining the data learning difficulty attribute of the labeled data based on the true value data according to an embodiment of the present application. Figure 3 As shown, the following steps S301 to S302 are included.

[0158] Step S301: Matching the true value data and the labeled data;

[0159] In some implementations, a Hungarian matching algorithm may be used to match true value data and labeled data in a category+center distance manner.

[0160] The Hungarian matching algorithm is mainly used to solve the problem of optimal matching between two groups of elements. Specifically, the optimal match of two groups of data (true value data and labeled data) can be calculated based on the distance between the category and the center point, and the best one-to-one correspondence between the target in the true value data and the target in the labeled data can be found.

[0161] Specifically, step S301 may include the following steps S3011 to S3013:

[0162] Step S3011: Obtain the category information and the position information of the center point of the annotation box corresponding to each target in the true value data and the annotation data respectively;

[0163] For each target, such as vehicles, pedestrians, etc., its category information and the location information of the center point of the annotation box can be extracted from the true value data and the annotation data respectively.

[0164] The category information is used to determine the type of the target, and the position information is used to determine the specific position of the target in the image, which may be the center point coordinates of a two-dimensional annotation box or a three-dimensional annotation box.

[0165] Step S3012: Based on the position information, obtain the center point distance of each target in the true value data and the labeled data;

[0166] That is, according to the position of the center point of the same target in the true value data and the labeled data, the distance of the center point is calculated according to the Euclidean formula and other distance formulas.

[0167] Step S3013: Obtain matching results based on category information and center point distance.

[0168] Specifically, when the categories are the same and the center point distance is less than a preset distance, it is determined that the target matching is successful; otherwise, it is determined that the target matching fails.

[0169] That is, when the distance between the center points of targets of the same category in the true value data and the labeled data is less than the preset distance, it is determined that the target is successfully matched in the true value data and the labeled data; otherwise, it is determined that the target fails to match in the true value data and the labeled data.

[0170] Matching failure includes any of the following:

[0171] The categories of the target in the true value data and the labeled data are different;

[0172] Targets of the same category are only in the true data, not in the labeled data; or

[0173] Objects with the same category are only in the labeled data, not in the ground-truth data.

[0174] The above is a further description of step S301.

[0175] Step S302: Determine whether the labeled data is the first sample data or the second sample data based on the matching result.

[0176] Specifically, the labeled data of the target with the same category and only in the true value data but not in the preset labeled data can be used as the first sample data; the labeled data of the target that is successfully matched can be used as the second sample data. The first sample data can be difficult sample data, and the second sample data can be simple sample data.

[0177] When a target is only in the true value data but not labeled in the labeled data output by the target detection model, it means that the target detection model cannot effectively identify the target. From the perspective of model learning, this situation reflects that the target detection model has not yet mastered the key features or rules for identifying such targets. At this time, the labeled data related to this target can be identified as the first sample data.

[0178] When a target is marked in both the true value data and the labeled data output by the target detection model, it means that the target detection model accurately identifies the target. At this time, the labeled data related to the target can be identified as the second sample data.

[0179] Through the above implementation, it is possible to mine difficult samples according to the difficulty of data learning and improve the quality of labeled data, thereby improving the accuracy and robustness of the autonomous driving business model and optimizing the user's driving experience.

[0180] The above is a further description of step S103 , and the following is a further description of step S104 .

[0181] In some implementations of the above step S104, see the attached Figure 4 , Figure 4 FIG. 1 is a flow chart of the main steps of processing the label quality inspection data to obtain the inspection data meeting the preset conditions according to an embodiment of the present application. Figure 4 As shown, the following steps S401 to S403 are included.

[0182] Step S401: Aggregate the label quality detection data of each target in the original image sequence;

[0183] That is, the annotation quality detection data of all objects in multiple frames of original images are summarized to obtain the annotation quality detection data of the original image sequence.

[0184] Step S402: Obtain the number of occurrences of each target in multiple frames of original images;

[0185] Specifically, multiple frames of original images may be traversed and analyzed to determine the number of frames in which each target appears, thereby obtaining its number of occurrences.

[0186] For example, in a traffic scene containing 100 frames of original images, a car appears in 30 of the original images, so the number of times the car appears in multiple frames of original images is 30 times; a pedestrian only appears in 10 of the images, so the number of times he appears is 10 times.

[0187] Step S403: Determine targets whose occurrence times are greater than or equal to a preset threshold, and use the labeling quality detection data corresponding to the targets whose occurrence times are greater than or equal to the preset threshold as detection data that meets the preset conditions.

[0188] The preset threshold includes a first preset threshold and a second preset threshold, and the first preset threshold is smaller than the second preset threshold.

[0189] Specifically, targets whose consecutive appearance times in multiple frames of original images are greater than or equal to a first preset threshold may be determined.

[0190] In the original image sequence, if a target appears in multiple consecutive frames of original images, it means that the target has a certain stability and coherence, and then it can be determined that the annotation quality detection data corresponding to the target meets the preset conditions.

[0191] For example, the original image sequence of a surveillance video includes 300 frames of original images, and the first preset threshold is set to 200. If a pedestrian appears in 220 consecutive frames of images, then the annotation quality detection data corresponding to the pedestrian meets the preset conditions; if a vehicle only appears in 180 consecutive frames of images, then the annotation quality detection data corresponding to the vehicle does not meet the preset conditions.

[0192] In some implementations, targets whose total number of appearances in multiple frames of original images is greater than or equal to a second preset threshold may be determined.

[0193] In the original image sequence, even if an object does not appear continuously in multiple frames of original images, it will be determined as a qualified object as long as the total number of its appearances is greater than or equal to the second preset threshold. Non-continuously appearing objects may appear and disappear in the image sequence, but they are still more important because of their high total frequency of appearance.

[0194] For example, an original image sequence of a traffic scene includes 300 frames of original images, and the second preset threshold is set to 240. If a vehicle appears a total of 252 times in the 300 frames of original images, then the annotation quality detection data corresponding to the vehicle meets the preset conditions.

[0195] Furthermore, the annotation quality detection data corresponding to the target meeting the preset conditions may be stored in a corresponding annotation file in a preset data type.

[0196] The above is a further description of step S104. The following is a further description of step S105.

[0197] In some implementations of the above step S105, the annotation quality detection may include an annotation quality detection score, which may specifically include a quality detection score of the target orientation, a quality detection score of the target three-dimensional annotation box, or a comprehensive quality detection score of the target orientation and the three-dimensional annotation box.

[0198] Furthermore, the quality of the labeled data can be judged by the target's labeling quality detection score to obtain a processed data set.

[0199] For details, see the attached Figure 5 , Figure 5 1 is a flow chart of the main steps of obtaining a processed data set based on the data learning difficulty attribute and the detection data that meets the preset conditions according to an embodiment of the present application. Figure 5 As shown, it mainly includes the following steps S501 to S504.

[0200] Step S501: Among the detection data that meet the preset conditions, the target labeled data whose labeling quality detection score is higher than the preset score is regarded as high-quality labeled data;

[0201] When the annotation quality detection score is higher than a certain score, it means that the annotation data has reached a relatively high level in quality, has high credibility and value, and can play an important role in subsequent data analysis, model training and other work. For example, it can be used to train more accurate and reliable machine learning models, thereby improving the performance and prediction effect of the model.

[0202] Step S502: Acquire the first sample data and the second sample data in the high-quality labeled data according to a preset ratio to obtain a processed data set.

[0203] Among them, the first sample data can be difficult sample data, whose characteristics are complex and difficult to process. When the model is trained with the first sample data, the generalization ability of the model can be improved; the second sample data can be simple sample data, whose characteristics are obvious and easy to process. When the model is trained with the second sample data, the model can converge quickly, but it is prone to overfitting.

[0204] Specifically, more first sample data and less second sample data may be obtained from the high-quality labeled data, such as obtaining the first sample data and the second sample data from the high-quality labeled data in a ratio of 7:3.

[0205] Among them, the first sample data usually represents the complex situation in the data, containing more information and changes. By increasing the proportion of the first sample data, when the model is subsequently trained through the data set, the model can learn more comprehensive and complex features, so as to better cope with various different input situations, improve the generalization ability on unknown data, and reduce the risk of overfitting. In addition, the first sample data is usually where the model is prone to errors. Increasing the proportion of the first sample data can balance the influence of the second sample data and the first sample data, so that the model can learn the characteristic distribution of the data more comprehensively, improve the model's prediction accuracy for the first sample data, and thus improve the overall performance.

[0206] Step S503: Among the detection data meeting the preset conditions, the target labeled data whose labeling quality detection score is lower than or equal to the preset score is regarded as low-quality labeled data;

[0207] Step S504: remove low-quality labeled data.

[0208] When the annotation quality detection score is lower than a certain score, it means that the annotation data is not accurate enough. There may be errors and noise in the annotation data (i.e., "dirty data"), which will have a negative impact on subsequent data analysis, model training and other work. Therefore, it is necessary to remove low-quality annotation data to ensure that the data used for subsequent processing has a relatively high quality.

[0209] In some embodiments, see Appendix Figure 6 , Figure 6 It is a schematic diagram of data learning difficulty attribute and labeled data quality according to an embodiment of the present application.

[0210] like Figure 6 As shown in the figure, the horizontal axis of the two-dimensional coordinate system represents the quality of the labeled data, and the quality gradually improves from left to right; the vertical axis represents the difficulty of learning, and the difficulty of learning gradually increases from bottom to top.

[0211] Among them, the labeled data of the target whose labeling quality detection score is higher than the preset score among the detection data that meet the preset conditions can be retained as high-quality labeled data (i.e., the right half axis of the horizontal axis); the labeled data of the target whose labeling quality detection score is lower than or equal to the preset score can be removed as low-quality labeled data (i.e., the right half axis of the horizontal axis);

[0212] Furthermore, the sampling of the first sample data (ie, the first quadrant) in the high-quality labeled data may be increased and the sampling of the second sample data (ie, the fourth quadrant) may be reduced according to a preset ratio to obtain a processed data set.

[0213] It should be pointed out that different types of sample data are of great significance for subsequent model training. In practical applications, technicians in this field can reasonably adjust the ratio of the first sample data and the simple sample to improve the classification or prediction capabilities of different types of data, which is not limited here.

[0214] The above is the description of step S105.

[0215] Furthermore, in some embodiments, after obtaining the processed data set through the above steps S101 to S105, the target detection model can be iteratively trained based on the processed data set until the target detection model converges to a preset error value, so as to improve the performance of the target detection model and thus improve the accuracy and efficiency of the model.

[0216] In addition, the processed data set can also be used to evaluate and optimize business models, or perform data analysis, etc., which is not limited here.

[0217] For further information, see Appendix Figure 7 , Figure 7 It is a main flow chart of a data processing method according to an embodiment of the present application.

[0218] like Figure 7As shown, firstly, an original image sequence 71 is obtained, and the true value data 72 corresponding to the original image sequence 71 is obtained and stored in a labeling file 73, and then the labeling data 75 of the original image sequence is obtained based on a target detection model 74, and the labeling quality detection data is obtained based on a labeling quality detection model 76 and the labeling data 75;

[0219] Further, the data learning difficulty attribute 77 of the labeled data can be determined based on the true value data 72, and the labeled quality detection data can be processed to obtain detection data 78 that meets the preset conditions, and then the labeled data of the target with a label quality detection score higher than the preset score in the detection data that meets the preset conditions is taken as high-quality labeled data 79, and the labeled data of the target with a label quality detection score lower than or equal to the preset score is taken as low-quality labeled data 710;

[0220] Further, the low-quality annotated data 710 is removed, and the first sample data and the second sample data in the high-quality annotated data are obtained according to a preset ratio to obtain a processed data set 711;

[0221] Finally, the target detection model 74 is iteratively trained based on the processed data set 711 until the target detection model converges to a preset error value.

[0222] Through the above implementation method, dirty data can be automatically identified and filtered, which improves the quality inspection efficiency of labeled data, reduces the impact of erroneous labeled data on the business model, improves the accuracy of the business model, and realizes the automation of the quality inspection process of newly added labeled data, thereby ensuring the quality of labeled data and optimizing the smart driving user experience.

[0223] The data processing method provided in this application can be flexibly applied in various scenarios such as business model training and data reflux annotation. It can enhance the accuracy and robustness of the autonomous driving business model and effectively optimize the user's riding experience.

[0224] The above is an explanation of the data processing method provided in this application.

[0225] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effect of the present application, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted schemes are equivalent to the technical schemes described in this application, and therefore will also fall within the scope of protection of this application.

[0226] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above-mentioned embodiment of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of each of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The non-temporary computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electrical carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0227] Furthermore, the present application also provides a smart device.

[0228] In an embodiment of an intelligent device according to the present application, the intelligent device may include at least one processor; and a memory connected to the at least one processor in communication; wherein a computer program is stored in the memory, and when the computer program is executed by at least one processor, the method described in any of the above embodiments is implemented. The processor may include hardware, software, or a combination of the two. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware, or a combination of the two. The intelligent device described in the present application may include driving devices, smart cars, robots, and other devices.

[0229] In some embodiments of the present application, the smart device may further include at least one sensor, which is used to sense information and is communicatively connected to any type of processor mentioned in the present application.

[0230] See attached Figure 8 , Figure 8 is a schematic diagram of a smart device according to an embodiment of the present application. Figure 8 As shown, the smart device may include a memory 81, a processor 82 and a sensor 83, wherein the memory 81, the processor 82 and the sensor 83 are communicatively connected via a bus.

[0231] Optionally, the smart device may also include an autonomous driving system, which is used to guide the smart device to drive itself or assist in driving. The processor communicates with the sensor and / or the autonomous driving system to complete the method described in any of the above embodiments. (The smart device described in this application may be, but is not limited to, a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, a vehicle-mounted device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), an augmented reality (AR)\virtual reality (VR) device, etc., and the embodiments of this application are not limited to this.)

[0232] Furthermore, the present application also provides a computer-readable storage medium.

[0233] In an embodiment of a computer-readable storage medium according to the present application, the computer-readable storage medium may be configured to store a program for executing the data processing method of the above method embodiment, and the program may be loaded and run by a processor to implement the above data processing method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium may be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-temporary computer-readable storage medium.

[0234] It should be pointed out that the relevant user personal information that may be involved in the various embodiments of the present application shall be strictly in accordance with the requirements of laws and regulations, follow the principles of legality, legitimacy and necessity, and based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0235] The user personal information processed by this application will vary depending on the specific product / service scenario, and shall be based on the specific scenario in which the user uses the product / service, and may involve the user's account information, device information, driving information, vehicle information or other related information. This application will treat the user's personal information and its processing with a high degree of diligence.

[0236] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that meet industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0237] So far, the technical solution of the present application has been described in conjunction with an embodiment shown in the accompanying drawings, but it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain the true value data corresponding to the original image sequence; Acquire annotation data of the original image sequence, and acquire annotation quality detection data based on an annotation quality detection model and the annotation data; Determining a data learning difficulty attribute of the labeled data based on the true value data; Processing the labeling quality inspection data to obtain inspection data that meets preset conditions; A processed data set is obtained based on the data learning difficulty attribute and the detection data that meets the preset conditions.

2. The data processing method according to claim 1, characterized in that: The step of acquiring the annotation data of the original image sequence comprises: Inputting multiple frames of original images in the original image sequence into the target detection model to obtain multiple frames of model annotated images corresponding to the multiple frames of original images; or obtaining multiple frames of manually annotated images corresponding to the multiple frames of original images; The annotation data is obtained based on the multiple frames of model annotation images or the multiple frames of manually annotated images.

3. The data processing method according to claim 2, characterized in that: The acquiring of the labeling quality detection data based on the labeling quality detection model and the labeling data comprises: The annotation data is input into the annotation quality detection model to obtain annotation quality detection data of each target in the original image sequence.

4. The data processing method according to claim 3, characterized in that: The step of processing the labeling quality detection data to obtain detection data that meets preset conditions includes: Aggregating the label quality detection data of each of the targets in the original image sequence; Obtaining the number of occurrences of each of the targets in the multiple frames of original images; The target whose occurrence number is greater than or equal to a preset threshold is determined, and the labeling quality detection data corresponding to the target whose occurrence number is greater than or equal to the preset threshold is used as the detection data that meets the preset condition.

5. The data processing method according to claim 4, characterized in that: The preset threshold includes a first preset threshold and a second preset threshold; Determining that the number of occurrences is greater than or equal to a preset threshold comprises: Determine targets whose consecutive appearance times in the multiple frames of original images are greater than or equal to the first preset threshold; Or, determining targets whose total number of occurrences in multiple frames of original images is greater than or equal to the second preset threshold; The first preset threshold is smaller than the second preset threshold.

6. The data processing method according to claim 1, characterized in that: The data learning difficulty attribute includes first sample data and second sample data, and the data learning difficulty of the first sample data is higher than that of the second sample data; and determining the data learning difficulty attribute of the labeled data based on the true value data includes: Matching the true value data with the labeled data; The labeled data is determined to be the first sample data or the second sample data based on the matching result.

7. The data processing method according to claim 6, characterized in that: The matching of the true value data and the labeled data comprises: Respectively obtain the category information and the position information of the center point of the annotation box corresponding to each of the targets in the true value data and the annotation data; Based on the position information, obtaining the center point distance of each target in the true value data and the labeled data; The matching result is obtained based on the category information and the center point distance.

8. The data processing method according to claim 7, characterized in that: The obtaining of the matching result based on the category information and the center point distance comprises: When the categories are the same and the center point distance is less than a preset distance, it is determined that the target is matched successfully; Otherwise, determining that the target matching fails; The matching failure includes any of the following: The categories of the target in the true value data and the labeled data are different; The target of the same category is only in the true value data and not in the labeled data; or The objects of the same category are only in the labeled data but not in the true value data.

9. The data processing method according to claim 8, characterized in that: The determining, based on the matching result, that the labeled data is the first sample data or the second sample data comprises: Taking the labeled data of the target that has the same category and is only in the true value data and not in the labeled data as the first sample data; The labeled data of the target that is successfully matched is used as the second sample data.

10. The data processing method according to claim 9, characterized in that: The labeling quality detection data includes a labeling quality detection score; the data set processed based on the data learning difficulty attribute and the detection data meeting the preset conditions includes: Among the detection data meeting the preset conditions, the target labeled data whose labeling quality detection score is higher than the preset score is used as high-quality labeled data; According to a preset ratio, the first sample data and the second sample data in the high-quality labeled data are obtained to obtain the processed data set.

Images