Model training method and device based on pre-labeled data, electronic device and medium

Abstract

The application discloses a model training method and device based on pre-labeled data, electronic equipment and medium. The method comprises the following steps: adding a pre-labeled data frame to an artificial labeling training data frame, and training the artificial labeling training data frame after the addition; in the training process, determining a confusion matrix error rate according to the number of false detections, the preset recall target number of the current class pre-labeled data frame and the number of false detections between different class pre-labeled data frames, and determining an overall error rate according to the preset recall target number of the different class pre-labeled data frame and the current class pre-labeled data frame; and adjusting a target attribute regression loss weight coefficient according to the confusion matrix error rate and the overall error rate, so as to calculate a loss value of the labeling data frame according to the adjusted weight coefficient. In this way, the influence of missed detection on model training is reduced, the long tail effect of pre-labeled data is alleviated, and the influence of pre-labeled data false detection of the model is also reduced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a model training method based on pre-labeled data, a model training device based on pre-labeled data, a computer-readable storage medium, and an electronic device. Background Technology

[0002] Autonomous driving technology is crucial for alleviating traffic problems and is currently a focal point of competition among major automakers. However, deep learning-based autonomous driving solutions require massive amounts of data, and accurate labeling and access to large volumes of data are costly. Furthermore, the data collection process inherently results in a long-tail distribution of collected data, with a majority of data consisting of vehicles and pedestrians, while smaller targets such as traffic cones, irregularly shaped vehicles, and tricycles, even with customized data collection, do not constitute a significant portion of the data. To alleviate this data demand issue, most solutions involve replicating the smaller targets, but this inevitably introduces larger targets, resulting in limited overall effectiveness. Another common approach is to train a deeper model with more parameters than the on-board model to predict unlabeled data, obtaining its coordinates, size, confidence level, and other attributes—known as pre-labeled bounding boxes. A training strategy is then designed using this pre-labeled data, primarily involving labeling low-confidence predicted boxes without including them in loss calculations. However, this training strategy inevitably introduces several problems. The first problem is the existence of serious false detections in the large model, including false detections of noise and false detections between categories, such as false detections of pedestrians, cones, and warning posts. Increasing the confidence level can reduce false detections, but it cannot completely eliminate them. The second problem is the serious under-detection in the pre-labeled data of the large model, that is, there are ground truth targets in the background. When calculating the loss value, the model will identify the target information as background information, which incorrectly guides the calculation of the model gradient. The third problem is that the pre-labeled data of the large model and the manually labeled data are homogenized, and the pre-labeled bounding boxes still have a serious long-tail distribution.

[0003] In related technologies, patent application CN202010450890.1, entitled "Target Detection Model Training Method, Target Detection Method and Apparatus," discloses a loss function using two factors: interval value and distance, to determine the loss value corresponding to the predicted box. However, it does not consider the impact of false detections between targets on the loss value. Patent application CN202311708714.3, entitled "Target Detection Model Training Method, Detection Method and Related Apparatus," discloses training the target detection model using ground truth and fully labeled data from weakly labeled long-tailed object data. However, it does not consider the probability issue when adding pre-labeled boxes to the human labeled data frame. Summary of the Invention

[0004] One objective of this invention is to provide a model training method based on pre-labeled data to improve the performance of vehicle-side models in extracting general features, while alleviating the severe long-tail distribution phenomenon of manually labeled data and reducing the impact of unavoidable false detections and false negatives on training in large models; another objective is to provide a model training device based on pre-labeled data; a third objective is to provide a computer-readable storage medium; and a fourth objective is to provide an electronic device.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A model training method based on pre-labeled data includes: adding pre-labeled data frames to manually labeled training data frames, and training on the manually labeled training data frames after the addition, wherein the pre-labeled data frames are added to the manually labeled training data frames based on the addition probability of the pre-labeled data and the position information of the pre-labeled data frames added to the manually labeled training data frames; during the training process, obtaining a confusion matrix between pre-labeled data frames of different classes; determining the number of false detections of other class pre-labeled data frames identified as the current class pre-labeled data frames based on the confusion matrix; determining the confusion matrix error rate based on the number of false detections, the preset recall target number of the current class pre-labeled data frames, and the number of false detections between different class pre-labeled data frames, and determining the overall error rate based on the preset recall target number of different class pre-labeled data frames and the current class pre-labeled data frames; adjusting the target attribute regression loss weight coefficients based on the confusion matrix error rate and the overall error rate, so as to calculate the loss value of the labeled data frames based on the adjusted weight coefficients, and performing backpropagation in the target detection model based on the loss value of the pre-labeled data frames, and updating the weight parameters of the target detection model during the backpropagation process.

[0007] According to one embodiment of the present invention, obtaining the probability of adding pre-labeled data includes: obtaining the ratio between the number of manually labeled training data frames and the number of pre-labeled data frames; if the ratio is less than a preset threshold, using the ratio as the probability of adding pre-labeled data; if the ratio is greater than the preset threshold, determining the probability of adding pre-labeled data based on the number of manually labeled training data frames, the ratio, and the number of pre-labeled data frames.

[0008] According to one embodiment of the present invention, determining the probability of adding pre-labeled data based on the number of manually labeled training data frames, the ratio, and the number of pre-labeled data frames includes: obtaining the product between the number of pre-labeled data frames of type i and the ratio; obtaining the difference between the number of manually labeled training data frames and the product to determine the remainder of the ratio between the number of manually labeled training data frames and the number of pre-labeled data frames of type i; and determining the probability of adding pre-labeled data based on the ratio between the remainder and the number of pre-labeled data frames of type i.

[0009] According to one embodiment of the present invention, obtaining the position information of a pre-annotated data frame added to a manually annotated training data frame includes: determining that the pre-annotated data frame is occluded by the manually annotated data frame based on the position information of the pre-annotated data frame and the attribute information of the manually annotated data frame; if the attribute information of the pre-annotated data frame determines that occlusion exists, keeping the center point position information of the pre-annotated data frame unchanged; if the attribute information of the pre-annotated data frame determines that occlusion does not exist, moving the pre-annotated data frame.

[0010] According to one embodiment of the present invention, obtaining the position information of a pre-annotated data frame added to a manually annotated training data frame includes: determining that the current pre-annotated frame is not occluded by the manually annotated frame based on the position information of the pre-annotated data frame and the attribute information of the manually annotated target frame; keeping the position information of the pre-annotated data frame unchanged if it is determined that there is no occlusion based on the attribute information of the previous pre-annotated data frame; and moving the pre-annotated data frame if it is determined that there is occlusion based on the attribute information of the pre-annotated data frame.

[0011] According to an embodiment of the present invention, moving a pre-annotated data frame includes: determining minimum position information and maximum position information based on the position information and size information of each manually annotated data frame, wherein the minimum position information and maximum position information include values ​​in the x, y, and z directions; obtaining a first absolute value of the difference between the position information of the pre-annotated data frame and the minimum position information and a second absolute value of the difference between the position information of the pre-annotated data frame and the maximum position information; and moving the pre-annotated data frame according to the first absolute value of the difference and the second absolute value of the difference, wherein the first absolute value of the difference and the second absolute value of the difference include values ​​in the x, y, and z directions.

[0012] According to one embodiment of the present invention, moving a pre-labeled data frame based on a first absolute value and a second absolute value of difference includes: moving the position information of the pre-labeled data frame to the maximum position information when the first absolute value of difference is greater than the second absolute value of difference; and moving the position information of the pre-labeled data frame to the minimum position information when the first absolute value of difference is less than or equal to the second absolute value of difference.

[0013] According to one embodiment of the present invention, determining minimum and maximum position information based on the position and size information of each manually labeled data frame includes: obtaining the difference between the x-direction coordinate of all manually labeled data frames and half the length of the corresponding manually labeled data frame, forming a first x-direction position set for all manually labeled data frames, and taking the minimum value in the first x-direction position set as the minimum position in the x-direction; obtaining the difference between the y-direction coordinate of all manually labeled data frames and half the width of the corresponding manually labeled data frame, forming a first y-direction position set for all manually labeled data frames, and taking the minimum value in the first y-direction position set as the minimum position in the y-direction; obtaining the difference between the z-direction coordinate of all manually labeled data frames and half the height of the corresponding manually labeled data frame, forming a first z-direction position set for all manually labeled data frames, and taking the minimum value in the first z-direction position set as the minimum position in the z-direction; obtaining all manually labeled data frames... The sum of the x-coordinates of all manually labeled data frames and half the length of the corresponding manually labeled data frame constitutes the second x-position set of all manually labeled data frames, and the maximum value in the second x-position set is taken as the maximum position in the x-direction. The sum of the y-coordinates of all manually labeled data frames and half the width of the corresponding manually labeled data frame constitutes the second y-position set of all manually labeled data frames, and the maximum value in the second y-position set is taken as the maximum position in the y-direction. The sum of the z-coordinates of all manually labeled data frames and half the height of the corresponding manually labeled data frame constitutes the second z-position set of all manually labeled data frames, and the maximum value in the second z-position set is taken as the maximum position in the z-direction. The minimum position information is determined based on the minimum positions in the x, y, and z directions, and the maximum position information is determined based on the maximum positions in the x, y, and z directions.

[0014] According to one embodiment of the present invention, determining the confusion matrix error rate based on the number of false detections, the preset recall target number of the current class pre-labeled data frame, and the number of false detections between different class pre-labeled data frames includes: determining the number of false detections between different class pre-labeled data frames based on the sum of the number of false detections and the number of noisy false detections; and determining the confusion matrix error rate based on the ratio between the number of false detections and the sum of the number of false detections between different class pre-labeled data frames and the preset recall target number of the current class pre-labeled data frame.

[0015] According to one embodiment of the present invention, determining the overall error rate based on the preset recall target number of different types of pre-labeled data frames and the current type of pre-labeled data frame includes: obtaining the sum of the number of false positives among different types of pre-labeled data frames and the preset recall target number of the current type of pre-labeled data frame; and determining the overall error rate based on the ratio between the number of false positives among different types of pre-labeled data frames and the sum.

[0016] According to an embodiment of the present invention, the target attribute regression loss weight coefficient includes: the weight coefficient of the classification loss of the current pre-labeled data frame, the weight coefficient of the regression loss of the center point of the current pre-labeled data frame, the weight coefficient of the regression loss of the center point of the current pre-labeled data frame, and the weight coefficient of the regression loss of the size of the current pre-labeled data frame. The adjustment of the target attribute regression loss weight coefficient based on the confusion matrix error rate and the overall error rate includes: obtaining the difference between a preset value and the confusion matrix error rate as a first probability, obtaining the difference between the preset value and the overall error rate as a second probability; multiplying the weight coefficient of the classification loss of the current pre-labeled data frame by the first probability to obtain its adjusted weight coefficient; and multiplying the weight coefficient of the center point of the current pre-labeled data frame by the first probability. The weighted coefficient of the regression loss is multiplied by the first probability to obtain its adjusted weighted coefficient; the weighted coefficient of the loss value of the current pre-labeled data frame height is multiplied by the first probability and then by the second probability to obtain its adjusted weighted coefficient; the weighted coefficient of the regression loss of the current pre-labeled data frame size is multiplied by the first probability and then by the second probability to obtain its adjusted weighted coefficient; the sum of the four factors—the classification loss of the current pre-labeled data frame multiplied by its adjusted weighted coefficient, the regression loss of the current pre-labeled data frame center point multiplied by its adjusted weighted coefficient, the loss value of the current pre-labeled data frame height multiplied by its adjusted weighted coefficient, and the regression loss of the current pre-labeled data frame size multiplied by its adjusted weighted coefficient—is used as the loss value of the labeled data frame.

[0017] A model training device based on pre-labeled data includes: an adding module for training on manually labeled training data frames after adding pre-labeled data frames to manually labeled training data frames, wherein the pre-labeled data frames are added to the manually labeled training data frames based on the addition probability of the pre-labeled data and the position information of the pre-labeled data frames added to the manually labeled training data frames; an acquisition module for acquiring a confusion matrix between different classes of pre-labeled data frames during training; a first determination module for determining the number of false positives when other classes of pre-labeled data frames are identified as the current class of pre-labeled data frames based on the confusion matrix; and a second determination module. The module is used to determine the confusion matrix error rate based on the number of false detections, the preset recall target number of the current class pre-labeled data frame, and the number of false detections between different class pre-labeled data frames, and to determine the overall error rate based on the preset recall target number of different class pre-labeled data frames and the current class pre-labeled data frame; the adjustment module is used to adjust the target attribute regression loss weight coefficients based on the confusion matrix error rate and the overall error rate, so as to calculate the loss value of the labeled data frame based on the adjusted weight coefficients, and to perform backpropagation in the target detection model based on the loss value of the pre-labeled data frame, and update the weight parameters of the target detection model during the backpropagation process.

[0018] A computer-readable storage medium storing a model training program based on pre-labeled data, which, when executed by a processor, implements the aforementioned model training method based on pre-labeled data.

[0019] An electronic device includes a memory, a processor, and a model training program based on pre-labeled data stored in the memory and executable on the processor. When the processor executes the model training program based on pre-labeled data, it implements the aforementioned model training method based on pre-labeled data.

[0020] The beneficial effects of this invention are:

[0021] (1) The pre-labeled data of the model inevitably contains missed detections. If such missed detections are not considered during the training process, the model will train the missed target point cloud as the background point cloud, which will lead to incorrect guidance of the gradient information of the model. Therefore, in the training process, the model pre-labeled data is added to the manually labeled data to reduce the impact of large model missed detections on training.

[0022] (2) The model annotation data and the manually annotated data are co-distributed. Therefore, the pre-annotated data of large models also have a serious long-tail distribution phenomenon. In order to alleviate the long-tail distribution phenomenon of pre-annotated data, this invention designs different addition probabilities for pre-annotated data, which can ensure the uniform distribution of pre-annotated data to a certain extent.

[0023] (3) There are still serious false detections in the model pre-labeled data. Therefore, this invention redesigns the loss weights between target attributes based on the false detection type, which can greatly reduce the impact of false detections in the model pre-labeled data. Attached Figure Description

[0024] Figure 1 This is a flowchart of a model training method based on pre-labeled data according to some embodiments of the present invention;

[0025] Figure 2 A flowchart of a probability determination method for adding pre-labeled data according to some embodiments of the present invention;

[0026] Figure 3 This is a flowchart of a method for moving a pre-labeled data frame according to some embodiments of the present invention;

[0027] Figure 4 A flowchart illustrating a method for determining the target attribute regression loss weight coefficient according to some embodiments of the present invention;

[0028] Figure 5 This is a block diagram of a model training apparatus based on pre-labeled data according to some embodiments of the present invention;

[0029] Figure 6This is a block diagram of an electronic device according to some embodiments of the present invention. Detailed Implementation

[0030] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0031] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0032] The following describes in detail, with reference to the accompanying drawings, the model training method, apparatus, electronic device, and medium based on pre-labeled data according to embodiments of the present invention.

[0033] In some embodiments, the pre-labeled model data inevitably contains missed detections. If these missed detections are not considered during training, the model will train the missed target point cloud as the background point cloud, leading to incorrect gradient information. Therefore, during training, the pre-labeled model data needs to be added to the manually labeled data to reduce the impact of large model missed detections on training. However, the model labeled data and the manually labeled data are homogenized, so the large model pre-labeled data also exhibits a severe long-tail distribution. To alleviate the long-tail distribution of the pre-labeled data, different addition probabilities are designed for the pre-labeled data, which can ensure a certain degree of uniform distribution of the pre-labeled data. In addition, the pre-labeled model data also has a relatively serious false detection phenomenon. Therefore, the model training method of this application redesigns the loss weights between target attributes based on the false detection type, which can significantly reduce the impact of false detections in the pre-labeled model data.

[0034] Figure 1 This is a flowchart illustrating a model training method based on pre-labeled data according to some embodiments of the present invention. (Refer to...) Figure 1 The model training method based on pre-labeled data in this application embodiment may include the following steps:

[0035] S110, when the pre-labeled data frame is added to the manually labeled training data frame, the manually labeled training data frame after the addition is trained. The pre-labeled data frame is added to the manually labeled training data frame based on the addition probability of the pre-labeled data and the position information of the pre-labeled data frame added to the manually labeled training data frame.

[0036] Specifically, pre-labeled data includes multiple data types, such as type I pre-labeled data, type II pre-labeled data, ..., type i pre-labeled data. The number of pre-labeled data frames can be represented as {s1, s2, ..., s...} i}, where s1 represents the number of first-class data frames, s2 represents the number of second-class data frames, ..., s i This represents the number of data frames of type i. During model training, each type of pre-labeled data frame should be added to each manually labeled training data frame, and only one pre-labeled data frame of one type can be added to each manually labeled training data frame in each training session. Therefore, for each type of pre-labeled data frame, the probability of adding it to the manually labeled training data frame is determined by the number of pre-labeled data frames and the number of manually labeled training data frames. For example, the addition probability can be the ratio of the number of manually labeled training data frames to the number of pre-labeled data frames, which to some extent ensures that the probability of each first-type pre-labeled data frame being added to the manually labeled training data frame is the same.

[0037] For example, assuming the number of pre-labeled data frames of the first type is 20 and the number of manually labeled training data frames is 10, then the addition probability is one in half. That is, half of the pre-labeled data frames of the first type should be selected and added to the manually labeled training data frames, and one pre-labeled data frame of the first type should be added to each manually labeled training data frame in each training session.

[0038] Furthermore, when adding pre-labeled data frames to manually labeled training data frames, the addition location information of each pre-labeled data frame must be determined based on its data attributes to ensure that the pre-labeled data frames maintain consistent imaging characteristics in both the original training data frame and the current manually labeled training data frame. Specifically, for each pre-labeled data frame, its data attributes can be represented as (num, x, y, z, l, w, h, Occlude, conf), where num represents the number of point clouds within the current pre-labeled data frame, (x, y, z) are the position coordinates of the pre-labeled data frame, (l, w, h) are the size of the pre-labeled data frame, Occlude is the occlusion label of the pre-labeled data frame, and conf represents the confidence level of the pre-labeled data frame. When adding pre-labeled data frames to manually labeled training data frames for training, the attributes of the pre-labeled data frames should be maintained to the greatest extent possible. For example, whether a pre-labeled data frame is occluded in the original training data frame affects whether it can maintain consistent imaging characteristics in both the original training data frame and the current manually labeled training data frame. The position coordinates of the pre-labeled data frame affect the number of point clouds within the frame and the confidence level.

[0039] Therefore, when adding a pre-labeled data frame to the current manually labeled training data frame, the pre-labeled data frame should maintain consistent occlusion properties. That is, if a pre-labeled data frame is occluded in the original training data frame, and becomes unoccluded after being added to the current manually labeled training data frame, then the addition position of the pre-labeled data frame needs to be determined based on its data attributes and the attribute information of the manually labeled data frames in the original manually labeled training data frame. Based on this position information, the pre-labeled data frame should be moved to maintain its occlusion property. Similarly, if a pre-labeled data frame is unoccluded in the original training data frame, and becomes occluded after being added to the current manually labeled training data frame, then the pre-labeled data frame also needs to be moved based on its position information to maintain its unoccluded property.

[0040] S120, during the training process, obtains the confusion matrix between different classes of pre-labeled data frames.

[0041] Specifically, the performance of a large model on the test set can be represented by two commonly used metrics: TP (True Positive) and FP (False Positive). TP represents the pre-defined recall number of pre-labeled data frames (the number of correctly identified targets), and FP represents the number of false positives between different classes of pre-labeled data frames (the number of falsely detected targets). The sum of TP and FP represents the total recall number. In this embodiment of the invention, FP is divided into two categories, specifically FP... noise and FP confusion_matrix , among which, FP noise FP represents the number of false noise detections generated by a noisy point cloud.confusion_matrix This indicates the number of false positives (the number of false positives between targets) that other pre-labeled data frames were identified as the current class pre-labeled data frame.

[0042] After successfully adding the pre-labeled data frame to the manually labeled training data frame to form the training dataset, the model is trained using this training dataset. The model is then used to predict the test dataset to obtain the prediction results. The samples in the test dataset are traversed, and the true type of each sample and the type predicted by the model are mapped to the corresponding position in the confusion matrix. The number of false positives between each type is counted, and finally, a confusion matrix between different classes of pre-labeled data frames is generated. Based on the confusion matrix, the number of false positives (FPconfusion_matrix) of other classes of pre-labeled data frames identified as the current class of pre-labeled data frames can be determined.

[0043] S130, determine the number of false positives that other class pre-labeled data frames are identified as the current class pre-labeled data frame based on the confusion matrix.

[0044] Specifically, the confusion matrix between different classes of pre-labeled data frames is as follows:

[0045]

[0046] Where, confusion_matrix represents the confusion matrix between different class pre-labeled data frames, n represents the number of target types trained, and α ij This represents the number of cases where the i-th class is mistakenly identified as the j-th class.

[0047] Then, based on the confusion matrix between different classes of pre-labeled data frames, the number of false detections where other classes of pre-labeled data frames are identified as class i can be determined as follows:

[0048] FP confusion_matrix =sum(α) 1,i +...+α i-1,i +α i+1,i +α n,i )

[0049] Among them, FP confusion_matrix This represents the number of false positives when other pre-labeled data frames are identified as pre-labeled data frames of class i. `sum()` represents the summation. α 1,i α represents the number of false positives of type 1 that are classified as type i. i-1,i α represents the number of false positives of class i (i-1) that are actually class i. i+1,i α represents the number of false positives of class i (i+1) belonging to class i. n,i This represents the number of false positives of class n that are classified as class i.

[0050] S140, determine the confusion matrix error rate based on the number of false positives, the preset recall target number of the current class pre-labeled data frame, and the number of false positives between different class pre-labeled data frames, and determine the overall error rate based on the preset recall target number of different class pre-labeled data frames and the current class pre-labeled data frame.

[0051] Specifically, as can be seen from the above embodiments, the number of false detections FP of other class pre-labeled data frames being identified as the current class pre-labeled data frame can be determined based on the confusion matrix. confusion_matrix The number of false noise detections (FP) formed by noise point clouds noise This can be obtained by comparing the prediction results with the test dataset. Therefore, the number of false positives (FP) between different classes of pre-labeled data frames can be calculated. i The number of false positives (FP) that identified other pre-labeled data frames as the current class pre-labeled data frame. confusion_matrix The number of false positives (FP) formed by noisy point clouds noise The sum. The current total recall target is the number of false positives (FP) between different class pre-labeled data frames. i And the preset recall target number TP of the current class pre-labeled data frame i The sum of these. Furthermore, the confusion matrix error rate refers to the number of false positives (FP) where other class pre-labeled data frames are identified as the current class pre-labeled data frame. confusion_matrix The percentage of the total recalled targets in the current class; the overall error rate refers to the number of false positives (FP) between pre-labeled data frames of different classes. i The percentage of the total number of targets to be recalled in the current category.

[0052] S150, adjust the target attribute regression loss weight coefficients according to the confusion matrix error rate and the overall error rate, so as to calculate the loss value of the labeled data frame based on the adjusted weight coefficients, and perform backpropagation in the target detection model based on the loss value of the pre-labeled data frame, and update the weight parameters of the target detection model during the backpropagation process.

[0053] Specifically, during model training, a loss value needs to be determined to measure model performance and guide parameter optimization. In this embodiment of the invention, the loss value includes the classification loss of the current pre-labeled data frame, the regression loss of the center point of the current pre-labeled data frame, the loss of the height of the current pre-labeled data frame, and the regression loss of the size of the current pre-labeled data frame, and each loss value has a corresponding loss weight coefficient. Since false detections between targets affect target classification, but the attributes of the pre-labeled data frame are not entirely incorrect, false detections caused by noisy point clouds not only affect target classification, but also make the attributes of the pre-labeled data frame incorrect. Therefore, the target attribute regression loss weight coefficient can be adjusted based on the confusion matrix error rate and the overall error rate, so that the loss value of the pre-labeled data frame can be calculated based on the adjusted weight coefficient, and backpropagation can be performed in the target detection model based on the loss value of the pre-labeled data frame, and the weight parameters of the target detection model can be updated during backpropagation.

[0054] For example, since both false positives between targets and false positives caused by noisy point clouds can affect target classification, the overall error rate is used to correct the loss weight coefficients corresponding to the classification loss of the current pre-labeled data frame and the regression loss of the current pre-labeled data frame center point. Because the attributes of the pre-labeled data frame are not entirely incorrect when there are false positives between targets, but are completely incorrect when there are false positives caused by noisy point clouds, the overall error rate and the confusion matrix error rate are used to correct the loss weight coefficients corresponding to the loss of the current pre-labeled data frame height and the regression loss of the current pre-labeled data frame size.

[0055] The model training method of this invention adds pre-labeled model data to manually labeled training data frames during the training process, which can reduce the impact of large model false negatives on training. At the same time, designing different addition probabilities for the pre-labeled data frames can ensure the uniform distribution of pre-labeled data to a certain extent, thereby alleviating the long-tail distribution phenomenon of pre-labeled data. In addition, redesigning the loss weights between target attributes based on the false detection type can significantly reduce the impact of false detections caused by model pre-labeled data.

[0056] In some embodiments, obtaining the probability of adding pre-labeled data includes: obtaining the ratio between the number of manually labeled training data frames and the number of pre-labeled data frames; if the ratio is less than a preset threshold, using the ratio as the probability of adding pre-labeled data; if the ratio is greater than the preset threshold, determining the probability of adding pre-labeled data based on the number of manually labeled training data frames, the ratio, and the number of pre-labeled data frames.

[0057] Specifically, the ratio between the number of manually labeled training data frames and the number of pre-labeled data frames is as follows:

[0058]

[0059] Where, r i This represents the ratio between the number of manually labeled training data frames and the number of pre-labeled data frames, where f represents the number of manually labeled training data frames, and s represents the ratio between the number of manually labeled training data frames and the number of pre-labeled data frames. i This indicates the number of pre-labeled data frames of type i.

[0060] Furthermore, the ratio between the number of manually labeled training data frames and the number of pre-labeled data frames is compared with a preset threshold, and the probability of adding pre-labeled data is determined based on the comparison result. The preset threshold is determined according to actual conditions; in this embodiment, a preset threshold of 1 is used as an example, but this is not intended to limit the invention.

[0061] For example, if the ratio is less than 1, it means the number of manually labeled training data frames is less than the number of pre-labeled data frames, and the ratio is used as the probability of adding pre-labeled data. For instance, assuming the number of first-class pre-labeled data frames is 20 and the number of manually labeled training data frames is 10, the ratio is less than 1, so the probability of adding is 1 / 2, meaning that 1 / 2 of the first-class pre-labeled data frames should be selected and added to the manually labeled training data frames. If the ratio is greater than 1, it means the number of manually labeled training data frames is greater than the number of pre-labeled data frames, and the probability of adding pre-labeled data will be further determined based on the number of manually labeled training data frames, the ratio, and the number of pre-labeled data frames.

[0062] In some embodiments, determining the probability of adding pre-labeled data based on the number of manually labeled training data frames, the ratio, and the number of pre-labeled data frames includes: obtaining the product between the number of pre-labeled data frames of type i and the ratio; obtaining the difference between the number of manually labeled training data frames and the product to determine the remainder of the ratio between the number of manually labeled training data frames and the number of pre-labeled data frames of type i; and determining the probability of adding pre-labeled data based on the ratio between the remainder and the number of pre-labeled data frames of type i.

[0063] Specifically, the probability of adding pre-labeled data can be determined using the following formula:

[0064] f = s i *r i +re i

[0065]

[0066] Where f represents the number of manually labeled training data frames, s i Represents the number of pre-labeled data frames of type i, re i t represents the remainder of the ratio of the number of manually labeled training data frames to the number of pre-labeled data frames of class i. ir represents the probability of adding pre-labeled data. i This represents the ratio between the number of manually labeled training data frames and the number of pre-labeled data frames.

[0067] For example, if the ratio between the number of manually labeled training data frames and the number of pre-labeled data frames is greater than 1, then the formula f = s i *r i +re i and formula t i =re i / s i Further determine the probability t of adding pre-labeled data i For example, assuming the number of pre-labeled data frames in the first category is 3, and the number of manually labeled training data frames is 4, the ratio of the two is greater than 1. Then, according to the formula f = s i *r i +re i The remainder re of the ratio of the number of manually labeled training data frames to the number of pre-labeled data frames of class i can be obtained. i The probability t of adding pre-labeled data is 1. i One-third. That is, if three Class I pre-labeled data frames are added to three manually labeled training data frames respectively, and there is one remaining manually labeled training data frame, then there is a one-third probability that these three Class I pre-labeled data frames will be added to the manually labeled training data frame.

[0068] As a specific example, the method for determining the probability of adding pre-labeled data in this application embodiment may include the following steps:

[0069] S201, Begin.

[0070] S202, the number of pre-labeled data frames can be represented as {s1, s2, ..., s...} i}, s i Let f represent the number of data in class i, and let f be the number of manually labeled training data frames.

[0071] S203, For data of type i, initialize i to 0.

[0072] S204, the ratio r between the number of manually labeled training data frames and the number of pre-labeled data frames. i =f / s i .

[0073] S205, Determine the ratio r i Is it less than a preset threshold? If yes, proceed to step S206; otherwise, proceed to step S207.

[0074] S206, the ratio r i As the probability of adding pre-labeled data.

[0075] S207, calculate f = s i *r i +re i , t i =re i / s i .

[0076] S208, for the current manually labeled test data frame, r should be added. i There are i types of pre-labeled data, with probability t. i Add an additional pre-labeled data for class i.

[0077] S209, i = i + 1.

[0078] S210, determine if i is less than n. If yes, execute S204; otherwise, execute S211.

[0079] S211, End.

[0080] In this way, designing different addition probabilities for pre-labeled data can ensure the uniform distribution of pre-labeled data to a certain extent, thereby alleviating the long-tail distribution phenomenon of pre-labeled data.

[0081] In some embodiments, obtaining the position information of a pre-labeled data frame added to a manually labeled training data frame includes: determining that the pre-labeled data frame is occluded by the manually labeled data frame based on the position information of the pre-labeled data frame and the attribute information of the manually labeled data frame; if the attribute information of the pre-labeled data frame determines that it is occluded, keeping the center point position information of the pre-labeled data frame unchanged; if the attribute information of the pre-labeled data frame determines that it is not occluded, moving the pre-labeled data frame.

[0082] In some embodiments, obtaining the position information of the pre-annotated data frame added to the manually annotated training data frame includes: determining that the current pre-annotated frame is not occluded by the manually annotated frame based on the position information of the pre-annotated data frame and the attribute information of the manually annotated target frame; keeping the position information of the pre-annotated data frame unchanged if it is determined that there is no occlusion based on the attribute information of the previous pre-annotated data frame; and moving the pre-annotated data frame if it is determined that there is occlusion based on the attribute information of the pre-annotated data frame.

[0083] Specifically, when adding a pre-labeled data frame to a manually labeled training data frame for training, the manually labeled training data frame establishes a 3D coordinate system with its origin O. The center point P of the pre-labeled data frame has coordinates (x, y, z). A ray OP is drawn from the origin O to the center point P of the pre-labeled data frame. Ray OP is used to determine whether the pre-labeled data frame is occluded by the manually labeled data frame. For example, if ray OP intersects with the manually labeled data frame, the pre-labeled data frame is determined to be occluded; if ray OP does not intersect with the manually labeled data frame, the pre-labeled data frame is determined not to be occluded. When adding the pre-labeled data frame to the manually labeled training data frame for training, it is also necessary to maintain the properties of the pre-labeled data frame as consistent as possible with the original training data frame. That is, it is also necessary to determine whether the occlusion state of the pre-labeled data frame after being added to the manually labeled training data frame is consistent with the occlusion state of the pre-labeled data frame in the original training data frame. If they are consistent, the center point position information of the pre-labeled data frame remains unchanged; if they are inconsistent, the pre-labeled data frame needs to be moved to keep its attributes consistent with the original training data frame.

[0084] For example, when the occlusion label Occlude of the pre-labeled data frame in the original training data frame is false, it indicates that the pre-labeled data frame is unoccluded in the original training data frame. In this case, if the ray OP intersects with the manually labeled data frame, the pre-labeled data frame needs to be moved until there is no intersection; if the ray OP does not intersect with the manually labeled data frame, the center point position information of the pre-labeled data frame remains unchanged. When the target occlusion label Occlude of the pre-labeled data frame in the original training data frame is true, it indicates that the pre-labeled data frame is occluded in the original training data frame. In this case, if the ray OP does not intersect with the manually labeled data frame, the pre-labeled data frame needs to be moved until there is an intersection; if the ray OP intersects with the manually labeled data frame, the center point position information of the pre-labeled data frame remains unchanged.

[0085] In some embodiments, moving a pre-annotated data frame includes: determining minimum and maximum position information based on the position and size information of each manually annotated data frame, wherein the minimum and maximum position information include values ​​in the x, y, and z directions; obtaining a first absolute value of the difference between the position information of the pre-annotated data frame and the minimum position information, and a second absolute value of the difference between the position information of the pre-annotated data frame and the maximum position information; and moving the pre-annotated data frame based on the first absolute value and the second absolute value of the difference, wherein the first absolute value and the second absolute value of the difference include values ​​in the x, y, and z directions.

[0086] Specifically, assuming the position information of the pre-labeled data frame is (x, y, z), the set of manually labeled data frames in a manually labeled training data frame is g = {m1, m2, ..., m...} k}, where the manually labeled data frame m k The position coordinates are (x k ,y k ,z k ), with dimensions (l k ,w k ,h k Furthermore, the positional information x in the x-direction of the manually labeled data frame should be satisfied. k The position information x in the x-direction of the pre-labeled data frame is smaller than the position information y in the y-direction of the manually labeled data frame. k The position information y in the y-direction of the pre-annotated data frame is smaller than the position information y in the y-direction. Based on the position and size information of each manually annotated data frame, the minimum position information (x) of the manually annotated data frame can be determined. min ,y min ,z min ) and maximum location information (x max ,y max ,z max ).

[0087] Obtain the position information (x, y, z) and minimum position information (x, z) of the pre-labeled data frame in the x, y, and z directions respectively. min ,y min ,z min The absolute value of the first difference between xx is obtained, i.e., abs(xx) min ), abs(yy) min ) and abs(zz) min Next, the position information (x, y, z) and maximum position information (x, z) of the pre-labeled data frame in the x, y, and z directions are obtained respectively. max ,y max ,z max The absolute value of the second difference between ) is obtained, i.e., the abs(xx) value is obtained. max ), abs(yy) max ) and abs(zz) max The pre-labeled data frame is moved by comparing the absolute values ​​of the first and second differences in each direction.

[0088] In some embodiments, moving the pre-labeled data frame according to the absolute value of a first difference and the absolute value of a second difference includes: moving the position information of the pre-labeled data frame to the maximum position information when the absolute value of the first difference is greater than the absolute value of the second difference; and moving the position information of the pre-labeled data frame to the minimum position information when the absolute value of the first difference is less than or equal to the absolute value of the second difference.

[0089] For example, if abs(xx) min ) greater than abs(xx max If so, the x-coordinate of the center point of the pre-labeled data frame will be designed as x max Conversely, if abs(xx) min ) less than or equal to abs(xx) max If so, the x-coordinate of the center point of the pre-labeled data frame will be designed as x min Similarly, the movement of the pre-labeled data frame in the y and z directions is the same as in the x direction, and will not be elaborated further here. The pre-labeled data frame is complete when movement is completed in all three directions (x, y, and z).

[0090] In some embodiments, determining the minimum and maximum position information based on the position and size information of each manually labeled data frame includes: obtaining the difference between the x-direction coordinate of all manually labeled data frames and half the length of the corresponding manually labeled data frame, forming a first x-direction position set for all manually labeled data frames, and using the minimum value in the first x-direction position set as the minimum position in the x-direction; obtaining the difference between the y-direction coordinate of all manually labeled data frames and half the width of the corresponding manually labeled data frame, forming a first y-direction position set for all manually labeled data frames, and using the minimum value in the first y-direction position set as the minimum position in the y-direction; obtaining the difference between the z-direction coordinate of all manually labeled data frames and half the height of the corresponding manually labeled data frame, forming a first z-direction position set for all manually labeled data frames, and using the minimum value in the first z-direction position set as the minimum position in the z-direction; obtaining the difference between the x ... length of the corresponding manually labeled data frame, forming a first z-direction position set for all manually labeled data frames; obtaining the difference between the x-direction coordinate of all manually labeled data frames and half the length of the corresponding manually labeled data frame, forming a first z-direction position set for all manually labeled data frames, and using the minimum value in the first z-direction position set as the minimum position in the z-direction; obtaining the difference between the x-direction coordinate of all manually labeled data frames and half the length of the corresponding The sum of the directional coordinates and half the length of the corresponding manually labeled data frame constitutes the second x-direction position set for all manually labeled data frames, and the maximum value in the second x-direction position set is taken as the maximum position in the x-direction. The sum of the y-direction coordinates and half the width of the corresponding manually labeled data frame constitutes the second y-direction position set for all manually labeled data frames, and the maximum value in the second y-direction position set is taken as the maximum position in the y-direction. The sum of the z-direction coordinates and half the height of the corresponding manually labeled data frame constitutes the second z-direction position set for all manually labeled data frames, and the maximum value in the second z-direction position set is taken as the maximum position in the z-direction. The minimum position information is determined based on the minimum positions in the x, y, and z directions, and the maximum position information is determined based on the maximum positions in the x, y, and z directions.

[0091] Specifically, the minimum and maximum position information can be determined using the following formula:

[0092] x min =min(x1-l1 / 2,x2-l2 / 2,...,x) k -l k / 2)

[0093] y min =min(y1-w1 / 2,y2-w2 / 2,...,y k -h k / 2)

[0094] z min =min(z1-h1 / 2,z2-h2 / 2,...,z) k -h k / 2)

[0095] x max =max(x1+l1 / 2,x2+l2 / 2,...,x k +l k / 2)

[0096] y max =max(y1+w1 / 2,y2+w2 / 2,...,y k +w k / 2)

[0097] z max =max(z1+h1 / 2,z2+h2 / 2,...,z) k +h k / 2)

[0098] Where, x min Represents the minimum position information in the x-direction, y min Represents the minimum position information in the y-direction, z min Represents the minimum position information in the z-direction, x max Represents the maximum position information in the x-direction, y max Represents the maximum position information in the y-direction, z max Represents the maximum position information in the z-direction, x k This represents the x-coordinate of the manually labeled data frame k, and the y-coordinate... k This represents the y-direction position information of the manually labeled data frame k, and the z-direction position information. k This represents the z-direction position information of the manually labeled data frame k, l k w represents the length of the manually labeled data frame k. k h represents the width of the manually labeled data frame k. k This indicates the height of the manually labeled data frame k.

[0099] For example, according to formula x min=min(x1-l1 / 2,x2-l2 / 2,...,x) k -l k / 2) Determine the minimum position information in the x-direction of the pre-labeled data frame, according to the formula y min =min(y1-w1 / 2,y2-w2 / 2,...,y k -h k / 2) Determine the minimum position information in the y-direction of the pre-labeled data frame, based on the formula z min =min(z1-h1 / 2,z2-h2 / 2,...,z) k -h k / 2) Determine the minimum position information in the z-direction of the pre-labeled data frame, according to the formula x max =max(x1+l1 / 2,x2+l2 / 2,...,x k +l k / 2) Determine the maximum position information in the x-direction of the pre-labeled data frame, according to the formula y max =max(y1+w1 / 2,y2+w2 / 2,...,y k +w k / 2) Determine the maximum position information in the y-direction of the pre-labeled data frame, based on the formula z max =max(z1+h1 / 2,z2+h2 / 2,...,z) k +h k / 2) Determine the maximum position information in the z-direction of the pre-labeled data frame.

[0100] As a specific example, the method for moving a pre-labeled data frame in this application embodiment may include the following steps:

[0101] S301, Begin.

[0102] S302, design ray OP, O is the origin, and P is the center point of the pre-labeled data frame.

[0103] S303, a manually labeled training data frame contains k manually labeled data frames. For the kth manually labeled data frame, its position is (x... k ,y k ,z k ), with dimensions (l k ,w k ,h k ).

[0104] S304, determine if the ray OP intersects with the manually labeled data frame. If yes, proceed to S305; otherwise, proceed to S307.

[0105] S305, determine whether the target occlusion label Occlude in the original training data frame of the pre-labeled data frame is true. If yes, proceed to S308; otherwise, proceed to S306.

[0106] S306, Design a set of manually labeled data frames g. For each manually labeled data frame, the following should be satisfied: x k Less than x, y k Less than y.

[0107] S307, determine whether the target occlusion label Occlude in the original training data frame of the pre-labeled data frame is true. If yes, execute S306; otherwise, execute S310.

[0108] S308, keep the center point position information of the pre-annotated data frame unchanged.

[0109] S309, Based on the position and size information of each manually labeled data frame, the minimum position information (x) of the manually labeled data frame can be determined. min ,y min ,z min ) and maximum location information (x max ,y max ,z max ).

[0110] S310, keep the center point position information of the pre-labeled data frame unchanged.

[0111] S311, obtain the absolute value of the first difference between the position information of the pre-labeled data frame and the minimum position information in the x, y, and z directions, and the absolute value of the second difference between the position information of the pre-labeled data frame and the maximum position information.

[0112] S312, determine whether the absolute value of the first difference is greater than the absolute value of the second difference. If yes, execute S313; otherwise, execute S314.

[0113] S313 moves the position information of the pre-labeled data frame to the maximum position information.

[0114] S314, move the position information of the pre-labeled data frame to the minimum position information.

[0115] S315: Determine if the x, y, and z positions have all been updated. If yes, proceed to S316; otherwise, proceed to S312.

[0116] S316, the pre-labeled data frame has been moved.

[0117] S317, End.

[0118] In this way, by adding the model's pre-labeled data to the manually labeled data with a certain probability, and keeping the attributes of the pre-labeled data frame consistent with the original training data frame, the impact of large model misses on training can be greatly reduced.

[0119] In some embodiments, determining the confusion matrix error rate based on the number of false detections, the preset recall target number of the current class pre-labeled data frame, and the number of false detections between different class pre-labeled data frames includes: determining the number of false detections between different class pre-labeled data frames based on the sum of the number of false detections and the number of noisy false detections; and determining the confusion matrix error rate based on the ratio between the number of false detections and the sum of the number of false detections between different class pre-labeled data frames and the preset recall target number of the current class pre-labeled data frame.

[0120] In some embodiments, determining the overall error rate based on the preset recall target number of different class pre-labeled data frames and the current class pre-labeled data frame includes: obtaining the sum of the number of false positives among different class pre-labeled data frames and the preset recall target number of the current class pre-labeled data frame; and determining the overall error rate based on the ratio between the number of false positives among different class pre-labeled data frames and the sum.

[0121] For example, the expression for the number of false positives between different class pre-labeled data frames is as follows:

[0122] FP i =FP confusion_matrix +FP noise

[0123] The formula for calculating the confusion matrix error rate is as follows:

[0124]

[0125] The formula for calculating the overall error rate is as follows:

[0126]

[0127] Among them, e i p represents the confusion matrix error rate. i FP represents the overall error rate. confusion_matrix TP represents the number of false positives. i This represents the preset recall target number of the current class pre-labeled data frame, FP. i FP represents the number of false detections between different classes of pre-labeled data frames. noise This indicates the number of false noise detections.

[0128] In some embodiments, the target attribute regression loss weight coefficient includes: the weight coefficient of the classification loss of the current pre-labeled data frame, the weight coefficient of the regression loss of the center point of the current pre-labeled data frame, the weight coefficient of the regression loss of the center point of the current pre-labeled data frame, and the weight coefficient of the regression loss of the size of the current pre-labeled data frame. Adjusting the target attribute regression loss weight coefficient based on the confusion matrix error rate and the overall error rate includes: obtaining the difference between a preset value and the confusion matrix error rate as a first probability, obtaining the difference between the preset value and the overall error rate as a second probability; multiplying the weight coefficient of the classification loss of the current pre-labeled data frame by the first probability to obtain its adjusted weight coefficient; and multiplying the weight coefficient of the regression loss of the center point of the current pre-labeled data frame by the first probability. The weight coefficient of the regression loss is multiplied by the first probability to obtain its adjusted weight coefficient; the weight coefficient of the loss value of the current pre-labeled data frame height is multiplied by the first probability and then by the second probability to obtain its adjusted weight coefficient; the weight coefficient of the regression loss of the current pre-labeled data frame size is multiplied by the first probability and then by the second probability to obtain its adjusted weight coefficient; the sum of the four factors—the classification loss of the current pre-labeled data frame multiplied by its adjusted weight coefficient, the regression loss of the current pre-labeled data frame center point multiplied by its adjusted weight coefficient, the loss value of the current pre-labeled data frame height multiplied by its adjusted weight coefficient, and the regression loss of the current pre-labeled data frame size multiplied by its adjusted weight coefficient—is used as the loss value of the labeled data frame.

[0129] Specifically, the target attribute regression loss weight coefficient can be determined using the following formula:

[0130] λ1=λ1*(1-p i )

[0131] λ2=λ2*(1-p i )

[0132] λ3=λ3*(1-p i )*(1-e i )

[0133] λ4=λ4*(1-p i )*(1-e i )

[0134] loss = λ1loss cls +λ2loss off +λ3loss z +λ4loss size

[0135] Where loss represents the loss value relative to the labeled data frame, e i p represents the confusion matrix error rate. iλ represents the overall error rate, and λ1 represents the adjusted classification loss of the current pre-labeled data frame. cls The weighting coefficients, where λ² represents the adjusted regression loss for the current pre-labeled data frame center point. off The weighting coefficient, λ3, represents the loss after adjusting the height of the current pre-labeled data frame. z The weighting coefficients, where λ4 represents the regression loss after adjusting for the current pre-labeled data frame size. size The weighting coefficients.

[0136] For example, using the overall error rate p i The formula for correcting the loss weight coefficient λ1 is λ1=λ1*(1-p i Using the overall error rate p i The formula for correcting the loss weight coefficient λ2 is λ2=λ2*(1-p i Using the overall error rate p i and confusion matrix error rate e i The formula for correcting the loss weight coefficient λ3 is λ3=λ3*(1-p i )*(1-e i Using the overall error rate p i and confusion matrix error rate e i The formula for correcting the loss weight coefficient λ4 is λ4=λ4*(1-p i )*(1-e i The expression for the loss value of the labeled data frame determined by each weight coefficient is loss = λ1loss cls +λ2loss off +λ3loss z +λ4loss size .

[0137] It should be noted that ei and pi are only used when calculating the attribute regression of pre-labeled bounding boxes in large models to reduce the impact of false detections in pre-labeled data of large models. However, they are not affected by these weights when performing bounding box regression and recognition of manually labeled data.

[0138] Therefore, redesigning the loss weights between target attributes based on the type of false detection can significantly reduce the impact of false detections in the model's pre-labeled data.

[0139] As a specific example, the method for determining the target attribute regression loss weight coefficient in this application embodiment may include the following steps:

[0140] S401, Begin.

[0141] S402, obtain the confusion matrix, TP, and FP of the model on the test set, and initialize i to 0 for target class i.

[0142] S403, Calculate the number of false positives (FP) where other types of pre-labeled data frames are identified as type i pre-labeled data frames. confusion_matrix_i .

[0143] S404, Calculate the TP of target i. i and FP i .

[0144] S405, Calculate the confusion matrix error rate e i and overall error rate p i .

[0145] S406, based on the confusion matrix error rate e i and overall error rate p i Adjust the weighting coefficients of the regression loss for the target attribute.

[0146] S407, i = i + 1.

[0147] S408, determine if i is less than n. If yes, execute S403; otherwise, execute S409.

[0148] S409, End.

[0149] In summary, the model training method of this invention adds pre-labeled model data to manually labeled data during the training process, reducing the impact of large model false negatives on training. Simultaneously, designing different addition probabilities for the pre-labeled data can ensure a uniform distribution of the pre-labeled data to a certain extent, thereby mitigating the long-tail distribution phenomenon of the pre-labeled data. Furthermore, redesigning the loss weights between target attributes based on the false detection type can significantly reduce the impact of false detections from the model's pre-labeled data.

[0150] Corresponding to the above embodiments, this application also proposes a model training device based on pre-labeled data.

[0151] Reference Figure 5 The model training device 500 based on pre-labeled data includes: an adding module 510, an acquisition module 520, a first determining module 530, a second determining module 540, and an adjustment module 550.

[0152] The addition module 510 is used to add pre-labeled data frames to manually labeled training data frames and then train the manually labeled training data frames after the addition. The addition of pre-labeled data frames to the manually labeled training data frames is based on the addition probability of the pre-labeled data and the position information of the pre-labeled data frames in the manually labeled training data frames. The acquisition module 520 is used to acquire the confusion matrix between different classes of pre-labeled data frames during training. The first determination module 530 is used to determine the number of false positives when other classes of pre-labeled data frames are identified as the current class of pre-labeled data frames based on the confusion matrix. The second determination module 540 is used to determine the confusion matrix error rate based on the number of false positives, the preset recall target number of the current class of pre-labeled data frames, and the number of false positives between different classes of pre-labeled data frames, and to determine the overall error rate based on the preset recall target numbers of different classes of pre-labeled data frames and the current class of pre-labeled data frames. The adjustment module 550 is used to adjust the target attribute regression loss weight coefficients according to the confusion matrix error rate and the overall error rate, so as to calculate the loss value of the labeled data frame based on the adjusted weight coefficients, and to perform backpropagation in the target detection model based on the loss value of the pre-labeled data frame, and update the weight parameters of the target detection model during the backpropagation process.

[0153] According to one embodiment of the present invention, the adding module 510 obtains the probability of adding pre-labeled data, specifically by obtaining the ratio between the number of manually labeled training data frames and the number of pre-labeled data frames; if the ratio is less than a preset threshold, the ratio is used as the probability of adding pre-labeled data; if the ratio is greater than the preset threshold, the probability of adding pre-labeled data is determined based on the number of manually labeled training data frames, the ratio, and the number of pre-labeled data frames.

[0154] According to one embodiment of the present invention, the adding module 510 is further configured to: obtain the product between the number of pre-labeled data frames of type i and the ratio; obtain the difference between the number of manually labeled training data frames and the product to determine the remainder of the ratio between the number of manually labeled training data frames and the number of pre-labeled data frames of type i; and determine the probability of adding pre-labeled data based on the ratio between the remainder and the number of pre-labeled data frames of type i.

[0155] According to one embodiment of the present invention, the adding module 510 obtains the position information of the pre-annotated data frame added to the manually annotated training data frame. Specifically, it is used to determine that the pre-annotated data frame is occluded by the manually annotated data frame based on the position information of the pre-annotated data frame and the attribute information of the manually annotated data frame; if it is determined that there is occlusion based on the attribute information of the pre-annotated data frame, the center point position information of the pre-annotated data frame remains unchanged; if it is determined that there is no occlusion based on the attribute information of the pre-annotated data frame, the pre-annotated data frame is moved.

[0156] According to one embodiment of the present invention, the adding module 510 is further configured to: determine that the current pre-annotation box is not occluded by the manual annotation box based on the position information of the pre-annotation data box and the attribute information of the manually annotated target box; keep the position information of the pre-annotation data box unchanged if it is determined that there is no occlusion based on the attribute information of the previous pre-annotation data box; and move the pre-annotation data box if it is determined that there is occlusion based on the attribute information of the pre-annotation data box.

[0157] According to an embodiment of the present invention, the adding module 510 is further configured to: determine minimum position information and maximum position information based on the position information and size information of each manually labeled data frame, wherein the minimum position information and maximum position information include values ​​in the x, y, and z directions; obtain a first absolute value of the difference between the position information of the pre-labeled data frame and the minimum position information and a second absolute value of the difference between the position information of the pre-labeled data frame and the maximum position information; and move the pre-labeled data frame according to the first absolute value of the difference and the second absolute value of the difference, wherein the first absolute value of the difference and the second absolute value of the difference include values ​​in the x, y, and z directions.

[0158] According to one embodiment of the present invention, the adding module 510 is further configured to, when the absolute value of the first difference is greater than the absolute value of the second difference, move the position information of the pre-labeled data frame to the maximum position information; and when the absolute value of the first difference is less than or equal to the absolute value of the second difference, move the position information of the pre-labeled data frame to the minimum position information.

[0159] According to one embodiment of the present invention, the adding module 510 is further configured to: obtain the difference between the x-direction coordinate of all manually labeled data frames and half the length of the corresponding manually labeled data frame, forming a first x-direction position set of all manually labeled data frames, and taking the minimum value in the first x-direction position set as the minimum position in the x-direction; obtain the difference between the y-direction coordinate of all manually labeled data frames and half the width of the corresponding manually labeled data frame, forming a first y-direction position set of all manually labeled data frames, and taking the minimum value in the first y-direction position set as the minimum position in the y-direction; obtain the difference between the z-direction coordinate of all manually labeled data frames and half the height of the corresponding manually labeled data frame, forming a first z-direction position set of all manually labeled data frames, and taking the minimum value in the first z-direction position set as the minimum position in the z-direction; obtain the difference between the x-direction coordinate of all manually labeled data frames and half the height of the corresponding manually labeled data frame, forming a first z-direction position set of all manually labeled data frames, and taking the minimum value in the first z-direction position set as the minimum position in the z-direction; obtain the difference between the x-direction coordinate of all manually labeled data frames and half the length of the corresponding manually labeled data frame; and obtain the difference between the x-direction coordinate of all manually labeled data frames and half the length of the corresponding manually labeled data frame. The sum of the values ​​between half the length of each manually labeled data frame and half the width of the corresponding data frame constitutes the second x-direction position set for all manually labeled data frames. The maximum value in the second x-direction position set is taken as the maximum position in the x-direction. The sum of the y-direction coordinates of each manually labeled data frame and half the width of the corresponding data frame constitutes the second y-direction position set for all manually labeled data frames. The maximum value in the second y-direction position set is taken as the maximum position in the y-direction. The sum of the z-direction coordinates of each manually labeled data frame and half the height of the corresponding data frame constitutes the second z-direction position set for all manually labeled data frames. The maximum value in the second z-direction position set is taken as the maximum position in the z-direction. The minimum position information is determined based on the minimum positions in the x, y, and z directions, and the maximum position information is determined based on the maximum positions in the x, y, and z directions.

[0160] According to one embodiment of the present invention, the second determining module 540 is specifically used to: determine the number of false detections between different classes of pre-labeled data frames based on the sum of the number of false detections and the number of noise false detections; and determine the confusion matrix error rate based on the ratio between the number of false detections and the sum of the number of false detections between different classes of pre-labeled data frames and the preset recall target number of the current class of pre-labeled data frame.

[0161] According to one embodiment of the present invention, the second determining module 540 is specifically used to obtain the sum of the number of false detections between different types of pre-labeled data frames and the preset recall target number of the current type of pre-labeled data frame; and to determine the overall error rate based on the ratio between the number of false detections between different types of pre-labeled data frames and the sum.

[0162] According to one embodiment of the present invention, the adjustment module 550 is specifically used to adjust the weight coefficients of the classification loss of the current pre-labeled data frame, the regression loss of the center point of the current pre-labeled data frame, the regression loss of the center point of the current pre-labeled data frame, and the regression loss of the size of the current pre-labeled data frame. The adjustment of the target attribute regression loss weight coefficients based on the confusion matrix error rate and the overall error rate includes: obtaining the difference between a preset value and the confusion matrix error rate as a first probability, obtaining the difference between the preset value and the overall error rate as a second probability; multiplying the weight coefficient of the classification loss of the current pre-labeled data frame by the first probability to obtain its adjusted weight coefficient; and adjusting the weight coefficient of the center point of the current pre-labeled data frame... The weighted coefficient of the regression loss is multiplied by the first probability to obtain its adjusted weighted coefficient; the weighted coefficient of the loss value of the current pre-labeled data frame height is multiplied by the first probability and then by the second probability to obtain its adjusted weighted coefficient; the weighted coefficient of the regression loss of the current pre-labeled data frame size is multiplied by the first probability and then by the second probability to obtain its adjusted weighted coefficient; the sum of the following four factors—the classification loss of the current pre-labeled data frame multiplied by its adjusted weighted coefficient, the regression loss of the current pre-labeled data frame center point multiplied by its adjusted weighted coefficient, the loss value of the current pre-labeled data frame height multiplied by its adjusted weighted coefficient, and the regression loss of the current pre-labeled data frame size multiplied by its adjusted weighted coefficient—is used as the loss value of the labeled data frame.

[0163] It should be noted that the above explanation of the embodiments and beneficial effects of the model training method based on pre-labeled data also applies to the model training device based on pre-labeled data in the embodiments of the present invention. To avoid redundancy, it will not be elaborated in detail here.

[0164] Corresponding to the above embodiments, this application also proposes a computer-readable storage medium.

[0165] The computer-readable storage medium of this application stores a model training program based on pre-labeled data, which, when executed by a processor, implements the aforementioned model training method based on pre-labeled data.

[0166] It should be noted that the above explanation of the embodiments and beneficial effects of the model training method based on pre-labeled data also applies to the computer-readable storage medium of the embodiments of the present invention. To avoid redundancy, it will not be elaborated in detail here.

[0167] Corresponding to the above embodiments, this application also proposes an electronic device 600.

[0168] Reference Figure 6The electronic device 600 of this application includes a memory 610, a processor 620, and a model training program based on pre-labeled data stored in the memory 610 and run on the processor 620. When the processor 620 executes the model training program based on pre-labeled data, it implements the aforementioned model training method based on pre-labeled data.

[0169] It should be noted that the above explanation of the embodiments and beneficial effects of the model training method based on pre-labeled data also applies to the electronic devices of the present invention. To avoid redundancy, they will not be elaborated in detail here.

[0170] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0171] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0172] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0173] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0174] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0175] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

[0176] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A model training method based on pre-labeled data, characterized in that, The method includes: When the pre-labeled data frame is added to the manually labeled training data frame, the manually labeled training data frame after the addition is trained, wherein the pre-labeled data frame is added to the manually labeled training data frame based on the addition probability of the pre-labeled data and the position information of the pre-labeled data frame added to the manually labeled training data frame; During training, the confusion matrix between different classes of pre-labeled data frames is obtained; The number of false positives identified as the current class pre-labeled data frame based on the confusion matrix is ​​determined. The confusion matrix error rate is determined based on the number of false detections, the preset recall target number of the current class pre-labeled data frame, and the number of false detections between different class pre-labeled data frames; and the overall error rate is determined based on the preset recall target number of the different class pre-labeled data frames and the current class pre-labeled data frame. The target attribute regression loss weight coefficients are adjusted according to the confusion matrix error rate and the overall error rate, so as to calculate the loss value of the labeled data frame based on the adjusted weight coefficients, and backpropagation is performed in the target detection model based on the loss value of the pre-labeled data frame, and the weight parameters of the target detection model are updated during the backpropagation process. Obtaining the probability of adding the pre-labeled data includes: Obtain the ratio between the number of manually labeled training data frames and the number of pre-labeled data frames; If the ratio is less than a preset threshold, the ratio is used as the probability of adding the pre-labeled data; If the ratio is greater than the preset threshold, the probability of adding the pre-labeled data is determined based on the number of manually labeled training data frames, the ratio, and the number of pre-labeled data frames. Obtaining the position information of the pre-annotated data frame added to the manually annotated training data frame includes: Based on the position information of the pre-annotated data frame and the attribute information of the manually annotated data frame, it is determined that the pre-annotated data frame is obscured by the manually annotated data frame; If it is determined that there is occlusion based on the attribute information of the pre-labeled data frame, the center point position information of the pre-labeled data frame shall remain unchanged; If it is determined from the attribute information of the pre-labeled data frame that there is no occlusion, the pre-labeled data frame is moved.

2. The model training method based on pre-labeled data according to claim 1, characterized in that, Determining the probability of adding the pre-labeled data based on the number of manually labeled training data frames, the ratio, and the number of pre-labeled data frames includes: Obtain the product between the number of the i-th type of pre-labeled data frames and the ratio; The difference between the number of manually annotated training data frames and the product is used to determine the remainder of the ratio of the number of manually annotated training data frames to the number of pre-annotated data frames of class i. The probability of adding the pre-labeled data is determined based on the ratio between the remainder and the number of the i-th type of pre-labeled data frames.

3. The model training method based on pre-labeled data according to claim 1, characterized in that, Obtaining the position information of the pre-annotated data frame added to the manually annotated training data frame includes: Based on the position information of the pre-annotated data frame and the attribute information of the manually annotated target frame, it is determined that the pre-annotated data frame is not obscured by the manually annotated frame; If it is determined from the attribute information of the pre-labeled data frame that there is no occlusion, the position information of the pre-labeled data frame remains unchanged; If it is determined that the pre-labeled data frame is occluded based on its attribute information, the pre-labeled data frame is moved.

4. The model training method based on pre-labeled data according to claim 1 or 3, characterized in that, Moving the pre-labeled data frame includes: The minimum and maximum position information are determined based on the position and size information of each manually labeled data frame, wherein the minimum and maximum position information include values ​​in the x, y, and z directions; Obtain the absolute value of the first difference between the position information of the pre-labeled data frame and the minimum position information, and the absolute value of the second difference between the position information of the pre-labeled data frame and the maximum position information; The pre-labeled data frame is moved based on the first absolute value of the difference and the second absolute value of the difference, wherein the first absolute value of the difference and the second absolute value of the difference include values ​​in the x, y, and z directions.

5. The model training method based on pre-labeled data according to claim 4, characterized in that, The step of moving the pre-labeled data frame based on the absolute value of the first difference and the absolute value of the second difference includes: If the absolute value of the first difference is greater than the absolute value of the second difference, the position information of the pre-labeled data frame is moved to the maximum position information; If the absolute value of the first difference is less than or equal to the absolute value of the second difference, the position information of the pre-labeled data frame is moved to the minimum position information.

6. The model training method based on pre-labeled data according to claim 4, characterized in that, Determine the minimum and maximum position information based on the position and size information of each manually labeled data frame, including: Obtain the difference between the x-direction coordinates of all the manually labeled data frames and half the length of the corresponding manually labeled data frame, and form a first x-direction position set of all the manually labeled data frames. Take the minimum value in the first x-direction position set as the minimum position in the x-direction. The difference between the y-direction coordinates of all the manually labeled data frames and half the width of the corresponding manually labeled data frame is obtained to form a first y-direction position set of all the manually labeled data frames. The minimum value in the first y-direction position set is taken as the minimum position in the y-direction. Obtain the difference between the z-direction coordinates of all the manually labeled data frames and half the height of the corresponding manually labeled data frame, and form a first z-direction position set of all the manually labeled data frames. Take the minimum value in the first z-direction position set as the minimum position in the z-direction. The sum of the x-coordinates of all the manually labeled data frames and half the length of the corresponding manually labeled data frame is obtained to form a second x-position set of all the manually labeled data frames. The maximum value in the second x-position set is taken as the maximum position in the x-direction. The sum of the y-coordinates of all the manually labeled data frames and half the width of the corresponding manually labeled data frame is obtained to form a second y-position set of all the manually labeled data frames. The maximum value in the second y-position set is taken as the maximum position in the y-direction. The sum of the z-coordinates of all the manually labeled data frames and half the height of the corresponding manually labeled data frame is obtained to form a second z-position set of all the manually labeled data frames. The maximum value in the second z-position set is taken as the maximum position in the z-direction. The minimum position information is determined based on the minimum position in the x direction, the minimum position in the y direction, and the minimum position in the z direction, and the maximum position information is determined based on the maximum position in the x direction, the maximum position in the y direction, and the maximum position in the z direction.

7. The model training method based on pre-labeled data according to claim 1, characterized in that, The confusion matrix error rate is determined based on the number of false detections, the preset recall target number of the current class pre-labeled data frame, and the number of false detections between different class pre-labeled data frames, including: The number of false detections among the different types of pre-labeled data frames is determined based on the sum of the number of false detections and the number of noise false detections; The confusion matrix error rate is determined based on the ratio between the number of false detections and the sum of the number of false detections between different class pre-labeled data frames and the preset recall target number of the current class pre-labeled data frame.

8. The model training method based on pre-labeled data according to claim 7, characterized in that, The overall error rate is determined based on the preset recall target number of the different class pre-labeled data frames and the current class pre-labeled data frame, including: Obtain the sum of the number of false positives among the different types of pre-labeled data frames and the preset recall target number of the current type of pre-labeled data frame; The overall error rate is determined based on the ratio between the number of false detections among the different types of pre-labeled data frames and the sum value.

9. The model training method based on pre-labeled data according to claim 7, characterized in that, The target attribute regression loss weight coefficients include: the weight coefficient of the classification loss of the current class pre-labeled data frame, the weight coefficient of the regression loss of the center point of the current class pre-labeled data frame, the weight coefficient of the loss value of the height of the current class pre-labeled data frame, and the weight coefficient of the regression loss of the size of the current class pre-labeled data frame. The adjustment of the target attribute regression loss weight coefficients based on the confusion matrix error rate and the overall error rate includes: The difference between the preset value and the error rate of the confusion matrix is ​​obtained as a first probability, and the difference between the preset value and the overall error rate is obtained as a second probability; Multiply the weight coefficient of the classification loss of the current class pre-labeled data frame by the first probability to obtain its adjusted weight coefficient; Multiply the weight coefficient of the regression loss of the center point of the current class pre-labeled data frame by the first probability to obtain its adjusted weight coefficient. The weight coefficient of the loss value of the height of the current class pre-labeled data frame is multiplied by the first probability and then by the second probability to obtain the adjusted weight coefficient. The weight coefficient of the regression loss of the current class pre-labeled data frame size is multiplied by the first probability and then by the second probability to obtain its adjusted weight coefficient. The loss value of the labeled data frame is the sum of the following four factors: the classification loss of the current class pre-labeled data frame multiplied by its adjusted weight coefficient, the regression loss of the center point of the current class pre-labeled data frame multiplied by its adjusted weight coefficient, the loss value of the height of the current class pre-labeled data frame multiplied by its adjusted weight coefficient, and the regression loss of the size of the current class pre-labeled data frame multiplied by its adjusted weight coefficient.

10. A model training apparatus based on pre-labeled data, implementing the model training method based on pre-labeled data according to any one of claims 1-9, characterized in that, The device includes: An addition module is used to train the manually labeled training data frame after adding the pre-labeled data frame to the manually labeled training data frame, wherein the pre-labeled data frame is added to the manually labeled training data frame based on the addition probability of the pre-labeled data and the position information of the pre-labeled data frame added to the manually labeled training data frame; The acquisition module is used to obtain the confusion matrix between different classes of pre-labeled data frames during training; The first determining module is used to determine the number of false detections of other class pre-labeled data frames being identified as the current class pre-labeled data frame based on the confusion matrix; The second determining module is used to determine the confusion matrix error rate based on the number of false detections, the preset recall target number of the current class pre-labeled data frame, and the number of false detections between different class pre-labeled data frames, and to determine the overall error rate based on the preset recall target number of the different class pre-labeled data frames and the current class pre-labeled data frame; The adjustment module is used to adjust the target attribute regression loss weight coefficients according to the confusion matrix error rate and the overall error rate, so as to calculate the loss value of the labeled data frame based on the adjusted weight coefficients, and to perform backpropagation in the target detection model based on the loss value of the pre-labeled data frame, and to update the weight parameters of the target detection model during the backpropagation process.

11. A computer-readable storage medium, characterized in that, It stores a model training program based on pre-labeled data, which, when executed by a processor, implements the model training method based on pre-labeled data according to any one of claims 1-9.

12. An electronic device, characterized in that, The method includes a memory, a processor, and a model training program based on pre-labeled data stored in the memory and executable on the processor. When the processor executes the model training program based on pre-labeled data, it implements the model training method based on pre-labeled data according to any one of claims 1-9.

Images