Sample generation methods, model training methods, object detection methods and devices

Abstract

This disclosure provides a sample generation method, a model training method, an object detection method, and an apparatus, relating to the field of artificial intelligence technology, particularly to image processing and deep learning. The specific implementation scheme is as follows: Multiple first deep learning models are used to process first image data to obtain multiple first output information for the first image data; wherein, the multiple first output information corresponds one-to-one with the multiple first deep learning models, and each first output information includes a first initial pseudo-label and a confidence level of the first initial pseudo-label; at least two of the multiple first output information are fused to obtain an intermediate pseudo-label and a confidence level of the intermediate pseudo-label; and based on the intermediate pseudo-label and the confidence level of the intermediate pseudo-label, first annotation information for the first image data is determined to obtain a first training sample; the first annotation information includes a first label and a confidence level of the first label.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of image processing and deep learning. More specifically, this disclosure provides a method for generating training samples, a method for training deep learning models, an object detection method, an apparatus, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] With the development of deep learning technology, the demand for training samples in deep learning models is increasing. Data can be manually labeled and cleaned to form a training set, which can then be used to train the deep learning model. However, manually labeling data is costly and inefficient. Summary of the Invention

[0003] This disclosure provides a method for generating training samples, a method for training deep learning models, an object detection method, an apparatus, an electronic device, a storage medium, and a computer program product.

[0004] According to one aspect of this disclosure, a method for generating training samples is provided, comprising: processing first image data using multiple first deep learning models respectively to obtain multiple first output information for the first image data; wherein the multiple first output information corresponds one-to-one with the multiple first deep learning models, and each first output information includes a first initial pseudo-label and a confidence level of the first initial pseudo-label; fusing at least two of the multiple first output information to obtain an intermediate pseudo-label and a confidence level of the intermediate pseudo-label; and determining first annotation information for the first image data based on the intermediate pseudo-label and the confidence level of the intermediate pseudo-label to obtain a first training sample; the first annotation information includes a first label and a confidence level of the first label.

[0005] According to one aspect of this disclosure, a method for training a deep learning model is provided, comprising: acquiring training samples; and training a second deep learning model using the training samples; wherein the training samples are obtained using the method described above for generating training samples.

[0006] According to one aspect of this disclosure, an object detection method is provided, comprising: acquiring third image data; inputting the third image data into a third deep learning model to obtain third output information of the third image data, the third output information including confidence level, and further including at least one of the following: detection bounding box and classification result of target object in the third image data; wherein the third deep learning model is trained using the training method of the aforementioned deep learning model.

[0007] According to another aspect of this disclosure, an apparatus for generating training samples is provided, comprising: a first information acquisition module, a fusion module, and a first determination module. The first information acquisition module is used to process first image data using multiple first deep learning models respectively, to obtain multiple first output information for the first image data; wherein the multiple first output information corresponds one-to-one with the multiple first deep learning models, and each first output information includes a first initial pseudo-label and a confidence level of the first initial pseudo-label. The fusion module is used to fuse at least two of the multiple first output information to obtain an intermediate pseudo-label and a confidence level of the intermediate pseudo-label. The first determination module is used to determine first annotation information for the first image data based on the intermediate pseudo-label and the confidence level of the intermediate pseudo-label, to obtain a first training sample; the first annotation information includes a first label and a confidence level of the first label.

[0008] According to another aspect of this disclosure, a training apparatus for a deep learning model is provided, comprising: a first acquisition module and a training module. The first acquisition module is used to acquire training samples, wherein the training samples are obtained using a device for generating training samples. The training module is used to train a second deep learning model using the training samples.

[0009] According to another aspect of this disclosure, an object detection apparatus is provided, comprising: a second acquisition module and a second information acquisition module. The second acquisition module is used to acquire third image data. The second information acquisition module is used to input the third image data into a third deep learning model to obtain third output information of the third image data. The third output information includes a confidence level and at least one of the following: a detection box and a classification result for a target object in the third image data. The third deep learning model is obtained using the training apparatus for the aforementioned deep learning model.

[0010] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods provided in this disclosure.

[0011] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods provided in this disclosure.

[0012] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods provided in this disclosure.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0015] Figure 1 This is a schematic diagram illustrating an application scenario of the method and apparatus for generating training samples according to embodiments of this disclosure.

[0016] Figure 2 This is a schematic flowchart of a method for generating a first training sample according to an embodiment of the present disclosure;

[0017] Figure 3 This is a schematic diagram illustrating the principle of a method for generating training samples according to an embodiment of the present disclosure;

[0018] Figure 4 This is a schematic flowchart of a method for generating a second training sample according to an embodiment of the present disclosure;

[0019] Figure 5 This is a schematic flowchart of a training method for a deep learning model according to an embodiment of the present disclosure;

[0020] Figure 6 This is a schematic flowchart of an object detection method according to an embodiment of the present disclosure;

[0021] Figure 7 This is a schematic structural block diagram of an apparatus for generating training samples according to an embodiment of the present disclosure;

[0022] Figure 8 This is a schematic structural block diagram of a training apparatus for a deep learning model according to an embodiment of the present disclosure;

[0023] Figure 9 This is a schematic structural block diagram of an object detection apparatus according to an embodiment of the present disclosure; and

[0024] Figure 10 This is a structural block diagram of an electronic device used to implement the sample generation method, model training method, and object detection method of the embodiments of this disclosure. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] Figure 1 This is a schematic diagram illustrating an application scenario of the method and apparatus for generating training samples according to embodiments of this disclosure.

[0027] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0028] like Figure 1 As shown, the application scenario 100 of this embodiment may include an electronic device 110 and a server 140.

[0029] The electronic device 110 can be any electronic device with processing capabilities, including but not limited to smartphones, tablets, laptops, desktop computers, and servers. The electronic device 110 can communicate with the server 140 via a network, which can include wireless or wired communication links.

[0030] Server 140 can pre-train multiple first deep learning models. Electronic device 110 can use the trained multiple first deep learning models 150 to process first image data 120, obtain first annotation information 130 for the first image data 120, and then determine training samples based on the first image data 120 and the first annotation information 130.

[0031] It should be noted that the methods for generating training samples and object detection provided in this disclosure can both be executed by electronic device 110 or server 140, and the training method for the deep learning model provided in this disclosure can be executed by server 140.

[0032] Accordingly, the apparatus for generating training samples and the object detection apparatus provided in this disclosure can be located in the electronic device 110 or the server 140, and the training apparatus for the deep learning model provided in this disclosure can be located in the server 140.

[0033] It should be understood that Figure 1 The number and type of electronic devices and servers shown are merely illustrative. Depending on implementation needs, there can be any number and type of electronic devices, servers, and databases.

[0034] Figure 2 This is a schematic flowchart of a method for generating a first training sample according to an embodiment of the present disclosure.

[0035] like Figure 2 As shown, the method 200 for generating training samples may include operations S210 to S240.

[0036] In operation S210, multiple first deep learning models are used to process the first image data respectively to obtain multiple first output information for the first image data; wherein, the multiple first output information corresponds one-to-one with the multiple first deep learning models, and each first output information includes a first initial pseudo-label and the confidence level of the first initial pseudo-label.

[0037] For example, multiple first deep learning models are pre-trained and can be used for tasks such as image data classification and object detection. These first deep learning models can employ CNNs (Convolutional Neural Networks), Transformers, etc., and this embodiment does not limit the structure of the first deep learning models. Furthermore, it is understood that the multiple first deep learning models differ from each other; for example, their structures and parameters may vary.

[0038] For example, the number of the first image can be 1 or N, where N is an integer greater than or equal to 2. For example, the first image data can be labeled or unlabeled.

[0039] For example, the first image data can be input into a first deep learning model, which outputs first output information. The first initial pseudo-label in the first output information may include information such as the location information and category of the detection box.

[0040] In operation S220, at least two of the multiple first output information are fused to obtain the intermediate pseudo-label and the confidence level of the intermediate pseudo-label.

[0041] For example, the intermediate pseudo-label and the confidence level of the intermediate pseudo-label can be determined based on at least one of a plurality of first output information.

[0042] For example, the first initial pseudo-label corresponding to the highest confidence level among multiple first output information can be determined as the intermediate pseudo-label, and the highest confidence level can be determined as the confidence level of the intermediate pseudo-label.

[0043] For example, the confidence scores of multiple first output information can be sorted from highest to lowest. A pre-set value for k, where k is an integer greater than or equal to 2, can be used. If k is less than or equal to the number of first output information, the top k first output information can be selected and fused. If k is greater than the number of first output information, all first output information can be fused. This confidence-based sorting allows for the selection of the more reliable first output information evaluated by the first deep learning model, thereby improving the accuracy of intermediate pseudo-labels and confidence scores.

[0044] For example, the fusion process may include: for at least two selected first output information, the average of the first initial pseudo-labels can be determined as the intermediate pseudo-label, and the average of the confidence scores can be determined as the confidence score of the intermediate pseudo-label.

[0045] In operation S230, based on the intermediate pseudo-label and the confidence level of the intermediate pseudo-label, the first annotation information for the first image data is determined, and the first training sample is obtained; the first annotation information includes the first label and the confidence level of the first label.

[0046] For example, the confidence levels of intermediate pseudo-labels and intermediate pseudo-labels can be determined as the confidence levels of the first label and the first label, respectively.

[0047] This embodiment of the disclosure can utilize multiple first deep learning models to obtain multiple first output information for the same first image data. Then, based on these first output information, the annotation information of the first image data is determined. Combining the first image data and the annotation information yields the first training sample. It can be seen that since the above process does not require manual annotation of the training samples, it reduces labor costs and improves annotation efficiency.

[0048] Furthermore, since the fusion process is based on at least two of the multiple first output information rather than on a single first output information to determine the fused intermediate pseudo-label, the intermediate pseudo-label can reflect the evaluation results of at least two first deep learning systems on the first image data, thus alleviating the problem of incorrect first annotation information caused by inaccurate first output information from a single first deep learning system.

[0049] Figure 3 This is a schematic diagram illustrating the method for generating training samples according to embodiments of the present disclosure.

[0050] Taking the first deep learning model for object detection as an example, the method for generating training samples in this embodiment may include a preprocessing stage and a fusion stage, and may also include a data cleaning stage and a sample augmentation stage.

[0051] In the preprocessing stage, multiple first deep learning models 302 can be used to process the first image data 301 respectively to obtain multiple first output information 303 for the first image data 301. Each first output information 303 includes a first initial pseudo-label and the confidence level of the first initial pseudo-label.

[0052] For example, multiple first deep learning models 302 are used to process the current input first image data 301. For the same target object in the first image data, multiple first deep learning models 302 can output different detection boxes, classification results corresponding to the detection boxes, and confidence scores. The confidence score can characterize the probability that the target object in the detection box belongs to a specific category.

[0053] During the fusion phase, at least two of the multiple first output information 303 can be fused to obtain the intermediate pseudo-label 304 and the confidence level 305 of the intermediate pseudo-label 304. The specific process of fusion processing will be described below and will not be repeated here.

[0054] Next, after the fusion stage is completed, different processing methods can be adopted based on the confidence level 307 of whether the first image data 301 corresponds to the original label 306.

[0055] For example, if the first image data is unlabeled data, the data cleaning stage can be omitted, and the intermediate pseudo-labels and their confidence levels obtained in the fusion stage can be directly determined as the first label and its confidence level, respectively, thus obtaining the first training sample.

[0056] For example, if the first image data 301 is labeled data, it can proceed to the data cleaning stage. The data cleaning stage can further determine the first annotation information 308 based on the intermediate pseudo-label 304, the confidence level 305 of the intermediate pseudo-label 304, the original label 306, and the confidence level 307 of the original label 306. This allows the first image data 301 and the first annotation information 308 to be combined into the first training sample. The processing procedure of the data cleaning stage will be explained below and will not be repeated here.

[0057] As can be seen, the above method can be used to obtain the first annotation information for the first image data, regardless of whether the first image data is unlabeled or labeled.

[0058] After obtaining the initial annotation information, the data distribution of the initial annotation information can be used to determine whether to proceed to the sample expansion stage. The processing procedure for the sample expansion stage will be described below and will not be repeated here.

[0059] The following explains the fusion processing operations involved in the fusion stage described above.

[0060] In some embodiments, the fusion process described above may include: determining the position information of the detection boxes in the intermediate pseudo-label based on the position and size information of each of the multiple detection boxes contained in the first initial pseudo-label of at least two first output information. For example, the position information of the detection boxes in the fused information may be determined by weighted averaging the target point coordinates, length, and width of the multiple detection boxes, where the target point may be a center point, vertex, etc. For example, the weights of the multiple detection boxes may be the same, or the weights may be pre-configured for the multiple first deep learning models and used as the weights of the multiple detection boxes during the fusion process.

[0061] In some embodiments, the fusion process described above may include: determining the categories in the intermediate pseudo-labels based on the multiple categories contained in the first initial pseudo-labels in at least two first output information. For example, the category with the highest frequency or the category with the highest confidence level in the at least two first output information may be determined as the category in the intermediate pseudo-labels.

[0062] In some embodiments, the fusion process described above may include determining the confidence level of the intermediate pseudo-label based on a weighted average of multiple confidence levels from at least two first output information sources. For example, the weights of the multiple confidence levels may be the same, or weights may be pre-configured for multiple first deep learning models and used as the weights of the multiple confidence levels during the fusion process.

[0063] The above fusion process can process each piece of information in the first output information and accurately measure the confidence level of the fused intermediate pseudo-labels.

[0064] The following describes the processing operations involved in the data cleaning stage described above.

[0065] In this embodiment, the original label of the first image data is referred to as the original label. The original label can be a manually labeled label or a label labeled by other deep learning models. The accuracy of the other deep learning models can be lower than that of the multiple first deep learning models mentioned above. In addition, the original label corresponds to a confidence level, which can be 0 or 1, or other values ​​between 0 and 1.

[0066] For example, the confidence level of the first label can be determined based on the confidence levels of the intermediate pseudo-labels and the original label. Alternatively, the weighted average of the confidence levels of the intermediate pseudo-labels and the original label can be used as the confidence level in the first annotation information, where the weights of the intermediate pseudo-labels and the original label can be equal. This method can accurately determine the confidence level of the first label.

[0067] For example, the first tag can be determined based on the intermediate pseudo-tag and the original tag.

[0068] In one embodiment for determining the first label, if an original label and an intermediate pseudo-label exist, and the original label and the intermediate pseudo-label are identical, then the weighted average of the detection boxes in the original label and the detection boxes in the intermediate pseudo-label can be determined as the detection boxes in the first label, wherein the weights of the intermediate pseudo-label and the original label can be equal. Alternatively, the detection boxes in the original label or the detection boxes in the intermediate pseudo-label can be determined as the detection boxes in the first label.

[0069] In another embodiment for determining the first label, if there is an original label and an intermediate pseudo-label, and the original label and the intermediate pseudo-label are inconsistent, it indicates that the original label has been mislabeled, and the intermediate pseudo-label can be determined as the first label.

[0070] In another embodiment for determining the first label, if an intermediate pseudo-label exists and the original label is missing, it indicates that the original label has been omitted, and the intermediate pseudo-label can be determined as the first label.

[0071] In another embodiment of determining the first label, if an original label exists and an intermediate pseudo label is missing, it indicates that the original label has been mislabeled. In this case, the original label can be deleted, and the first label obtained based on the original label is determined to be empty. That is, it is not necessary to determine the first label based on the original label.

[0072] In this embodiment, during the data cleaning stage, multiple first deep learning models can be used to obtain intermediate pseudo-labels to perform secondary cleaning of the mislabeled original labels in the first image data. Furthermore, missing original labels in the first image data can be supplemented, thereby improving the richness of the sample data.

[0073] In other embodiments, the first deep learning model can also be used for classification tasks. It should be noted that if the first deep learning model is used to process an object detection task, the first output information 'a' obtained after processing the first image data includes a detection box, a classification result corresponding to the detection box, and a confidence score. The confidence score can characterize the probability that the target object in the detection box belongs to a specific classification result. If the first deep learning model is used to process a classification task, the first output information 'b' obtained after processing the first image data includes a classification result and a confidence score. The confidence score can characterize the probability that the target object in the detection box belongs to a specific classification result.

[0074] As can be seen, the first output information a is similar to the first output information b, the difference being that the first output information b has fewer detection boxes compared to the first output information a. Therefore, the method for determining the first annotation information based on the first output information b can refer to the method for determining the first annotation information based on the first output information a, and will not be elaborated further in this embodiment.

[0075] The above text details the method for determining the first annotation information of the first image data. After obtaining the first annotation information, the data distribution of the first annotation information can be used to determine whether to enter the sample augmentation stage to augment the first training samples. The following section combines... Figure 4 The method of sample expansion will be explained.

[0076] Figure 4 This is a schematic flowchart of a method for generating a second training sample according to an embodiment of the present disclosure.

[0077] like Figure 4 As shown, after obtaining the first annotation information, the method 400 for generating training samples may further include operations S440 to S470.

[0078] In operation S440, it is determined whether the data distribution of the N first annotation information for the N first images is consistent with the predetermined data distribution. If yes, operation S450 can proceed; otherwise, the process can end.

[0079] For example, the predetermined data distribution can characterize the uneven distribution of the first annotation information, such as uneven distribution of detection box size, uneven distribution of categories, uneven distribution of confidence scores, etc.

[0080] For example, if the first label includes a detection box and the second label includes a detection box, the predetermined data distribution may include a predetermined distribution of detection box sizes, which can characterize the uneven distribution of detection box sizes in the first annotation information. For example, there may be far more first training samples with large detection boxes than with small detection boxes.

[0081] For example, the first label includes a category, the second label includes a category, and correspondingly, the predetermined data distribution may include a predetermined category distribution, which can characterize the imbalance in the category distribution in the first annotation information. For example, there are far more first training samples of the first category than the first training samples of the second category.

[0082] For example, the first label includes confidence level, the second label includes confidence level, and correspondingly, the predetermined data distribution may include a predetermined confidence level distribution, which can characterize the imbalance in the confidence level distribution in the first labeled information. For example, there are far more first training samples with a confidence level of 1 than first training samples with a confidence level of 0.7.

[0083] For example, evaluation metrics can be determined for the first labeled information. Evaluation metrics can assess the balance of the first labeled information. Evaluation metrics may include precision, recall, F1 score, AUC value, etc.

[0084] Next, if the value of the evaluation index exceeds the predetermined range, it can be determined that there is an imbalance in the data distribution of the N first annotation information, that is, it can be determined that the data distribution of the N first annotation information is consistent with the predetermined data distribution.

[0085] For example, the predetermined data distribution can follow a normal distribution, a uniform distribution, etc.

[0086] In operation S450, second annotation information is determined for each of the M second image data, where M is an integer greater than or equal to 2, and the second annotation information may include at least one of the second label and the confidence level of the second label.

[0087] For example, the second image data can be unlabeled data, and the preprocessing and fusion processes mentioned above can be used to process the second image data to obtain the second annotation information.

[0088] For example, for each of the M second image data sets, multiple first deep learning models are used to process the second image data, resulting in multiple second output information sets. Each second output information set corresponds one-to-one with one of the multiple first deep learning models, and each second output information set includes a second initial pseudo-label and its confidence level. Then, at least two of the multiple second output information sets are fused to obtain the second label and its confidence level. The processing procedure is described above and will not be repeated here.

[0089] This embodiment processes the second image data through multiple first deep learning models to obtain second output information, and then determines the second label based on the multiple second output information, which can reduce the cost of manually annotating data.

[0090] In other embodiments, the second annotation information of the first image data can also be determined by manual annotation or other methods. In this embodiment, the method of determining the second annotation information is not limited.

[0091] In operation S460, at least one target image data is determined from the M second image data based on the second annotation information, data distribution, and predetermined data distribution of each of the M second image data.

[0092] For example, the target annotation information and the target quantity can be determined based on the deviation between the data distribution of N first annotation information and the predetermined data distribution. Then, from M second image data, the target quantity of second image data whose second annotation information is consistent with the target annotation information can be selected as at least one target image data.

[0093] For example, the target annotation information includes at least one of the following: the detection box in the second label, the category in the second label, and the confidence level of the second label. The detection box can be divided into large-size detection boxes and small-size detection boxes according to its size. The category can be specifically divided into the first category, the second category, the third category, etc., and the confidence level can be divided into multiple confidence level intervals.

[0094] For example, the data distribution of N first-level annotation information indicates that the ratio of training samples of the first, second, and third categories is 1:100:100.2, while the ratio of training samples of the first, second, and third categories in the predetermined data distribution is 93:100:100.2. This shows that the target annotation information is of the first category, meaning that the training samples that need to be supplemented are of the first category. Furthermore, the number Num to be supplemented can be determined. Then, based on this number, target image data is selected from M second-level image data. The selected target image data belongs to the first category, and the number of selected target image data is equal to or close to the aforementioned number Num.

[0095] This embodiment can accurately determine the required number of targets and target annotation information based on the deviation between the data distribution of N first annotation information and the predetermined data distribution, and then accurately determine the second training samples that need to be supplemented, so as to obtain a training set with high data distribution balance.

[0096] In other embodiments, some second image data can be selected from the M second image data and added to the training set to supplement the first training samples by random sampling, manual selection, or other methods.

[0097] In operation S470, a second training sample is generated based on at least one target image data and the second annotation information of each of the at least one target image data.

[0098] The second training sample is used to supplement the first training sample. After obtaining the second training sample, both the second and first training samples can be added to the training set, and the training set can be used to train the second deep learning model to be trained.

[0099] In this embodiment, when the data distribution of the first training sample is unbalanced, a second training sample is generated to supplement the first training sample in a targeted manner based on the data distribution of the first training sample. For example, in a certain classification scenario, the number of training samples in the first category is much greater than the number of training samples in the second category. Multiple first deep learning models can be used to determine the second label for the unlabeled data. Then, based on the second label, second image data of the second category is selected from multiple second image data sets, thereby using the selected second image data to expand the training set. For example, in an object detection scenario, the number of training samples for small objects is much greater than the number of training samples for large objects, so the training samples for large objects can be supplemented.

[0100] As can be seen, the above method can make the overall training sample data distribution more balanced, improve the effectiveness of the training samples, thereby improving the training effect of the second deep learning model and its performance in business scenarios.

[0101] Figure 5 This is a schematic flowchart of a training method for a deep learning model according to an embodiment of the present disclosure.

[0102] like Figure 5 As shown, the training method 500 of the deep learning model may include operations S510 to S520.

[0103] In operation S510, training samples are obtained. For example, the training samples are obtained using the method described above for generating training samples.

[0104] When operating the S520, a second deep learning model is trained using training samples.

[0105] Figure 6 This is a schematic flowchart of an object detection method according to an embodiment of the present disclosure.

[0106] like Figure 6 As shown, the object detection method 600 may include operations S610 to S620.

[0107] During operation of S610, third image data is acquired;

[0108] In operation S620, the third image data is input into the third deep learning model to obtain the third output information of the third image data, which includes the confidence score. Furthermore, if the third deep learning model is used for image classification, the third output information may also include the classification result. If the third deep learning model is used for object detection, the third output information may also include the detection bounding box and classification result for the target object in the third image data. For example, the above-mentioned third deep learning model is trained using the training method described above for deep learning models.

[0109] Figure 7 This is a schematic structural block diagram of an apparatus for generating training samples according to an embodiment of the present disclosure.

[0110] like Figure 7 As shown, the apparatus 700 for generating training samples may include: a first information acquisition module 710, a fusion module 720, and a first determination module 730.

[0111] The first information acquisition module 710 is used to process the first image data using multiple first deep learning models respectively to obtain multiple first output information for the first image data; wherein, the multiple first output information corresponds one-to-one with the multiple first deep learning models, and each first output information includes a first initial pseudo-label and the confidence level of the first initial pseudo-label.

[0112] The fusion module 720 is used to fuse at least two of the multiple first output information to obtain the intermediate pseudo-label and the confidence level of the intermediate pseudo-label.

[0113] The first determining module 730 is used to determine the first annotation information for the first image data based on the intermediate pseudo-label and the confidence level of the intermediate pseudo-label, and obtain the first training sample; the first annotation information includes the first label and the confidence level of the first label.

[0114] According to another embodiment of this disclosure, the fusion module includes at least one of the following: a first determining submodule, a second determining submodule, and a third determining submodule. The first determining submodule is used to determine the position information of the detection boxes in the intermediate pseudo-label based on the position and size information of each of the detection boxes, given multiple detection boxes contained in the first initial pseudo-label in at least two first output information sets. The second determining submodule is used to determine the category in the intermediate pseudo-label based on multiple categories contained in the first initial pseudo-label in at least two first output information sets. The third determining submodule is used to determine the confidence level of the intermediate pseudo-label based on a weighted average of multiple confidence levels in at least two first output information sets.

[0115] According to another embodiment of this disclosure, the first determining module includes a fourth determining submodule and a fifth determining submodule. The fourth determining submodule is configured to, in response to detecting that the first image corresponds to an original label and a confidence level of the original label, determine a first label based on an intermediate pseudo-label and the original label; and determine a confidence level of the first label based on the confidence level of the intermediate pseudo-label and the confidence level of the original label. The fifth determining submodule is configured to, in response to detecting that the first image lacks an original label, determine the intermediate pseudo-label and its confidence level as the first label and its confidence level, respectively.

[0116] According to another embodiment of this disclosure, the fourth determining submodule includes: a first determining unit, a second determining unit, a third determining unit, and a fourth determining unit. The first determining unit is configured to, in response to detecting the existence of an original label and an intermediate pseudo-label, and that the original label and the intermediate pseudo-label are identical, determine one of the following as the first label: a weighted average of the original label and the intermediate pseudo-label, or the original label and the intermediate pseudo-label. The second determining unit is configured to, in response to detecting the existence of an original label and an intermediate pseudo-label, and that the original label and the intermediate pseudo-label are inconsistent, determine the intermediate pseudo-label as the first label. The third determining unit is configured to, in response to detecting the existence of an intermediate pseudo-label and that the original label is missing, determine the intermediate pseudo-label as the first label. The fourth determining unit is configured to, in response to detecting the existence of an original label and that the intermediate pseudo-label is missing, determine that the first label is empty.

[0117] According to another embodiment of this disclosure, the above-mentioned apparatus further includes: a second determining module, configured to determine at least two first output information from the plurality of first output information according to the order of multiple confidence levels among the plurality of first output information.

[0118] According to another embodiment of this disclosure, the number of first images is N, where N is an integer greater than or equal to 2. The apparatus further includes a third determining module, a fourth determining module, and a generating module. The third determining module is configured to determine second annotation information for each of M second image data in response to detecting that the data distribution of N first annotation information for the N first images is consistent with a predetermined data distribution. The second annotation information includes at least one of the following: a second label and a confidence level of the second label, where M is an integer greater than or equal to 2. The fourth determining module is configured to determine at least one target image data from the M second image data based on the second annotation information of each of the M second image data, the data distribution, and the predetermined data distribution. The generating module is configured to generate a second training sample based on the at least one target image data and the second annotation information of each of the at least one target image data.

[0119] According to another embodiment of this disclosure, the fourth determining module includes a sixth determining submodule and a selecting submodule. The sixth determining submodule is used to determine the target annotation information and the target quantity based on the deviation between the data distribution of N first annotation information and a predetermined data distribution. The selecting submodule is used to select, from M second image data, a target quantity of second image data whose second annotation information matches the target annotation information, as at least one target image data.

[0120] According to another embodiment of this disclosure, the third determining module includes an obtaining submodule and a fusing submodule. The obtaining submodule is used to process each of the M second image data using multiple first deep learning models to obtain multiple second output information for each second image data. Each of the multiple second output information corresponds one-to-one with one of the multiple first deep learning models, and each second output information includes a second initial pseudo-label and a confidence level of the second initial pseudo-label. The fusing submodule is used to fuse at least two of the multiple second output information to obtain a second label and a confidence level of the second label.

[0121] According to another embodiment of this disclosure, the first label, the second label, and the predetermined data distribution satisfy at least one of the following: the first label includes a detection box, the second label includes a detection box, and the predetermined data distribution includes a predetermined distribution of detection box sizes. The first label includes a category, the second label includes a category, and the predetermined data distribution includes a predetermined category distribution. The first label includes a confidence level, the second label includes a confidence level, and the predetermined data distribution includes a predetermined confidence level distribution.

[0122] Figure 8 This is a schematic structural block diagram of a training apparatus for a deep learning model according to an embodiment of the present disclosure.

[0123] like Figure 8 As shown, the training device 800 for the deep learning model may include a first acquisition module 810 and a training module 820.

[0124] The first acquisition module 810 is used to acquire training samples. For example, the training samples are obtained using a device for generating training samples.

[0125] Training module 820 is used to train a second deep learning model using training samples.

[0126] Figure 9 This is a schematic structural block diagram of an object detection device according to an embodiment of the present disclosure.

[0127] like Figure 9 As shown, the object detection device 900 may include a second acquisition module 910 and a second information acquisition module 920.

[0128] The second acquisition module 910 is used to acquire the third image data.

[0129] The second information acquisition module 920 is used to input the third image data into the third deep learning model to obtain the third output information of the third image data. The third output information includes confidence level, and also includes at least one of the following: detection boxes and classification results for target objects in the third image data. The third deep learning model is obtained using the training device of the aforementioned deep learning model.

[0130] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0131] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0132] According to embodiments of this disclosure, this disclosure also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform at least one of the above-described methods for generating training samples, training deep learning models, and object detection methods.

[0133] According to embodiments of this disclosure, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute at least one of the above-described methods for generating training samples, training deep learning models, and object detection methods.

[0134] According to embodiments of this disclosure, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements at least one of the above-described methods for generating training samples, training a deep learning model, and object detection.

[0135] Figure 10 This is a structural block diagram of an electronic device used to implement the sample generation method, model training method, and object detection method of the embodiments of this disclosure.

[0136] Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0137] like Figure 10As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0138] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0139] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as at least one of the methods for generating training samples, training deep learning models, and object detection methods described above. For example, in some embodiments, at least one of the methods for generating training samples, training deep learning models, and object detection methods described above can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by computing unit 1001, it can perform one or more steps of at least one of the methods described above for generating training samples, training deep learning models, and object detection. Alternatively, in other embodiments, computing unit 1001 can be configured by any other suitable means (e.g., by means of firmware) to execute at least one of the methods described above for generating training samples, training deep learning models, and object detection.

[0140] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0144] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0145] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0146] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0147] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating training samples, comprising: Multiple first deep learning models are used to process the first image data to obtain multiple first output information for the first image data; wherein, the multiple first output information corresponds one-to-one with the multiple first deep learning models, and each first output information includes a first initial pseudo-label and the confidence level of the first initial pseudo-label; At least two of the plurality of first output information are fused to obtain an intermediate pseudo-label and the confidence level of the intermediate pseudo-label; and Based on the intermediate pseudo-labels and their confidence levels, first annotation information for the first image data is determined to obtain the first training sample; the first annotation information includes a first label and its confidence level. The number of the first image data is N, where N is an integer greater than or equal to 2; The method further includes: In response to detecting that the data distribution of N first annotation information for N first image data is consistent with a predetermined data distribution, second annotation information for each of M second image data is determined; wherein, the second annotation information includes at least one of the following: a second label and the confidence level of the second label, and M is an integer greater than or equal to 2; Based on the deviation between the data distribution of the N first annotation information and the predetermined data distribution, the target annotation information and the number of targets are determined; From the M second image data, select a target number of second image data whose second annotation information is consistent with the target annotation information, and use them as at least one target image data; and A second training sample is generated based on the at least one target image data and the second annotation information of each of the at least one target image data, so that the second training sample supplements the first training sample to obtain a training set whose data distribution satisfies the predetermined data distribution, wherein the predetermined data distribution follows a normal distribution or a uniform distribution.

2. The method according to claim 1, wherein, The process of fusing at least two of the plurality of first output information to obtain an intermediate pseudo-label and the confidence level of the intermediate pseudo-label includes at least one of the following: For the multiple detection boxes contained in the first initial pseudo-label in the at least two first output information, the position information of the detection box in the intermediate pseudo-label is determined based on the position information and size information of each of the multiple detection boxes; Based on the multiple categories contained in the first initial pseudo-label in the at least two first output information, determine the category in the intermediate pseudo-label; as well as The confidence level of the intermediate pseudo-label is determined based on the weighted average of multiple confidence levels in the at least two first output information.

3. The method according to claim 1, wherein, The step of determining the first annotation information for the first image data based on the intermediate pseudo-label and the confidence level of the intermediate pseudo-label includes: In response to detecting that the first image data corresponds to an original label and the confidence level of the original label, the first label is determined based on the intermediate pseudo label and the original label; The confidence level of the first label is determined based on the confidence levels of the intermediate pseudo-label and the original label; and In response to the detection that the first image data lacks an original label, the confidence level of the intermediate pseudo-label and the intermediate pseudo-label are respectively determined as the confidence level of the first label and the first label.

4. The method according to claim 3, wherein, The step of determining the first tag based on the intermediate pseudo tag and the original tag includes: In response to the detection of the existence of the original label and the intermediate pseudo label, and the original label and the intermediate pseudo label being consistent, one of the following is determined as the first label: the weighted average of the original label and the intermediate pseudo label, or the original label and the intermediate pseudo label; In response to the detection that the original label and the intermediate pseudo label exist and that the original label and the intermediate pseudo label are inconsistent, the intermediate pseudo label is determined to be the first label; In response to the detection of the presence of the intermediate pseudo-tag and the absence of the original tag, the intermediate pseudo-tag is identified as the first tag; and In response to the detection of the presence of the original label and the absence of the intermediate pseudo-label, it is determined that the first label is empty.

5. The method according to claim 1, further comprising: Based on the order of multiple confidence levels among the multiple first output information, at least two first output information are determined from the multiple first output information.

6. The method according to claim 1, wherein, The determination of the second annotation information for each of the M second image data includes: For each of the M second image data sets, the multiple first deep learning models are used to process the second image data sets respectively, resulting in multiple second output information sets for the second image data sets; wherein, each of the multiple second output information sets corresponds one-to-one with the multiple first deep learning models, and each second output information set includes a second initial pseudo-label and the confidence level of the second initial pseudo-label; and At least two of the multiple second output information are fused to obtain the second label and the confidence level of the second label.

7. The method according to claim 1, wherein, The first label, the second label, and the predetermined data distribution satisfy at least one of the following: The first label includes a detection box, and the second label includes a detection box, and the predetermined data distribution includes a predetermined distribution of detection box sizes; The first label includes a category, and the second label includes a category, and the predetermined data distribution includes a predetermined category distribution; and The first label includes a confidence level, and the second label includes a confidence level, and the predetermined data distribution includes a predetermined confidence level distribution.

8. A method for training a deep learning model, comprising: Obtain training samples; as well as The second deep learning model is trained using the training samples. The training samples are obtained using the method described in any one of claims 1 to 7.

9. An object detection method, comprising: Acquire third image data; The third image data is input into a third deep learning model to obtain third output information of the third image data. The third output information includes confidence and at least one of the following: detection boxes and classification results for target objects in the third image data. The third deep learning model is trained using the method described in claim 8.

10. An apparatus for generating training samples, comprising: The first information acquisition module is used to process the first image data using multiple first deep learning models respectively to obtain multiple first output information for the first image data; wherein, the multiple first output information corresponds one-to-one with the multiple first deep learning models, and each first output information includes a first initial pseudo-label and the confidence level of the first initial pseudo-label; A fusion module is used to fuse at least two of the plurality of first output information to obtain an intermediate pseudo-label and the confidence level of the intermediate pseudo-label; and The first determining module is used to determine first annotation information for the first image data based on the intermediate pseudo-label and the confidence level of the intermediate pseudo-label, thereby obtaining a first training sample; the first annotation information includes a first label and the confidence level of the first label; The number of the first image data is N, where N is an integer greater than or equal to 2; The device further includes: The third determining module is used to determine the second annotation information of each of the M second image data in response to detecting that the data distribution of the N first annotation information for the N first image data is consistent with a predetermined data distribution; wherein the second annotation information includes at least one of the following: a second label and the confidence level of the second label, and M is an integer greater than or equal to 2; The fourth determining module is configured to determine at least one target image data from the M second image data based on the second annotation information of each of the M second image data, the data distribution, and the predetermined data distribution; and The generation module is configured to generate a second training sample based on the at least one target image data and the second annotation information of each of the at least one target image data, so that the second training sample supplements the first training sample to obtain a training set whose data distribution satisfies the predetermined data distribution, wherein the predetermined data distribution follows a normal distribution or a uniform distribution. The fourth determining module includes: The sixth determining submodule is used to determine the target annotation information and the number of targets based on the deviation between the data distribution of the N first annotation information and the predetermined data distribution; and The selection submodule is used to select, from the M second image data, a number of second image data whose second annotation information is consistent with the target annotation information, as the at least one target image data.

11. The apparatus according to claim 10, wherein, The fusion module includes at least one of the following: The first determining submodule is used to determine the position information of the detection box in the intermediate pseudo-label based on the position information and size information of the multiple detection boxes contained in the first initial pseudo-label in the at least two first output information. The second determining submodule is used to determine the category in the intermediate pseudo-label based on the multiple categories contained in the first initial pseudo-label in the at least two first output information; as well as The third determining submodule is used to determine the confidence level of the intermediate pseudo-label based on the weighted average of multiple confidence levels in the at least two first output information.

12. The apparatus according to claim 10, wherein, The first determining module includes: The fourth determining submodule is configured to, in response to detecting that the first image data corresponds to an original label and the confidence level of the original label, determine the first label based on the intermediate pseudo-label and the original label; and determine the confidence level of the first label based on the confidence level of the intermediate pseudo-label and the confidence level of the original label; and The fifth determining submodule is used to, in response to the detection that the first image data lacks an original label, determine the intermediate pseudo-label and the confidence level of the intermediate pseudo-label as the first label and the confidence level of the first label, respectively.

13. The apparatus according to claim 12, wherein, The fourth determining submodule includes: A first determining unit is configured to, in response to detecting the existence of the original label and the intermediate pseudo label, and that the original label and the intermediate pseudo label are consistent, determine one of the following as the first label: the weighted average of the original label and the intermediate pseudo label, or the original label and the intermediate pseudo label; The second determining unit is configured to determine the intermediate pseudo-label as the first label in response to detecting the existence of the original label and the intermediate pseudo-label, and the original label and the intermediate pseudo-label being inconsistent; A third determining unit is configured to, in response to detecting the presence of the intermediate pseudo-label and the absence of the original label, determine the intermediate pseudo-label as the first label; and The fourth determining unit is configured to determine that the first label is empty in response to the detection that the original label exists and the intermediate pseudo label is missing.

14. The apparatus of claim 10, further comprising: The second determining module is used to determine at least two first output information from the plurality of first output information according to the order of multiple confidence levels among the plurality of first output information.

15. The apparatus according to claim 10, wherein, The third determining module includes: The submodule is configured to process each of the M second image data sets using the plurality of first deep learning models to obtain a plurality of second output information sets for each second image data set; wherein, the plurality of second output information sets correspond one-to-one with the plurality of first deep learning models, and each second output information set includes a second initial pseudo-label and the confidence level of the second initial pseudo-label; and The fusion submodule is used to fuse at least two of the plurality of second output information to obtain the second label and the confidence level of the second label.

16. The apparatus according to claim 10, wherein, The first label, the second label, and the predetermined data distribution satisfy at least one of the following: The first label includes a detection box, and the second label includes a detection box, and the predetermined data distribution includes a predetermined distribution of detection box sizes; The first label includes a category, and the second label includes a category, and the predetermined data distribution includes a predetermined category distribution; and The first label includes a confidence level, and the second label includes a confidence level, and the predetermined data distribution includes a predetermined confidence level distribution.

17. A training device for a deep learning model, comprising: The first acquisition module is used to acquire training samples; as well as The training module is used to train a second deep learning model using the training samples; The training samples are obtained using the apparatus described in any one of claims 10 to 16.

18. An object detection device, comprising: The second acquisition module is used to acquire the third image data; The second information acquisition module is used to input the third image data into the third deep learning model to obtain the third output information of the third image data. The third output information includes confidence and at least one of the following: detection boxes and classification results for target objects in the third image data. The third deep learning model is trained using the apparatus described in claim 17.

19. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 9.

21. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images