Model training methods, devices, equipment, storage media, and program products

By employing a dual-branch model training method, and utilizing labeled sample data and pseudo-labeled data, a shared feature extraction network is used for data prediction. This solves the problems of high cost and low accuracy of manual labeling, and improves the robustness and accuracy of the data prediction model.

CN116994081BActive Publication Date: 2026-11-17TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202211242807.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-10-11
Publication Date
2026-11-17
Estimated Expiration
2042-10-11

AI Technical Summary

Technical Problem

In existing technologies, manual labeling of data is costly, time-consuming, and prone to errors or omissions, resulting in poor robustness of deep learning models and reduced accuracy of data prediction.

Method used

A dual-branch model training method is adopted, which uses the first data labeled with sample labels and the second data labeled with pseudo-labels. The first candidate prediction model and the second candidate prediction model of the shared feature extraction network are used to predict the data respectively, calculate the loss value and train the model to improve the tolerance to pseudo-labels.

Benefits of technology

It improves the robustness of the data prediction model, increases the accuracy of data prediction, and reduces the negative impact of pseudo-labels on the training process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116994081B_ABST
    Figure CN116994081B_ABST
Patent Text Reader

Abstract

This application discloses a model training method, apparatus, device, storage medium, and program product, relating to the field of artificial intelligence. The method includes: acquiring a first sample dataset and a second sample dataset; performing data prediction on the first data using a first candidate prediction model, and determining a first loss value based on the difference between the obtained first prediction result and the sample label; performing data prediction on the second data using a second candidate prediction model, and determining a second loss value based on the difference between the obtained second prediction result and the pseudo-label; sharing a feature extraction network between the first and second candidate prediction models; and training the first candidate prediction model based on the first and second loss values ​​to obtain a data prediction model. This method improves the data prediction model's tolerance to biased pseudo-labels to a certain extent, thereby enhancing the robustness of the data prediction model. It can be applied to various scenarios such as cloud technology, artificial intelligence, intelligent transportation, and assisted driving.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a model training method, apparatus, device, storage medium, and program product. Background Technology

[0002] In the process of data identification, various types of data are often labeled. These labels can be used to indicate the content, scene, type, etc. of the data. For example, an image can be labeled with the scene label, such as labeling image 'a' with the word 'landscape'. It is evident that data labeling is an important technique for understanding different types of data.

[0003] In related technologies, manual annotation is usually used to label large batches of data, and then deep learning models are used for training to obtain the data labels.

[0004] However, in this approach, manual labeling is expensive and time-consuming, and the large number of data labels can easily lead to mislabeling or omissions, resulting in poor robustness of the final trained deep learning model and reducing the accuracy of data prediction to some extent. Summary of the Invention

[0005] This application provides a model training method, apparatus, device, storage medium, and program product to improve the accuracy of data prediction. The technical solution is as follows.

[0006] On the one hand, a model training method is provided, which includes:

[0007] Obtain a first sample dataset and a second sample dataset. The first sample dataset includes first data labeled with sample labels, and the second sample dataset includes second data labeled with pseudo-labels. The sample labels are used to indicate the data content of the first data, and the pseudo-labels are labels obtained by predicting the content of the second data through a pseudo-label prediction model that has been pre-trained.

[0008] The first data is predicted using a first candidate prediction model, and a first loss value is determined based on the difference between the obtained first prediction result and the sample label.

[0009] The second data is predicted using a second candidate prediction model, and a second loss value is determined based on the difference between the obtained second prediction result and the pseudo label; wherein, the first candidate prediction model and the second candidate prediction model share a feature extraction network.

[0010] The first candidate prediction model is trained based on the first loss value and the second loss value to obtain the data prediction model.

[0011] On the other hand, a model training apparatus is provided, the apparatus comprising:

[0012] The acquisition module is used to acquire a first sample dataset and a second sample dataset. The first sample dataset includes first data labeled with sample labels, and the second sample dataset includes second data labeled with pseudo labels. The sample labels are used to indicate the data content of the first data, and the pseudo labels are labels obtained by predicting the content of the second data through a pseudo label prediction model that has been pre-trained.

[0013] The determination module is used to perform data prediction on the first data using a first candidate prediction model, and determine a first loss value based on the difference between the obtained first prediction result and the sample label.

[0014] The determining module is further configured to perform data prediction on the second data using a second candidate prediction model, and determine a second loss value based on the difference between the obtained second prediction result and the pseudo label; wherein the first candidate prediction model and the second candidate prediction model share a feature extraction network;

[0015] The training module is used to train the first candidate prediction model based on the first loss value and the second loss value to obtain the data prediction model.

[0016] On the other hand, a computer device is provided, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the model training method as described in any of the embodiments of this application above.

[0017] On the other hand, a computer-readable storage medium is provided, which stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the model training method as described in any of the embodiments of this application above.

[0018] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the model training methods described in the above embodiments.

[0019] The beneficial effects of the technical solutions provided in this application include at least the following:

[0020] The processing of the first data labeled with sample labels and the processing of the second data labeled with pseudo-labels are decoupled. Both processes share a single feature extraction network. Different candidate prediction models are then used to predict the input data, resulting in a first prediction result for the first data and a second prediction result for the second data. A first loss value, determined by the difference between the first prediction result and the sample labels, and a second loss value, determined by the difference between the second prediction result and the pseudo-labels, are then used to train the first candidate prediction model. This allows the first candidate prediction model to learn the influence of pseudo-labels in the second data during training, improving the model's tolerance to biased pseudo-labels and thus enhancing its robustness. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the data prediction process in related technologies. Figure 1 A schematic diagram of the implementation environment provided by an exemplary embodiment;

[0023] Figure 2 This is an overall framework diagram of a model training method provided in an exemplary embodiment of this application;

[0024] Figure 3 This is a schematic diagram of an implementation environment provided by an exemplary embodiment of this application;

[0025] Figure 4 This is a flowchart of a model training method provided in an exemplary embodiment of this application;

[0026] Figure 5 This is a flowchart illustrating data processing provided in an exemplary embodiment of this application;

[0027] Figure 6 Based on Figure 4 A flowchart is provided for determining a data prediction model;

[0028] Figure 7 Based on Figure 4 Another flowchart is provided for determining the data prediction model;

[0029] Figure 8 This is a flowchart of a model training method provided in another exemplary embodiment of this application;

[0030] Figure 9 This is a schematic diagram illustrating the result of data prediction provided in an exemplary embodiment of this application;

[0031] Figure 10 This is a structural block diagram of a model training apparatus provided in an exemplary embodiment of this application;

[0032] Figure 11 This is a structural block diagram of a model training apparatus provided in another exemplary embodiment of this application;

[0033] Figure 12 This is a structural block diagram of a server provided in an exemplary embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0035] First, a brief introduction to the terms used in the embodiments of this application will be given.

[0036] Sample dataset: refers to the data used to train the model. This sample data includes, but is not limited to, image data, audio data, video data, and text data. In this embodiment, the sample dataset includes a first sample dataset and a second sample dataset. The first sample dataset includes first data labeled with sample tags, and the second sample dataset includes second data labeled with pseudo-labels. Both sample tags and pseudo-labels are used to characterize the data content, scene information, and data type of the sample data. For example, when the sample data is image data, the sample tags and pseudo-labels can be used to characterize the scene information and / or image content corresponding to the image data. Pseudo-labels are used to indicate that after inputting unlabeled data into the pseudo-label prediction model, the pseudo-label prediction model will label the unlabeled data with tag information identifying the data content. In this embodiment, the labeling accuracy of the second data is lower than that of the first data.

[0037] Sample label: Used to indicate at least one of the following: data content, scene information, and data type representing the sample data. In the embodiments of this application, sample data may have a unique sample label identifying itself, or it may have at least two sample labels identifying itself; this application does not limit this. For example, when the sample data is image data, the image may have one sample label, or it may have multiple sample labels; for example, image a may have the sample label "landscape", or it may simultaneously have the sample labels "landscape", "river", "trees", etc.

[0038] Pseudo-labels: These are labels added to unlabeled sample data after it has been input into a pseudo-label prediction model. For example, when the sample data is image data, it is unlabeled. The image data is input into the pseudo-label prediction model, which then labels the image data with corresponding identifiers. In this embodiment, these labels are referred to as pseudo-labels. Similarly, there can be one or more pseudo-labels; this application does not limit the number.

[0039] Tag set: When identifying sample data, the data content corresponding to the sample data is usually diverse. Staff members synthesize the data content corresponding to various types of sample data to generate a corresponding tag set. For example, when the sample data is image data, the sample tags are used to indicate scene information representing the image data; in this embodiment, the tag set contains 390 scene tags.

[0040] In related technologies, methods for predicting data content mainly employ the following approaches:

[0041] Traditional model training algorithms mainly employ manual annotation, labeling large batches of data samples and using these manually labeled samples to train the data prediction model.

[0042] Then, based on the above scheme, a semi-supervised method is introduced to obtain a data prediction model for data prediction. Typically, manual labeling is used to label a small amount of data samples, and then a large amount of unlabeled, noisy data is used to train the candidate prediction model to obtain the data prediction model. This semi-supervised approach mainly falls into two categories:

[0043] (1) As Figure 1 As shown in Figure a above, using a small amount of labeled data X containing sample data... l Train student network 10 and converge it by fixing the loss function L of student network 10. BCE Then, using a large amount of unlabeled data X without labeled sample data... u Train teacher network 11, update the loss value corresponding to teacher network 11 using error direction propagation, and fix the loss function L. MSE Finally, an exponential moving average is applied to the network parameters of student network 10 to update the network parameters of teacher network 11. The updated teacher network is then used as the data prediction network.

[0044] (2) Figure 1As shown in Figure b, a consistency regularization method is used to enhance unlabeled data with different intensities to learn the data feature distribution. The weakly enhanced sample data is input into the candidate prediction network 13 to obtain the predicted value, and this predicted value is used as the pseudo-label corresponding to the strongly enhanced sample data. During this process, the network parameters of the candidate prediction network 13 are continuously updated until convergence. Finally, the converged candidate prediction network is determined as the data prediction network.

[0045] Even though the semi-supervised algorithms mentioned above utilize a small amount of manually labeled sample data to generate pseudo-labels for a large amount of unlabeled data and achieve some results, these semi-supervised algorithms only consider the data classification case and reduce the interference of erroneous pseudo-labels by normalizing the distribution of predicted values. In this process, if the manually labeled sample data contains errors, these errors will affect the loss values ​​generated in each iteration during network training, leading to a large accumulation of errors in the loss values ​​and ultimately affecting the prediction efficiency of the generated data prediction network.

[0046] The aforementioned negative impact occurs in single-class classification scenarios. Imagine the consequences in multi-class classification (multi-label) scenarios: directly using predicted values ​​for thresholding to generate pseudo-labels easily leads to incorrect noise labels (interference labels or erroneous labels). These noise labels are highly unreliable for identifying the content of the sample data. Using such biased noise labels causes errors to accumulate during training, ultimately leading to model instability. For poorly performing categories (here, "poorly performing categories" refers to categories that are difficult to identify, or less common categories, etc.), the bias of the pseudo-labels is even greater. As training progresses, this bias is further amplified, ultimately leading to the Matthew effect.

[0047] To address the technical shortcomings of the aforementioned solutions, this application employs different candidate prediction models for training the first data with labeled samples and the second data without labeled samples. During training, only the first candidate prediction model is trained using the first data with labeled samples (also known as clean data), ensuring that the bias from generated pseudo-labels does not accumulate during training. Simultaneously, to enable the candidate prediction model to acquire better data features from the second data without labeled samples, the loss of the pseudo-labels corresponding to the second data without labeled samples is calculated in gradient backpropagation as a perturbation term for the second sample data (noisy data), thereby increasing the tolerance for biased pseudo-labels and improving the overall performance of the data prediction model.

[0048] Based on the above discussion, the overall framework of the algorithm corresponding to the model training method provided in the embodiments of this application is described in detail below. Figure 2 The entire framework 20 is divided into two sub-modules: the pseudo-label generation module 21 and the dual-branch data prediction module 22.

[0049] In this embodiment of the application, the two modules have different feature classifier (Backbone) and classifier (head) structures.

[0050] In the pseudo-label generation module 21, the first sample dataset X labeled with sample labels is used. l (Referring to a clean dataset) train candidate prediction models 210.

[0051] Optionally, the training process includes: processing the first sample dataset X l The first data is input into the candidate prediction model 210 to obtain the feature representation p corresponding to the first data. l Based on feature representation p l The pseudo-labels of the first data are determined, and based on the difference between the pseudo-labels and the sample labels, the loss value (L) of the fixed candidate prediction model 210 is converged. BCE3 In this embodiment, the loss value is determined as the third loss value, and a trained pseudo-label prediction model is obtained. In this embodiment, the pseudo-label prediction model is used to generate corresponding feature representations for noisy sample data that are not labeled with sample labels. These feature representations are used to indicate the predicted values ​​of each label corresponding to the noisy sample data. The predicted values ​​are converted into one-hot codes as pseudo-labels corresponding to the noisy sample data through threshold division, and the noisy sample data labeled with pseudo-labels is stored as the second data to obtain the second sample dataset.

[0052] Since the pseudo-label prediction model in pseudo-label generation module 21 generates unreliable labels for noisy sample data without labeled samples, combining it with the first candidate prediction model in dual-branch data prediction module 22 can learn more robust features from the noisy sample data.

[0053] Optionally, the dual-branch data prediction module 22 can be configured with two different candidate prediction models, each of which includes a feature extraction network and a classification layer.

[0054] In some embodiments, different candidate prediction models include their respective feature extraction networks and classification layers. That is, the dual-branch data prediction module 22 in this application includes a first candidate prediction model and a second candidate prediction model. The first candidate prediction model includes a first feature extraction network and a first prediction network 221, and the second candidate prediction model includes a second feature extraction network and a second prediction network 222.

[0055] In another exemplary embodiment, different candidate prediction models share a feature extraction network and each has its own corresponding prediction network. That is, the dual-branch data prediction module 22 in this application includes a first candidate prediction model and a second candidate prediction model. The first and second candidate prediction models share a feature extraction network 220. The first candidate prediction model includes a first prediction network 221, and the second candidate prediction model includes a second prediction network 222. The two candidate prediction models share a single feature extraction network, and different prediction networks are used to eliminate unlabeled data X. u Training bias caused by pseudo-labels.

[0056] As can be seen from the above, in the model training method provided in this application embodiment, the first sample dataset (clean dataset, labeled data X) is labeled with sample labels. l The dataset consists of a second sample dataset labeled with pseudo-labels (noisy dataset, unlabeled data X). u Decoupling is performed.

[0057] To prevent the overall model from overfitting the first sample dataset, the second sample dataset (noisy dataset, unlabeled data X) is included in the backpropagation of the two-branch data prediction module 22. u The gradient effect caused by noisy sample data is used as a perturbation term, enabling the final data prediction model to have a higher prediction accuracy. Illustratively, the training process of the two classifiers in the dual-branch data prediction module 22 is as follows: the first candidate prediction model is used to predict the first data; based on the difference between the obtained first prediction result and the sample label, a first loss value L is obtained. BCE1 The second candidate prediction model is used to predict the second data, and the second loss value L is determined based on the difference between the obtained second prediction result and the pseudo-label. BCE2 Finally, based on the first loss value L BCE1 Second loss value L BCE2 The first candidate prediction model is trained to obtain the data prediction model.

[0058] Based on the above brief description, the implementation environment corresponding to this application will be described in detail. Figure 1 This is a schematic diagram of an implementation environment provided by an exemplary embodiment of this application, such as... Figure 3 As shown, this implementation environment includes a terminal 310, a server 320, and a communication network 330, wherein the terminal 310 and the server 320 are connected through the communication network 330. In some optional embodiments, the communication network 330 can be a wired network or a wireless network, and this application embodiment does not limit this.

[0059] In some embodiments, terminal 310 includes, but is not limited to, smartphones, tablets, laptops, desktop computers, smart home appliances, smart vehicle terminals, smart speakers, digital cameras, smart voice interaction devices, and aircraft. In this embodiment, terminal 310 is implemented as a smartphone as an example. Optionally, terminal 310 has a target application installed, which provides a prediction algorithm or module for predicting the content of the data. This target application can be a traditional application, a cloud application, or a mini-program or application module within a host application, such as a quick app, or a web platform; this application does not limit its implementation. This target application can directly achieve the purpose of data prediction, or it can act as an intermediate carrier to upload data to server 320, where server 320 performs content prediction.

[0060] This illustration is based on the example of terminal 310 operating independently. Specifically, the data prediction model 340 is deployed in terminal 310, and terminal 310 performs the data prediction. Illustratively, terminal 310 receives data to be predicted and invokes a prediction algorithm or module to predict the data's content. Terminal 310 can receive data sent from other terminals, directly access local data, or download data from server 320; this application does not limit its scope in this regard.

[0061] In this embodiment, server 320 is used as the execution entity for describing the process. Server 320 provides data prediction services for the target application installed on terminal 310. Optionally, server 320 is equipped with a data prediction model 340, which is a network model trained using the model training method provided in this embodiment. For example, after receiving target data, server 320 inputs the target data into the data prediction model, predicts the data content of the target data through the data prediction model, obtains the prediction result corresponding to the target data, and feeds back the prediction result to terminal 310 for display.

[0062] It is worth noting that server 320 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), big data and artificial intelligence platforms.

[0063] Cloud technology refers to a managed technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to achieve data computing, storage, processing, and sharing.

[0064] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, be used on demand, and flexibly accessed. Cloud computing technology will become a crucial support. The backend servers of technical network systems require substantial computing and storage resources. Examples include video websites, image-based websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to a backend system for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing. Optionally, server 320 can also be implemented as a node in a blockchain system.

[0065] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user data. These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without user confirmation), the steps for collecting user data end, meaning no user data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0066] Secondly, the application scenarios of this application will be explained.

[0067] I. Application in Image Data Prediction Scenarios. The image includes, but is not limited to, images captured by cameras, images obtained through screenshot tools, images saved or previewed in applications or web pages, and user-generated images. Operations such as data content recognition (prediction), scene recognition (prediction), feature recognition (prediction), and text recognition (prediction) are required for the image. In this case, the data prediction method provided in this application embodiment can be used to predict the image data and obtain the target result corresponding to the image; the target result can be one result or multiple results, and this application does not limit this. For example, scene recognition can be performed on the image data to obtain one or more scene labels corresponding to the image data, such as: landscape, river, trees, etc.

[0068] II. Application in video data prediction scenarios. This video includes, but is not limited to, user-recorded images, videos created through editing applications, online videos, and local videos. In such cases, it may be necessary to identify (predict) the type of audio data contained in the video data, the name of the audio data, and the genre to which it belongs. The data prediction method provided in this application can be used to predict the video data and obtain the target result corresponding to the target. For example, identifying (predicting) the name of the audio data contained in the video data yields an identification (prediction) result corresponding to the video data, such as "aaa" and "abc".

[0069] III. Application in Text Data Prediction Scenarios. Text data refers to editable character data, such as articles and communication information in instant messaging applications. The data prediction method provided in this application can analyze and identify the text structure, descriptive content, keywords, etc., of text data, ultimately yielding the target result corresponding to the text data. For example, taking an article as an example, the method provided in this embodiment analyzes the text structure of the article and marks the title and body text.

[0070] IV. Application in Data Classification Scenarios. This is primarily used to predict the data type of input target data, including but not limited to audio / video, text, image, and medical data. For example, batch data is input into the data prediction model obtained using the model training method provided in this application embodiment. The data prediction model classifies each data point according to its built-in algorithm logic and obtains prediction results for each data type. For instance, inputting data 1-10 into the data prediction model yields the following prediction results: [Image type (data 1, data 2, data 4, data 5), ​​Text type (data 3, data 7), Medical type (data 6, data 8, data 9, data 10)]; it should also be noted that the data prediction model provided in this application embodiment can be used to perform more detailed classification of individual data types.

[0071] It is worth noting that the above application scenarios are merely illustrative examples, and can also be applied to scenarios requiring data prediction, such as cloud technology, artificial intelligence, smart transportation, and assisted driving. This application does not limit the application scenarios of the data prediction method.

[0072] Next, in combination with the above Figure 2 The overall framework of the data prediction model is discussed, and the training process of the data prediction model is described in detail using the model training method provided in the embodiments of this application. For details, please refer to [link to relevant documentation]. Figure 4 , Figure 4A flowchart illustrating a data prediction method provided in an exemplary embodiment of this application is shown. Figure 3 As shown, the method includes steps 401 to 404. It should be noted that... Figure 4 The illustrated embodiment describes the process of training a model on data with a single label.

[0073] Step 401: Obtain the first sample dataset and the second sample dataset.

[0074] The first sample dataset includes first data labeled with sample labels, and the second sample dataset includes second data labeled with pseudo-labels.

[0075] Optionally, after obtaining the first sample dataset, the candidate prediction model can be trained using the first sample dataset; a pseudo-label prediction model is obtained when the model parameters of the candidate prediction model meet the second parameter threshold. Alternatively, other labeled sample data can also be used to train the candidate prediction model.

[0076] Optionally, the pseudo-label prediction model can be applied to the subsequent process of determining the pseudo-label of the second data. The second data is input into the pseudo-label prediction model, and the pseudo-label corresponding to the second data is output.

[0077] Indicatively, the process includes inputting the second data into a pseudo-label prediction model to obtain a third sample representation corresponding to the second data, the third sample representation being used to indicate the data content and / or scene information corresponding to the second data; matching the third feature representation with a label set, and outputting the third value distribution of the label set corresponding to the third feature representation as the pseudo-label result of the second data.

[0078] Optionally, the second data labeled with pseudo-labels can be stored in a preset dataset to obtain the second sample dataset.

[0079] The first and second data are of the same data type, and the data content includes, but is not limited to, image data, text data, video data, audio data, medical samples, etc. For example, the first and second data are used to indicate image data, collecting a first sample dataset labeled with sample tags, and a second sample dataset labeled with pseudo-labels.

[0080] Sample labels are used to indicate the data content of the first data. In this embodiment, sample labels are manually labeled to form a first sample dataset, which is referred to as a clean dataset. That is, the sample labels are labels that truly and correctly represent the data content of the first data. For example, taking image data as an example, the first data is labeled with the sample labels "landscape" and "river," meaning the first data is used to indicate a landscape image with a river; taking audio data as an example, the first data is labeled with the sample labels "dialogue," "boy 1," and "boy 2," meaning the first data is used to indicate that boy 1 and boy 2 are having a dialogue.

[0081] Pseudo-labels are labels obtained by predicting the content of second data using a pre-trained pseudo-label prediction model. In this embodiment, second data is acquired, and after inputting the second data into the pseudo-label prediction model, pseudo-labels representing the data content of the second data are obtained. Since pseudo-labels are labeled using a pseudo-label prediction model, inaccurate pseudo-labels are inevitable. Therefore, in this embodiment, the second data is referred to as a noisy dataset. For example, taking image data as an example, the second data is labeled with pseudo-labels "landscape" and "river." That is, the second data may be used to indicate a landscape image with a river, but the label prediction for "landscape" or "river" may be incorrect.

[0082] Please refer to the following for the process of obtaining sample labels and pseudo-labels. Figure 5 , Figure 5 A flowchart illustrating the specific processes for the first and second sample datasets is shown. This process includes data acquisition (500), data cleaning (501), manual annotation (502), and model processing (503). During data acquisition (500), a large amount of data is collected. In this embodiment, at least two data points corresponding to each label category are obtained according to the label system (the label set mentioned later). During the collection process, data unrelated to the label set is inevitably acquired. Therefore, data cleaning (501) is used to clean the data collected in data acquisition (500), removing data irrelevant to the label set to obtain a candidate dataset. The candidate dataset is then divided into two batches for different processing: the first candidate dataset and the second candidate dataset. The number of data points in the first and second candidate datasets is set by the relevant personnel. Manual annotation (502) is used to label the first data points in the first candidate dataset. For example, 50-100 data points are selected for each label category according to the label set for annotation, resulting in the first sample dataset. Model processing (503) is used to output a pseudo-label prediction model to label the second data points in the second candidate dataset, resulting in the second dataset.

[0083] In other words, the accuracy of the sample labels in representing the data content of the first data is higher than the accuracy of the pseudo-labels in representing the data content of the second data.

[0084] In another exemplary embodiment, sample tags can also be used to represent scene information of the first data, and pseudo tags can also be used to represent scene information of the second data; additionally, sample tags can also be used to represent the data content and scene information of the first data, and pseudo tags can also be used to represent the data content and scene information of the second data. In the embodiments of this application, scene information is used to characterize the overall environmental information in which the data is located. For example, when the data is image data, the scene information can be natural scenery, residential places, sports activity venues, etc.; when the data is audio data, the scene information can be dialogue scenes, monologue scenes, animal scenes, music scenes, etc.

[0085] The label results in the aforementioned sample labels and pseudo labels are directly used to characterize the data content and / or scene information of the first data (corresponding to sample labels) and the second data (corresponding to pseudo labels). In another exemplary embodiment, the sample labels and pseudo labels include all data content and / or corresponding label results. In this embodiment, all labels used to characterize data content and / or scene information are referred to as a label set. For example, when implemented as image data, there are 200 content labels for the data content and 390 scene labels for the scene information. When only the data content is labeled, the label set includes 200 labels; when only the scene information is labeled, the label set includes 390 labels; and when both data content and scene information are labeled, the label set includes 590 labels.

[0086] In other words, sample labels are used to represent the value distribution of the label set corresponding to the first data, while pseudo-labels are used to represent the value distribution of the label set corresponding to the second data.

[0087] In the value distribution, labels with the first value are considered relevant to the data content, while labels with the second value are considered irrelevant to the data content. Alternatively, labels with the first value are considered relevant to the scene information, while labels with the second value are considered irrelevant to the scene information.

[0088] In this embodiment, the first value is set to 1, and the second value is set to 0. For example, taking image data as an example, the label set is [natural scenery, sky, trees, river, volcano]. The value distribution corresponding to the sample labels of the first image data is [1, 1, 0, 1, 0]. This value distribution actually expresses the meaning that the first image data is a natural scenery image including the sky and river. The value distribution of the pseudo-labels corresponding to the second image data is [1, 0, 0, 0, 1]. This value distribution actually expresses the meaning that the second image data is a natural scenery image including a volcano.

[0089] Because of the content of the value distribution, the data prediction method provided in this application embodiment can predict or identify data for both single labels and multiple labels. This application does not limit this.

[0090] In the subsequent process, a candidate prediction model is trained using the clean dataset (the first sample dataset) and the noisy dataset (the second dataset) to determine the data prediction model.

[0091] Optionally, the number of first data points in the first sample dataset is less than the number of second data points in the second dataset; or, the number of first data points in the first sample dataset is equal to the number of second data points in the second dataset; or, the number of first data points in the first sample dataset is greater than the number of second data points in the second dataset. In this embodiment, the number of first data points in the first sample dataset is less than the number of second data points in the second dataset. To improve the robustness of the data prediction model and its ability to learn noisy data, the number of first data points in the first sample dataset is much smaller than the number of second data points in the second dataset.

[0092] Step 402: Perform data prediction on the first data using the first candidate prediction model, and determine the first loss value based on the difference between the obtained first prediction result and the sample label.

[0093] Optionally, the first data in the first sample dataset is input into the first candidate prediction model for data prediction to obtain the first prediction result corresponding to the first data. The first prediction result includes the label result of predicting the data content of the first data, or the value distribution result of the label set corresponding to the first data.

[0094] In this embodiment, the first candidate prediction model includes a feature extraction network and a first prediction network. The feature extraction network is used to instruct the extraction of at least one feature from the input data's data content, scene information, and data type. The first prediction network is used to classify the feature representation input by the feature extraction network; that is, the first prediction network outputs the label result corresponding to the input data, or the value distribution result of the label set corresponding to the input data. Optionally, the data content represents the actual data content information represented by the input data, the scene information represents the overall environmental information in which the input data exists, and the data type represents the data type corresponding to the input data, wherein the input data is the first data.

[0095] Feature extraction networks are used to extract features from the first data to obtain a first feature representation corresponding to the first data. This first feature representation can be represented in the form of a matrix, list, etc.; for example, the matrix values ​​within the matrix represent the extracted data content and / or scene information. Combining this with the concept of a tag set, the data content and / or scene information here are equivalent to the various tags within the tag set.

[0096] Optionally, the first feature representation is processed by the first prediction network in the first candidate prediction model, the first feature representation is matched with the label set, the first threshold distribution result corresponding to each label in the label set is determined, and the first loss value is determined based on the difference between the first threshold distribution result and the sample label.

[0097] In a schematic way, the label corresponding to the maximum value in the first threshold distribution result is taken as the first prediction result. That is, the label corresponding to the maximum value in the first threshold distribution result is used to label the first data.

[0098] Alternatively, the labels that meet the threshold range in the first threshold distribution result can be used as the first prediction result. In this case, the first data corresponds to at least two labels.

[0099] Alternatively, the aforementioned first and second values ​​can be used to characterize whether there is a correlation with the labels in the label set. That is, the first feature represents the first value distribution of the corresponding label set as the first prediction result; the first loss value is determined based on the difference between the first value distribution and the sample labels; under this determination method, the first data also has at least two labels.

[0100] In this embodiment of the application, a first loss value is obtained based on the difference between the first prediction result and the sample label.

[0101] Step 403: Perform data prediction on the second data using the second candidate prediction model, and determine the second loss value based on the difference between the obtained second prediction result and the pseudo label.

[0102] Optionally, the second data in the second sample dataset is input into the second candidate prediction model for data prediction to obtain the second prediction result corresponding to the second data. The second prediction result includes the label result of predicting the data content of the first data, or the value distribution result of the label set corresponding to the first data.

[0103] In this embodiment, the second candidate prediction model includes a feature extraction network and a second prediction network. It is important to note that the first and second candidate prediction models share a single feature extraction network. The feature extraction network is used to extract at least one feature from the input data's content, scene information, and data type. The second prediction network is used to classify the feature representation input by the feature extraction network; that is, the second prediction network outputs the label result corresponding to the input data, or the value distribution result of the label set corresponding to the input data. In this scenario, the input data is the second data.

[0104] The second data is processed by a feature extraction network to extract features, resulting in a second feature representation. This second feature representation can also be represented in the form of a matrix, list, etc., and this application does not limit its representation to this form.

[0105] Optionally, the second feature representation is processed by the second prediction network in the second candidate prediction model, the second feature representation is matched with the label set, the second threshold distribution result corresponding to each label in the label set is determined, and the second loss value is determined based on the difference between the second threshold distribution result and the pseudo label.

[0106] In a schematic way, the label corresponding to the maximum value in the second threshold distribution result is used as the second prediction result. That is, the label corresponding to the maximum value in the second threshold distribution result is used to label the second data.

[0107] Alternatively, the labels that meet the threshold range in the second threshold distribution results can be used as the second prediction results. In this case, the second data corresponds to at least two labels.

[0108] Alternatively, the aforementioned first and second values ​​can be used to characterize whether there is a correlation with the labels in the label set. That is, the second feature represents the second value distribution of the corresponding label set as the second prediction result; the second loss value is determined based on the difference between the second value distribution and the pseudo label; under this determination method, the first data also has at least two labels.

[0109] In this embodiment of the application, a second loss value is obtained based on the difference between the second prediction result and the sample label.

[0110] Step 404: Train the first candidate prediction model based on the first loss value and the second loss value to obtain the data prediction model.

[0111] In this embodiment of the application, after determining the first loss value and the second loss value, different weight values ​​are assigned to the first loss value and the second loss value; the model parameters of the first candidate prediction model are updated based on the sum of the first loss values ​​corresponding to the first weight and the sum of the second loss values ​​corresponding to the second weight; and the first candidate prediction model with updated model parameters is determined as the data prediction model.

[0112] Optionally, the model parameters of the first candidate prediction model can be updated using the first and second loss values ​​through backpropagation to obtain the data prediction model.

[0113] Optionally, the data prediction model is trained on the first candidate prediction model. Throughout the training process, the first candidate prediction model does not process the second dataset labeled with pseudo-labels. Instead, it continuously utilizes the first dataset labeled with sample labels. During the update of the model parameters of the first candidate prediction model, the second dataset labeled with pseudo-labels is introduced as a gradient influence for backpropagation (update). In other words, throughout the training process, the first candidate prediction model, which has always been used to process clean datasets, learns a wider range of possibilities from the noisy dataset (second dataset), improving its ability to learn from noisy data and thus enhancing the robustness of the data prediction model.

[0114] In this application's embodiments, the data prediction model is used to predict the input data. This prediction can be data content prediction, data type prediction, or scenario prediction; this application does not limit this. The determined data prediction model can be widely applied to scenarios such as video scenarios, text scenarios, audio scenarios, medical scenarios, and intelligent vehicle scenarios. For example, in intelligent vehicle scenarios, it can be used to analyze road condition information captured by the vehicle and provide corresponding driving suggestions based on the analysis results.

[0115] In summary, the data prediction method provided in this application decouples the processing of first data labeled with sample labels from the processing of second data labeled with pseudo-labels. The two processes share a single feature extraction network, and then use different candidate prediction models to predict their respective input data, obtaining a first prediction result for the first data and a second prediction result for the second data. Then, a first loss value determined based on the difference between the first prediction result and the sample label, and a second loss value determined based on the difference between the second prediction result and the pseudo-label, are used to train the first candidate prediction model. This allows the first candidate prediction model to learn the influence of pseudo-labels in the second data during training, improving the data prediction model's tolerance to biased pseudo-labels to some extent, thereby enhancing the robustness of the data prediction model.

[0116] Please see details. Figure 6 , Figure 6 A flowchart of a data prediction method provided by an exemplary embodiment of this application is shown. Figure 6 The process of determining the data prediction model is described in detail below. That is, step 404 above is described in detail, and step 404 may also include steps 601 to 603 below.

[0117] Step 601: Determine the first number of first data in the first sample dataset and the second number of second data in the second sample dataset.

[0118] Optionally, the first sample dataset includes first data labeled with sample labels, and the second data includes second data labeled with pseudo-labels.

[0119] For an introduction to the first and second sample datasets, please refer to step 401 above.

[0120] In this embodiment, a first quantity of first data and a second quantity of second data are selected to complete the training process of the pseudo-label prediction model, the first candidate prediction model, and the second candidate prediction model, so as to obtain the final data prediction model. The first quantity is less than the second quantity.

[0121] Optionally, the specific selection rules for the first and second quantities can be found in the following examples of the verification of model metrics.

[0122] Step 602: Based on the weighted processing result of the first loss value on the first weight corresponding to the first quantity, and the weighted processing result of the second loss value on the second weight corresponding to the second quantity, the predicted loss value is obtained.

[0123] Optionally, the first loss value includes the loss sum corresponding to the first number of first data points, and the second loss value includes the loss sum corresponding to the second number of second data points.

[0124] In this embodiment of the application, since the quantities of the first data and the second data are different, the average values ​​of the loss values ​​corresponding to the first data and the second data are different. That is, the first quantity is set as the first weight for calculating the first loss value, and the second quantity is set as the second weight for calculating the second loss value.

[0125] In this embodiment of the application, the aforementioned first value and second value are used to characterize whether there is an association relationship with the labels in the label set. That is, the first feature represents the first value distribution of the corresponding label set as the first prediction result; the first loss value is determined based on the difference between the first value distribution and the sample labels; the second feature represents the second value distribution of the corresponding label set as the second prediction result; and the second loss value is determined based on the difference between the second value distribution and the pseudo labels.

[0126] In some embodiments, a first ratio of a first loss value to a first quantity is determined as the average loss value of the first data in the first sample dataset; and a second ratio of a second loss value to a second quantity is determined as the average loss value of the second data in the first sample dataset.

[0127] The sum of the first ratio and the second ratio is determined as the predicted loss value. Optionally, the process for determining the predicted loss value can be found in Formula 1 below.

[0128] Formula 1:

[0129] In formula 1, n l F is used to represent the first number of the first data points in the first sample dataset. c Used to represent a feature extraction network, h l Used to indicate the first prediction network within the first candidate prediction model, (F c °h l This is used to indicate that the first candidate prediction model is used to predict the first data. This is used to indicate the i-th data point in the first sample dataset, where i represents less than or equal to n. l positive integers, Used to indicate The first prediction result obtained from the first candidate prediction model is input; that is, the first part of Formula 1 (to the left of the + sign, to the right of the equals sign) is the process of determining the first loss value (first ratio); n u h is used to represent the second quantity of the second data within the second dataset. u Used to indicate the second prediction network in the second candidate prediction model, (F c °h u This is used to indicate that a second candidate prediction model is used to predict the second data. This is used to indicate the j-th data point in the second dataset, where j represents less than or equal to n. u positive integers, Used to indicate The second prediction result obtained from the second candidate prediction model is input. In other words, the second half of Formula 1 (to the right of the + sign) is the process of determining the second loss value (second ratio).

[0130] Optionally, the model parameters of the first candidate prediction model can be updated using Formula 1 above.

[0131] Step 603: Train the first candidate prediction model based on the prediction loss value to obtain the data prediction model.

[0132] Optionally, in response to the predicted loss value meeting the first parameter threshold, the model parameters corresponding to the first candidate prediction model are fixed, and a data prediction model is obtained based on the fixed model parameters. Throughout the model training process, both the feature extraction network and the first prediction network in the first candidate prediction model are trained together.

[0133] In summary, the data prediction method provided in this application decouples the processing of first data labeled with sample labels from the processing of second data labeled with pseudo-labels. The two processes share a single feature extraction network, and then use different candidate prediction models to predict their respective input data, obtaining a first prediction result for the first data and a second prediction result for the second data. Then, a first loss value determined based on the difference between the first prediction result and the sample label, and a second loss value determined based on the difference between the second prediction result and the pseudo-label, are used to train the first candidate prediction model. This allows the first candidate prediction model to learn the influence of pseudo-labels in the second data during training, improving the data prediction model's tolerance to biased pseudo-labels to some extent, thereby enhancing the robustness of the data prediction model.

[0134] Please see details. Figure 7 , Figure 7 A flowchart of a data prediction method provided by another exemplary embodiment of this application is shown. Figure 7 The process of determining the data prediction model is described in detail below. The method includes the following steps 701 to 702.

[0135] Step 701: Obtain the third loss value corresponding to the candidate prediction model.

[0136] In this embodiment of the application, after obtaining the first sample dataset, the candidate prediction model is trained using the first sample dataset to obtain the pseudo-label prediction model.

[0137] During the training of the pseudo-label prediction model, the candidate prediction model is converged and fixed, and a third loss value is determined for the candidate prediction model. The third loss value is used to represent the difference between the prediction result of the candidate prediction model and the sample label of the first data.

[0138] Step 702: Train the first candidate prediction model based on the first loss value, the second loss value, and the third loss value to obtain the data prediction model.

[0139] Optionally, the process for determining the first loss value and the second loss value can be found in step 602 above, and will not be repeated here.

[0140] In the embodiments of this application, during the generation of pseudo-labels, that is, the process of using the pseudo-label prediction model to label the second data with pseudo-labels, the accuracy of the pseudo-labels cannot be guaranteed. Therefore, when considering the influence of the first candidate prediction model and the introduced noisy data (the second data labeled with pseudo-labels), the third loss value corresponding to the pseudo-label prediction model is included in the second loss value to update the first loss value.

[0141] Optionally, the calculation process of the loss value corresponding to the training of the first candidate prediction model based on the first loss value, the second loss value and the third loss value can be found in Formula 2 below.

[0142] Formula 2:

[0143] In formula 2, n l F is used to represent the first number of the first data points in the first sample dataset. c Used to represent a feature extraction network, h l Used to indicate the first prediction network in the first candidate prediction model. Used to instruct the use of a first candidate prediction model to make data predictions on the first data. This is used to indicate the i-th data point in the first sample dataset, where i represents less than or equal to n. l positive integers, Used to indicate The first prediction result obtained from the first candidate prediction model is input; that is, the first part of Formula 2 (to the left of the + sign, to the right of the equals sign) is the process of determining the first loss value (first ratio); n u h is used to represent the second quantity of the second data within the second dataset. u Used to indicate the second prediction network in the second candidate prediction model. Used to instruct the second candidate prediction model to make data predictions for the second data. This is used to indicate the j-th data point in the second dataset, where j represents less than or equal to n. u positive integers, Used to indicate The second prediction result obtained from the input second candidate prediction model means that the second half of Formula 2 (to the right of the + sign) is the impact of the noise data introduced during the pseudo-label training process on the second candidate prediction model.

[0144] Optionally, the model parameters of the first candidate prediction model are updated using Formula 2 above. In response to the model parameters of the first candidate prediction model meeting the first parameter threshold, a data prediction model is obtained.

[0145] In an optional embodiment, after obtaining the data prediction model, its model parameters can be transferred to the pseudo-label prediction model for practical application. Since the final model parameters of the data prediction model are calculated under the influence of pseudo-label data, the model learns the perturbation effect of the pseudo-label data during application. Based on this perturbation effect, the completeness and accuracy of the data prediction model can be improved. Therefore, in this embodiment, directly transferring the model parameters of the data prediction model to the pseudo-label prediction model can achieve the same technical effect.

[0146] In another optional embodiment, after obtaining the data prediction model, the data prediction model is used as the updated pseudo-label prediction model to perform pseudo-label recognition on the second data to obtain the recognition result; the recognition result is used as the updated pseudo-label of the second data, and the second data and the updated pseudo-label are used to iteratively train the first candidate prediction model. In this scheme, the model parameters of the data prediction model are transferred to the pseudo-label prediction model, and the model parameters corresponding to the subsequent first candidate prediction model are further converged to obtain a more robust data prediction model.

[0147] In summary, the model training method provided in this application decouples the processing of the first data labeled with sample labels from the processing of the second data labeled with pseudo-labels. The two processes share a single feature extraction network, and then different candidate prediction models are used to predict their respective input data, obtaining a first prediction result for the first data and a second prediction result for the second data. Then, a first loss value determined based on the difference between the first prediction result and the sample labels, and a second loss value determined based on the difference between the second prediction result and the pseudo-labels, are used to train the first candidate prediction model. This allows the first candidate prediction model to learn the influence of pseudo-labels in the second data during training, improving the data prediction model's tolerance to biased pseudo-labels to some extent, thereby enhancing the robustness of the data prediction model.

[0148] Please see details. Figure 8 , Figure 8 A flowchart of a data prediction method provided by an exemplary embodiment of this application is shown. Figure 8 As shown, the method includes the following steps 801 to 805.

[0149] Step 801: Obtain the labeled dataset and the unlabeled dataset.

[0150] Optionally, the labeled dataset is the first sample dataset in step 401 above, and the unlabeled dataset is the second sample dataset in step 401 above that has not yet been labeled with pseudo-labels. In this embodiment of the application, the number of data in the labeled dataset is much smaller than the number of data in the unlabeled dataset.

[0151] Step 802: Input the labeled dataset into the pseudo-label generation module for training until the model parameters of the pseudo-label generation module converge.

[0152] Optionally, the pseudo-label generation module is trained using a labeled dataset. If the model parameters of the pseudo-label generation module meet the parameter threshold during training, the convergence of the model parameters corresponding to the pseudo-label generation module is determined.

[0153] Please refer to step 701 above for details, which will not be repeated here.

[0154] Step 803: Fix the model parameters of the pseudo-label generation module.

[0155] Optionally, in response to the model parameters of the pseudo-label generation module meeting the parameter threshold, the corresponding model parameters are fixed and applied to the subsequent pseudo-labeling process for unlabeled data.

[0156] Please refer to step 701 above for details, which will not be repeated here.

[0157] Step 804: Input the labeled dataset and the unlabeled dataset processed by the fixed pseudo-label generation module into the dual-branch noise label module for training until the model parameters of the dual-branch noise label module converge.

[0158] Optionally, the unlabeled data can be input into the pseudo-label generation module determined in step 804 above to label the unlabeled data with pseudo-labels, thus obtaining an unlabeled dataset labeled with pseudo-labels.

[0159] The unlabeled dataset with pseudo-labels and the labeled dataset are input into the dual-branch noisy label module for training.

[0160] Optionally, the dual-branch noise label module includes a first candidate prediction model and a second candidate prediction model. The first and second candidate prediction models share a single feature extraction network.

[0161] The first candidate prediction model is used to process the labeled dataset, and the second candidate prediction model is used to process the unlabeled dataset. Illustratively, the process includes: predicting data for a first feature representation using the first candidate prediction model, and determining a first loss value based on the difference between the obtained first prediction result and the sample label; predicting data for a second feature representation using the second candidate prediction model, and determining a second loss value based on the difference between the obtained second prediction result and the pseudo-label; obtaining the third loss value corresponding to the pseudo-label generation module determined in step 803 above. The first candidate prediction model is trained based on the first loss value, the second loss value, and the third loss value until the model parameters corresponding to each of the first candidate prediction models converge.

[0162] Please refer to step 702 above for details, which will not be repeated here.

[0163] Step 805: Fix the model parameters of the dual-branch noise label module and apply it.

[0164] Optionally, in response to the model parameters of the dual-branch noise label module meeting the parameter threshold, the model parameters corresponding to meeting the parameter threshold are fixed, and the dual-branch noise module with fixed parameters is used for practical application.

[0165] In the embodiments of this application, during the training process of the first candidate prediction model, in response to the model parameters of the first candidate prediction model meeting the parameter threshold, the model parameters of the first candidate prediction model are fixed; and a fixed dual-branch noise label module is generated based on the first candidate prediction model after the model parameters are fixed.

[0166] In another embodiment, the model parameters of the second candidate prediction model may be updated synchronously with each training round throughout the model training process, or they may not be updated; this application does not limit this.

[0167] Please refer to step 702 above for details, which will not be repeated here.

[0168] In summary, the model training method provided in this application decouples the processing of the first data labeled with sample labels from the processing of the second data labeled with pseudo-labels. The two processes share a single feature extraction network, and then different candidate prediction models are used to predict their respective input data, obtaining a first prediction result for the first data and a second prediction result for the second data. Then, a first loss value determined based on the difference between the first prediction result and the sample labels, and a second loss value determined based on the difference between the second prediction result and the pseudo-labels, are used to train the first candidate prediction model. This allows the first candidate prediction model to learn the influence of pseudo-labels in the second data during training, improving the data prediction model's tolerance to biased pseudo-labels to some extent, thereby enhancing the robustness of the data prediction model.

[0169] Based on the descriptions of the four embodiments above, the following discussion addresses whether the quantity of data in the first sample dataset has varying degrees of impact on the final data prediction model. In the setup process, this application embodiment primarily utilizes the fast-training residual model ResNet-101, trained on different quantities of the first dataset (with the same meaning as the aforementioned first sample dataset). Other models can also be used to complete this experiment; this application does not limit its application to this.

[0170] Optionally, throughout the process, the mean accuracy (mAP), precision, recall, and the harmonic mean F1 of precision and recall corresponding to ResNet-101 are used as experimental metrics to evaluate different amounts of test data.

[0171] In the application embodiment, three first datasets are set: first dataset cls50, first dataset cls100, and first dataset full. First dataset cls50 indicates data within the label set where the number of each label category is selected is a first preset quantity. For example, if the label set has 390 labels and the first preset quantity is 50, the number of first data in first dataset cls50 is 390 * 50. First dataset cls100 indicates the number of first data within the label set where the number of each label category is selected is a second quantity. For example, if the label set has 390 labels and the second preset quantity is 100, the number of first data in first dataset cls100 is 390 * 100. Third dataset full indicates a third preset quantity of labeled first data. Optionally, the first preset quantity < the second preset quantity < the third preset quantity.

[0172] Table 1 below shows the experimental metrics after training ResNet-101 on the first dataset cls50, the first dataset cls100, and the first dataset full.

[0173] Table 1:

[0174] Data volume mAP F1 precision Recall First dataset full 0.7566 0.7622 0.7231 0.8058 First dataset cls50 0.6166 0.6693 0.6544 0.6849 First dataset cls100 0.6786 0.6945 0.6574 0.7360

[0175] As shown in Table 1 above, the parameters of the model cannot meet the actual requirements for either the first dataset cls50 or the first dataset cls100. However, the model trained using the first dataset full achieves a metric higher than 0.7, meeting the model's usage requirements. But the first dataset full requires high precision in label annotation for each data point, leading to high costs for collecting this type of data.

[0176] In all the above experiments, model training was conducted using clean data (data with correct sample labels).

[0177] Therefore, based entirely on clean data for training, and on the first dataset cls50, different proportions of unlabeled data are introduced to train ResNet-101. That is, the model training method provided in the embodiments of this application is used for training. In the following discussion, the model training method provided in the embodiments of this application will be briefly referred to as SSN.

[0178] Specifically, it includes the following five training methods:

[0179] The first method is to introduce an SSN, that is, to introduce a fourth preset number of unlabeled datasets on the first dataset cls50; during the training process, data from the first dataset cls50 and the unlabeled datasets are randomly selected.

[0180] The second approach involves introducing a fifth preset number of unlabeled data points onto the first dataset cls50. During training, data from both the first dataset cls50 and the unlabeled dataset are randomly selected. The fifth preset number is greater than the fourth preset number.

[0181] The third approach is to introduce an SSN, which involves introducing a fifth preset number of unlabeled datasets onto the first dataset cls50. However, during training, data is selected from the first dataset cls50 and the unlabeled datasets using a preset mixing ratio in a batch processing manner.

[0182] The fourth method involves introducing a sixth preset number of unlabeled datasets onto the first dataset cls50. During training, data from both the first dataset cls50 and the unlabeled datasets are randomly selected. The sixth preset number is greater than the fifth preset number.

[0183] The fifth method involves introducing an SSN, which means adding a seventh preset number of unlabeled data points to the first dataset cls50; during training, data from the first dataset cls50 and the unlabeled dataset are randomly selected. The seventh preset number is greater than the sixth preset number.

[0184] Table 2 below shows the experimental metrics results after introducing SSN and training ResNet-101 on the first dataset cls50, the first dataset cls100, and the first dataset full. The fourth preset number is 100,000, the fifth preset number is 200,000, the sixth preset number is 500,000, and the seventh preset number is 1,000,000. The preset mixing ratio is 1:10 (first dataset: unlabeled data).

[0185] Table 2:

[0186]

[0187] As shown in Table 2 above, method a in Table 2, based on training using only the first dataset cls50 in Table 1, introduces the model training method provided in the embodiments of this application. From the perspective of various model metrics, the model's mAP is slightly improved, but the overall F1 score decreases. Furthermore, different amounts of unlabeled data are introduced to explore the impact on model metrics. That is, combining methods b, d, and e in Table 2, the model's mAP and F1 score are improved to varying degrees. Moreover, as the amount of unlabeled data increases, the model's precision and recall reach 0.7, meeting the requirements of the actual model.

[0188] In methods b and c, we need to determine whether the mixing method of the first data and unlabeled data in the first dataset improves the model metrics. Method b randomly samples the first data and unlabeled data, while method c mixes the first data and unlabeled data in a 1:10 ratio during each batch training. From the perspective of model metrics, the performance of methods c and b is similar, which means that the data mixing method is not a factor affecting model performance. The key is the overall ratio of the unlabeled data to the first data (clean data). The more noisy data, the better the model performance.

[0189] To further explore the impact of the amount of labeled data on model metrics, an initial version of the ResNet-101 model was trained using the first dataset cls100, where the first dataset cls100 was used to indicate a second preset number of data points labeled for each label in the label set.

[0190] Table 3 below shows the possible training methods using ResNet-101 on the first dataset cls100, where the second preset number is 10.

[0191] Table 3:

[0192]

[0193] In method f in Table 3, 100 unlabeled data points are selected for semi-supervised noisy training (using the model training method provided in the embodiments of this application, i.e., SSN). The model performance is further improved, and the accuracy is enhanced.

[0194] Both the recall and the model performance are above 0.7. In method g in Table 3, data with model prediction scores below a preset threshold in unlabeled data are removed. Data with low confidence scores are called hard samples, which may interfere with the model's performance. However, Table 3 shows that removing low-confidence data does not improve the overall model performance. This indirectly shows that the model training method proposed in this application has a good tolerance for unlabeled data (noisy data). In methods h and l in Table 3, the data prediction model obtained by using the model training method provided in this application is used as the pre-training model for labeled data. The training effects of methods h and l are compared with the training effects of the original model in Table 1. Both methods show varying degrees of improvement. This proves that the model training method provided in this application can learn more generalized features from unlabeled noisy data, thus improving the model's performance.

[0195] Based on the above discussion, three initial models were obtained: a model trained using the full dataset, a model trained on the cls50 dataset using the SSN method (i.e., the model training method provided in the embodiments of this application), and a model trained on the cls100 dataset using the SSN method (i.e., the model training method provided in the embodiments of this application).

[0196] Optionally, based on the three initial models mentioned above, a manual annotation method can be introduced to visualize the data prediction results for the input data. In this embodiment, image data is used as the input data for illustration. Figure 9As shown, using the manually labeled model 900 and the three initial models mentioned above, data prediction was performed on images a and b, yielding label results corresponding to four methods. The three initial models are the full data model 910, the SSN+cls100 model 920, and the SSN+cls50 model 930. From Figure 9 It can be intuitively seen that using the manual annotation model 900 to label data is prone to missing labels; for example, in image a, labels such as "cloud" and "grass" are easily missed. Similarly, using the full data model 910 is limited by the labeling quality of the full data, and it is also prone to missing labels for difficult data or difficult labels. However, the two models trained on the cls50 and cls100 datasets using the model training method provided in this application can effectively recall some labels. Combined with the experiments on different dataset sizes, it is clear that the accuracy of the SSN+cls100 model 920 is better than that of the SSN+cls50 model 930. This further demonstrates that introducing noise interference during model training with clean data can improve the robustness of the model to a certain extent.

[0197] It should be noted that in the above embodiments, noise data is introduced as interference during the model training process using clean data to improve the accuracy and completeness of data prediction. From another perspective, the model training method provided in this application can also be applied to scenarios where clean data is identified from noisy data. This application does not limit this application.

[0198] Please refer to Figure 10 The diagram illustrates a model training apparatus structure block diagram provided in an exemplary embodiment of this application, which includes the following modules.

[0199] The acquisition module 1000 is used to acquire a first sample dataset and a second sample dataset. The first sample dataset includes first data labeled with sample labels, and the second sample dataset includes second data labeled with pseudo labels. The sample labels are used to indicate the data content of the first data, and the pseudo labels are labels obtained by predicting the content of the second data through a pseudo label prediction model that has been pre-trained.

[0200] The determination module 1010 is used to perform data prediction on the first data through a first candidate prediction model, and determine a first loss value based on the difference between the obtained first prediction result and the sample label.

[0201] The determining module 1010 is further configured to perform data prediction on the second data using a second candidate prediction model, and determine a second loss value based on the difference between the obtained second prediction result and the pseudo label; wherein the first candidate prediction model and the second candidate prediction model share a feature extraction network.

[0202] The determining module 1010 is further configured to train the first candidate prediction model based on the first loss value and the second loss value to obtain a data prediction model.

[0203] In an optional embodiment, the sample label is used to represent the value distribution of the label set corresponding to the first data, and the pseudo label is used to represent the value distribution of the label set corresponding to the second data. In the value distribution, the label marked with the first value indicates that it is related to the data content, and the label marked with the second value indicates that it is unrelated to the data content.

[0204] In an optional embodiment, such as Figure 11 As shown, the device also includes an extraction module 1020;

[0205] The extraction module 1020 is used to extract features from the first data through the feature extraction network to obtain a first feature representation corresponding to the first data;

[0206] The determining module 1010 is further configured to match the first feature representation with the label set to obtain a first value distribution of the label set corresponding to the first feature representation as the first prediction result; and determine the first loss value based on the difference between the first value distribution and the sample label.

[0207] The extraction module 1020 is further configured to extract features from the second data through the feature extraction network to obtain a second feature representation corresponding to the second data;

[0208] The determining module 1010 is further configured to match the second feature representation with the label set to obtain a second value distribution of the second feature representation corresponding to the label set as the second prediction result; and to determine the second loss value based on the difference between the second value distribution and the pseudo label.

[0209] In an optional embodiment, the determining module 1010 is further configured to determine a first quantity of the first data in the first sample dataset;

[0210] The determining module 1010 is further configured to determine a second quantity of the second data in the second sample dataset, wherein the second quantity is greater than the first quantity;

[0211] The determining module 1010 is further configured to obtain a predicted loss value based on the weighted processing result of the first loss value on the first weight corresponding to the first quantity and the weighted processing result of the second loss value on the second weight corresponding to the second quantity.

[0212] The determining module 1010 is further configured to train the first candidate prediction model based on the prediction loss value to obtain the data prediction model.

[0213] In an optional embodiment, the determining module 1010 is further configured to determine a first ratio of the first loss value to the first quantity as the average loss value of the first data in the first sample dataset.

[0214] The determining module 1010 is further configured to determine a second ratio of the second loss value to the second quantity, as the average loss value of the second data in the first sample dataset;

[0215] The determining module 1010 is further configured to determine the sum of the first ratio and the second ratio as the predicted loss value.

[0216] In an optional embodiment, such as Figure 11 As shown, the device further includes:

[0217] The acquisition module 1000 is further configured to acquire the first sample dataset;

[0218] The training module 1030 is used to train the candidate prediction model using the first sample dataset to obtain the pseudo-label prediction model.

[0219] Output module 1040 is used to input the second data into the pseudo-label prediction model and output pseudo-labels corresponding to the second data;

[0220] The storage module 1050 is used to store the second data labeled with the pseudo-label into a preset dataset to obtain the second sample dataset.

[0221] In an optional embodiment, such as Figure 11 As shown, the acquisition module 1000 is further configured to acquire a third loss value corresponding to the candidate prediction model during the process of training the candidate prediction model using the first sample dataset. The third loss value is used to represent the difference between the prediction result of the candidate prediction model and the sample label of the first data.

[0222] The determining module 1010 is further configured to train the first candidate prediction model based on the first loss value, the second loss value and the third loss value to obtain the data prediction model.

[0223] In an optional embodiment, such as Figure 11 As shown, the determining module 1010 is used to use the data prediction model as the updated pseudo-label prediction model to perform pseudo-label recognition on the second data and obtain the recognition result;

[0224] The training module 1030 is further configured to use the recognition result as the pseudo-label updated by the second data, and to iteratively train the first candidate prediction model using the second data and the updated pseudo-label.

[0225] In summary, the model training apparatus provided in this application decouples the processing of the first data labeled with sample labels from the processing of the second data labeled with pseudo-labels. The two processes share a single feature extraction network, and then use different candidate prediction models to predict their respective input data, obtaining a first prediction result for the first data and a second prediction result for the second data. Then, a first loss value determined based on the difference between the first prediction result and the sample labels, and a second loss value determined based on the difference between the second prediction result and the pseudo-labels, are used to train the first candidate prediction model. This allows the first candidate prediction model to learn the influence of pseudo-labels in the second data during training, improving the data prediction model's tolerance to biased pseudo-labels to some extent, thereby enhancing the robustness of the data prediction model.

[0226] Figure 12 This illustration shows a schematic diagram of a server provided in an exemplary embodiment of this application. The server may be as follows: Figure 3 The server shown here has the following structure.

[0227] Server 1200 includes a Central Processing Unit (CPU) 1201, a system memory 1204 including Random Access Memory (RAM) 1202 and Read Only Memory (ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the CPU 1201. Server 1200 also includes a mass storage device 1206 for storing an operating system 1213, application programs 1214, and other program modules 1215.

[0228] Mass storage device 1206 is connected to central processing unit 1201 via a mass storage controller (not shown) connected to system bus 1205. Mass storage device 1206 and its associated computer-readable media provide non-volatile storage for server 1200. That is, mass storage device 1206 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0229] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 1204 and mass storage device 1206 described above can be collectively referred to as memory.

[0230] According to various embodiments of this application, server 1200 can also be connected to a remote computer on the model via a model connection, such as the Internet. That is, server 1200 can be connected to network 1212 via network interface unit 1211 connected to system bus 1205, or network interface unit 1211 can be used to connect to other types of models or remote computer systems (not shown).

[0231] The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0232] Embodiments of this application also provide a computer device that can be implemented as follows: Figure 3The terminal or server shown. The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set, the at least one instruction, at least one program, code set, or instruction set being loaded and executed by the processor to implement the data prediction method provided in the above-described method embodiments.

[0233] Embodiments of this application also provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the data prediction method provided in the above-described method embodiments.

[0234] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data prediction method provided in the above-described method embodiments.

[0235] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments in this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0236] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0237] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A model training method, characterized in that, The method includes: Obtain a first sample dataset and a second sample dataset. The first sample dataset includes first data labeled with sample labels, and the second sample dataset includes second data labeled with pseudo labels. The sample labels are used to indicate the data content of the first data, and the pseudo labels are labels obtained by predicting the content of the second data through a pseudo label prediction model that has been pre-trained. The first data is first image data, and the second data is second image data. Determine the first quantity of the first data in the first sample dataset; Determine a second quantity of the second data in the second sample dataset, wherein the second quantity is greater than the first quantity; The first data is predicted by the first candidate prediction model, and a first loss value is determined based on the difference between the obtained first prediction result and the sample label. The first loss value includes the loss corresponding to the first number of first data. The second data is predicted by the second candidate prediction model, and a second loss value is determined based on the difference between the obtained second prediction result and the pseudo label; wherein the first candidate prediction model and the second candidate prediction model share a feature extraction network, and the second loss value includes the loss sum corresponding to the second number of second data; Determine a first ratio of the first loss value to the first quantity, and use it as the average loss value of the first data in the first sample dataset; Determine a second ratio of the second loss value to the second quantity, and use it as the average loss value of the second data in the second sample dataset; The sum of the first ratio and the second ratio is determined as the predicted loss value; The first candidate prediction model is trained based on the prediction loss value to obtain a data prediction model, which is used to predict the scene label corresponding to the input image data.

2. The method according to claim 1, characterized in that, The sample labels are used to represent the value distribution of the label set corresponding to the first data, and the pseudo labels are used to represent the value distribution of the label set corresponding to the second data. In the value distribution, the labels marked with the first value indicate that they are related to the data content, and the labels marked with the second value indicate that they are not related to the data content.

3. The method according to claim 2, characterized in that, The step of predicting the first data using a first candidate prediction model and determining a first loss value based on the difference between the obtained first prediction result and the sample label includes: The feature extraction network is used to extract features from the first data to obtain a first feature representation corresponding to the first data. The first feature representation is matched with the label set by the first prediction network in the first candidate prediction model to obtain the first value distribution of the label set corresponding to the first feature representation as the first prediction result; the first loss value is determined based on the difference between the first value distribution and the sample label. The step of predicting the second data using a second candidate prediction model and determining a second loss value based on the difference between the obtained second prediction result and the pseudo-label includes: The second data is subjected to feature extraction through the feature extraction network to obtain a second feature representation corresponding to the second data; The second feature representation is matched with the label set by the second prediction network in the second candidate prediction model to obtain the second value distribution of the label set corresponding to the second feature representation as the second prediction result; the second loss value is determined based on the difference between the second value distribution and the pseudo label.

4. The method according to any one of claims 1 to 3, characterized in that, The process of obtaining the first sample dataset and the second sample dataset includes: Obtain the first sample dataset; The pseudo-label prediction model is obtained by training the candidate prediction model using the first sample dataset; The second data is input into the pseudo-label prediction model, and the pseudo-label corresponding to the second data is output. The second data labeled with the pseudo-label is stored in a preset dataset to obtain the second sample dataset.

5. The method according to claim 4, characterized in that, The method further includes: During the training of the candidate prediction model using the first sample dataset, a third loss value corresponding to the candidate prediction model is obtained. The third loss value is used to represent the difference between the prediction result of the candidate prediction model and the sample label of the first data. The first candidate prediction model is trained based on the first loss value, the second loss value, and the third loss value to obtain the data prediction model.

6. The method according to any one of claims 1 to 3, characterized in that, After training the first candidate prediction model based on the predicted loss value to obtain the data prediction model, the process further includes: The data prediction model is used as the updated pseudo-label prediction model to perform pseudo-label recognition on the second data, and the recognition result is obtained. The identification result is used as the pseudo-label after updating the second data, and the second data and the updated pseudo-label are used to iteratively train the first candidate prediction model.

7. A model training device, characterized in that, The device includes: The acquisition module is used to acquire a first sample dataset and a second sample dataset. The first sample dataset includes first data labeled with sample labels, and the second sample dataset includes second data labeled with pseudo labels. The sample labels are used to indicate the data content of the first data, and the pseudo labels are labels obtained by predicting the content of the second data through a pseudo label prediction model that has been pre-trained. The first data is first image data, and the second data is second image data. A determination module is configured to: determine a first quantity of the first data in the first sample dataset; determine a second quantity of the second data in the second sample dataset, wherein the second quantity is greater than the first quantity; perform data prediction on the first data using a first candidate prediction model, and determine a first loss value based on the difference between the obtained first prediction result and the sample label, wherein the first loss value includes the sum of losses corresponding to the first quantity of the first data; perform data prediction on the second data using a second candidate prediction model, and determine a second loss value based on the difference between the obtained second prediction result and the pseudo label; wherein the first candidate prediction model and the second candidate prediction model share a feature extraction network, and the second loss value includes the sum of losses corresponding to the second quantity of the second data; determine a first ratio of the first loss value to the first quantity as the average loss value of the first data in the first sample dataset; determine a second ratio of the second loss value to the second quantity as the average loss value of the second data in the second sample dataset; and determine the sum of the first ratio and the second ratio as the prediction loss value. The training module is used to train the first candidate prediction model based on the prediction loss value to obtain a data prediction model, which is used to predict the scene label corresponding to the input image data.

8. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the model training method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is loaded and executed by a processor to implement the model training method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes a computer program or instructions that are executed by a processor to implement the model training method as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Method and device for training an event prediction model and carrying out event evaluation

    CN110659744A

  • Neural network training method and device

    CN113705769A