Data processing method, apparatus, device, computer program product, and storage medium

Abstract

This invention provides a data processing method, apparatus, device, computer program product, and computer-readable storage medium. The method considers that the preference data used in DPO training may have noisy labels that negatively impact training performance. It proposes a noise-aware metric to indicate the probability that the preference label of a pair of preference data is a noisy label, and incorporates this metric into the DPO training objective function. This incorporates the quality of the preference data into the DPO training process, thereby identifying and mitigating the impact of noisy preference data on DPO training performance. This method effectively identifies noisy data in the preference data used in DPO training and effectively reduces the impact of noisy data on training performance by introducing a noise-aware metric into the DPO objective function, thus improving DPO training performance. Furthermore, this method is widely applicable to various large models, including LLM and DM, to improve the DPO training performance of these large models.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more specifically, to a data processing method, apparatus, device, computer program product, and storage medium. Background Technology

[0002] In recent years, Direct Preference Optimization (DPO) has gradually attracted widespread attention from academia and industry as an effective method for aligning large models with human feedback in artificial intelligence research. The main purpose of DPO is to align the model with human feedback by modeling the output of the target model as an implicit representation of the reward indicating preference. Due to its straightforward formulation and robust performance, this method has demonstrated effectiveness and ease of use in multiple fields, especially in the training and application of Large Language Models (LLMs) and Diffusion Models (DMs).

[0003] Therefore, an effective data processing method is needed to improve the performance of DPO training. Summary of the Invention

[0004] Therefore, embodiments of this disclosure provide a data processing method, apparatus, device, computer program product, and computer-readable storage medium.

[0005] Embodiments of this disclosure provide a data processing method, comprising: for a set of preference data pairs used for direct preference optimization of a model to be trained, determining a noise-aware metric for each preference data pair in the set of preference data pairs using multiple fine-tuned models, wherein each preference data pair includes two preference data with preference labels, each preference data being text data or image data, and for each preference data pair, the noise-aware metric is used to indicate the probability that the preference label of the preference data pair is a noisy label; determining an objective function for direct preference optimization based on the determined noise-aware metric; and performing direct preference optimization on the model to be trained using the set of preference data pairs based on the determined objective function.

[0006] Embodiments of this disclosure provide a data processing apparatus, comprising: a noise measurement module configured to determine a noise-aware metric for each preference data pair in the preference data pair set for direct preference optimization of a model to be trained, using multiple fine-tuned models, wherein each preference data pair includes two preference data having preference labels, each preference data being text data or image data, and for each preference data pair, the noise-aware metric is used to indicate the probability that the preference label of the preference data pair is a noisy label; a target construction module configured to determine an objective function for direct preference optimization based on the determined noise-aware metric; and a model training module configured to perform direct preference optimization on the model to be trained using the preference data pair set based on the determined objective function.

[0007] Embodiments of this disclosure provide a data processing apparatus, including: one or more processors; and one or more memories, wherein the one or more memories store a computer-executable program, and when the processor executes the computer-executable program, it performs the data processing method as described above.

[0008] Embodiments of this disclosure provide a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, are used to implement the data processing method described above.

[0009] Embodiments of this disclosure provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a data processing method according to embodiments of this disclosure.

[0010] The method provided by embodiments of this disclosure considers that the preference data used in DPO training may have noisy labels that negatively impact training performance. It proposes a noise-aware metric to indicate the probability that the preference label of a pair of preference data is a noisy label, and introduces this metric into the DPO training objective function. This incorporates the quality of the preference data into the DPO training process, thereby identifying and mitigating the impact of noisy preference data on DPO training performance. The method of embodiments of this disclosure can effectively identify noisy data in the preference data used in DPO training and effectively reduce the impact of noisy data on training performance by introducing a noise-aware metric into the DPO objective function, thus improving DPO training performance. Furthermore, the method provided by embodiments of this disclosure is widely applicable to various large models, including LLM and DM, to improve the DPO training performance of these large models. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some exemplary embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0012] Figure 1 This is a diagram illustrating an example of preference data pairs for DPO training according to an embodiment of the present disclosure;

[0013] Figure 2 This is a flowchart illustrating a data processing method according to an embodiment of the present disclosure;

[0014] Figure 3 This is a graph illustrating the accuracy and consistency of annotation results from different human annotators according to embodiments of the present disclosure;

[0015] Figure 4 This is a graph illustrating the relationship between noise perception metrics and the noise rate of preference data according to embodiments of the present disclosure;

[0016] Figure 5 This is a diagram illustrating the image results generated by a first model trained by the data processing method according to embodiments of the present disclosure and other methods;

[0017] Figure 6 This is a diagram illustrating the image results generated by a second model trained by the data processing method according to embodiments of the present disclosure and other methods;

[0018] Figure 7 This is a diagram showing a comparison of the image results generated by the model obtained by the data processing method according to embodiments of the present disclosure, with and without optimization using a margin term;

[0019] Figure 8 This is a diagram illustrating the image results generated by the model obtained when the data processing method according to an embodiment of the present disclosure is applied to identity preference optimization (IPO);

[0020] Figure 9 This is a schematic diagram illustrating a data processing apparatus according to an embodiment of the present disclosure;

[0021] Figure 10 A schematic diagram of a data processing apparatus according to an embodiment of the present disclosure is shown; and

[0022] Figure 11 A schematic diagram of the architecture of an exemplary computing device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0024] In this specification and accompanying drawings, steps and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these steps and elements are omitted. Furthermore, in the description of this disclosure, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0025] In embodiments of this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0027] To facilitate the description of this disclosure, the following concepts related to this disclosure are introduced.

[0028] The data processing method disclosed herein can be based on artificial intelligence (AI). Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine capable of reacting in a manner similar to human intelligence.

[0029] The data processing method disclosed herein can be based on Direct Preference Optimization (DPO). DPO is a reinforcement learning optimization framework designed to improve the decision-making and generative capabilities of a model by directly utilizing human preference data. The core of DPO is training the model using human evaluators' preferences for different outputs. These preferences can be comparative (e.g., given two outputs, choose the better one) or rating (e.g., given an output, rate its quality). By collecting a large amount of human preference data, the DPO mechanism can learn a model capable of predicting the relative popularity of generated outputs among users. This approach is primarily applied to generative models, such as text generation and dialogue systems in natural language processing, with the aim of generating content that better aligns with human expectations and preferences.

[0030] The data processing method disclosed herein can be based on Reinforcement Learning from Human Feedback (RLHF). RLHF refers to using feedback information from human experts to enhance the model's performance. During the alignment process, the quality of the model-generated text can be improved by collecting evaluations of the model's generated text from human experts and then using this evaluation information to guide the model's training. For example, in DPO training, preference data pairs with human-annotated preference labels can be collected, and the model can be trained based on these preference data pairs, making the model generate content that better meets user expectations.

[0031] In summary, the solutions provided by the embodiments of this disclosure involve technologies such as artificial intelligence, DPO, and RLHF. The embodiments of this disclosure will be further described below with reference to the accompanying drawings.

[0032] As mentioned above, DPO (Depth-Point Object Modeling) serves as an effective method for aligning large models with human feedback, achieving a fit between the model and human input. However, despite the successful application of DPO to various large models, there is little discussion regarding the quality of the preference data used in DPO training, which is largely ignored in the process. Preference data used for DPO training is primarily labeled in two ways: automatic labeling based on deep learning metrics, such as using deep learning models like PickScore (i.e., machine labelers); and manual labeling, such as through crowdsourcing by human labelers. However, both labeling methods can produce noisy data. Specifically, machine labelers often produce biased labels due to their limited generalization ability, while human labelers inherently provide subjective evaluations of preferences, such as judgments of subjective visual attributes like "which image is better." Therefore, the quality of the preference dataset can be significantly reduced due to labeler negligence or deliberate misleading. Furthermore, due to the lack of standardized judgment criteria, unintentional human errors may exist within the preference dataset. Therefore, existing preference datasets inherently contain noisy data, meaning this noisy data has noisy labels (i.e., labels incorrectly labeled by the annotator). This noisy data can negatively impact the learning of relative attributes. For example, in some cases, users may not have clear preferences for certain attributes (such as color, shape, etc.), and the DPO attempts to summarize basic rules from these uncertain preferences, leading to the challenge of understanding clear preferences.

[0033] Figure 1 This is a diagram illustrating an example of preference data pairs for DPO training according to embodiments of the present disclosure. In embodiments of the present disclosure, image pairs are described as examples of preference data pairs; however, it should be understood that DPO training can also use preference data pairs such as text sequence pairs, which can be determined based on the specific generation task of the model, and the present disclosure does not limit this.

[0034] like Figure 1 As shown, in Figure 1 In (a), a pair of images generated based on the same prompt instruction (e.g., “A giant waffle cone melts and forms a river flowing through the city”) is shown, where both the left and right images depict a giant waffle cone and the scene described in the prompt instruction, but with different image styles. Similarly, in Figure 1 In (b), a pair of images generated based on the same prompt (e.g., “full body of an oriental dragon man in European armor”) is shown, wherein the left and right images both depict the scene described in the prompt and have different image styles.

[0035] For the image pairs mentioned above, due to the ambiguity of the preference criteria, annotators may find it difficult to make clear preferences between samples, and different annotators may have different preferences for the same image pair, resulting in noise in the preference labels for the preference data pairs.

[0036] Ideally, DPO training aims to capture these complex preference relationships and learn the rules behind them. However, if the labels in the preference data contain significant biases or noise, this will directly affect the accuracy of the ultimately learned rules, causing them to deviate from the expected results and even leading to serious consequences.

[0037] Therefore, in this disclosure, the inventors have found that a key issue that needs to be clarified is whether DPO training is really affected by noisy data in the preferred dataset, and if so, how to reduce or even eliminate the negative impact of noisy data on training.

[0038] In response, this disclosure presents a comprehensive study. Specifically, the disclosure will first verify through preliminary experiments that noisy data exists in the preference dataset and that this noisy data negatively impacts DPO training performance. Secondly, the disclosure will further propose a data processing method to reduce or even eliminate the negative impact of this noisy data on DPO training. Finally, the disclosure will experimentally verify the beneficial effects of the data processing method of this disclosure on DPO training.

[0039] Noisy label learning, a key research topic in machine learning, explores how inaccurate labels, inevitably present in training data, affect model performance. The introduction of noise is a pervasive problem, regardless of whether it relies on human or machine labelers. This phenomenon has attracted increasing attention in recent years, particularly regarding biased data used to train deep learning models. Therefore, this publication addresses the crucial role of understanding and resolving the noisy labeling problem in improving model reliability and performance.

[0040] First, we need to examine the presence of noisy data in the preference dataset, i.e., whether noise exists within the preference data. Existing preference data typically comes from two main sources: human annotators and machine annotators. Machine annotators, for example, use metrics such as PickScore. While these models report reliable performance metrics in original studies, they generally do not achieve 100% accuracy. This means that even in test samples within the same domain, these annotators inevitably introduce noisy labels. This noise problem becomes even more severe when considering data from outside the domain.

[0041] On the other hand, for data annotated by human annotators, this disclosure conducted a simple experiment to verify the existence of noisy labels. For example, the experiment randomly sampled 1000 pairs of images from an existing image preference dataset and labeled them by asking multiple human annotators "which image is better". These human annotators could choose a winning image from each image pair or record that neither was better.

[0042] Figure 2 This is a graph illustrating the accuracy and consistency of annotation results from different human annotators according to embodiments of the present disclosure.

[0043] As a non-restrictive example, Figure 2 The results of annotation by three human annotators are shown, where the annotation accuracy of each annotator was calculated against the original labels of these image pairs, and the mean standard deviation among the annotators' predictions was also calculated. Figure 2 As shown in the accuracy data of the individual annotators, there is a significant gap between these annotators and the baseline. This means that relying on crowdsourced annotators is not always a reliable and effective strategy, and it can directly lead to noisy labels in the preferred data. Furthermore, as indicated by the mean standard deviation, there is also significant inconsistency among the predictions of the three human annotators. In other words, the effectiveness of relying on naive solutions such as voting strategies to solve data labeling problems is greatly reduced.

[0044] Therefore, after verifying the existence of noisy data in the preference dataset, another key issue to consider is whether DPO training will be affected by this noisy data. The reason for investigating this issue is that if the DPO algorithm is robust to this noisy data, it can be trained directly using the current preference dataset without requiring additional processing for this noisy data.

[0045] To this end, this disclosure conducts an in-depth study of preference datasets based on diffusion DPO. Since noisy data is difficult to detect, in embodiments of this disclosure, different degrees of noise in the preference dataset can be simulated by manually adjusting the labels of some original preference data in the preference dataset (e.g., swapping the labels of preference data pairs).

[0046] Optionally, the labels of 10%, 20%, and 30% of the preference data in the preference dataset can be randomly changed (i.e., the noise rates are 10%, 20%, and 30%, respectively) to observe how the preference dataset with different noise levels affects the training effect and to record multiple performance metrics such as ImageReward, PickScore, Aesthetic rating, and Human Preference Score (HPS).

[0047] Alternatively, different models can be selected and trained using the aforementioned preference datasets with different noise levels to determine the impact of different noise levels on the training performance of DPO. For example, in this disclosure, by way of example and not limitation, variants of the Diffusion Model, SD1.5 and SDXL, can be used as models to be trained.

[0048] Table 1 below shows the performance of models trained with DPO at different noise levels. For each performance metric, a larger value corresponds to better performance.

[0049] Table 1. Performance of models trained via DPO at different noise levels.

[0050]

[0051] As shown in Table 1, with increasing noise rate, the proportion of noisy labels increases, leading to a significant decrease in training performance and ultimately severe degradation of model performance. Specifically, for the SD1.5 model, negative image rewards even occurred at a noise rate of 30%. This confirms the sensitivity of DPO training to noisy labels in the training data. Furthermore, it is worth noting that the manually created noisy labels described above are fundamentally different from noisy labels in real-world scenarios. Although the randomly selected noisy data in this disclosure forms a relatively uniform distribution in a statistical sense, the noise distribution in real-world scenarios is usually affected by various factors, such as the complexity of the labeling task, data quality, and the subjective judgment of the labelers. Therefore, it is necessary to design a data processing method that is robust to both types of noise (i.e., artificially synthesized noise and natural noise).

[0052] Based on this, this disclosure proposes a data processing method for solving the above problems.

[0053] Specifically, this disclosure considers that the preference data used in DPO training may have noisy labels that negatively impact training performance. It proposes a noise-aware metric to indicate the probability that a preference data pair has a noisy label, and incorporates this metric into the DPO training objective function. This incorporates the quality of the preference data into the DPO training process, thereby identifying and mitigating the impact of noisy preference data on DPO training performance. This method effectively identifies noisy data in the preference data used in DPO training and effectively reduces the impact of noisy data on training performance by introducing a noise-aware metric into the DPO objective function, thus improving DPO training performance. Furthermore, this method is broadly applicable to various large models, including LLM and DM, to improve the DPO training performance of these large models.

[0054] The following will refer to Figures 3-4 The data processing methods disclosed herein will be described in detail.

[0055] Figure 3 This is a flowchart illustrating a data processing method 300 according to an embodiment of the present disclosure.

[0056] In step S301, for the set of preference data pairs used for direct preference optimization of the model to be trained, a noise-aware metric for each preference data pair in the set of preference data pairs can be determined using multiple fine-tuned models, wherein each preference data pair includes two preference data with preference labels, each preference data being text data or image data, and for each preference data pair, the noise-aware metric is used to indicate the probability that the preference label of the preference data pair is a noisy label.

[0057] For the DPO training process, the supervision information comes from the preference labels; that is, the preference labels of each pair of preference data are used as supervision information for DPO training of the model to be trained. As mentioned above, the labeling process of preference data is not always reliable, which may lead to unexpected noisy labels, thus significantly affecting the efficiency and effectiveness of DPO training.

[0058] To better understand the above problems, in the embodiments of this disclosure, preference labels can be used as labels for binary classification problems, where preferred samples are labeled as the first class and non-preferred samples are labeled as the second class. Based on this, assuming that the noise in the training data will not reach a level that prevents the model from learning appropriate rules, training a binary classifier with such data will result in the following three possible phenomena caused by noisy labels:

[0059] ① A data pair is mislabeled but cannot provide actual supervision for the model to be trained. In this case, the model can learn to predict the true preferences. Therefore, the gap (i.e., bias) between the model's prediction of this data pair and the given label will remain significant as the training process continues.

[0060] ② A data pair is mislabeled, and the model easily learns the error information. In this case, because the model learns the error information from the mislabeled data pair during training, it is almost impossible to effectively detect the error in the data pair.

[0061] ③ If the mislabeled data pairs do not affect the training process, then multiple models trained with different randomness (e.g., various out-of-order sequences of a batch of data pairs) will produce highly unstable predictions about these data pairs. In other words, the variance of these models' predictions about the same mislabeled data pair will be high.

[0062] Based on the three possible phenomena described above, this disclosure finds that if the aforementioned bias and variance can be effectively measured, noisy samples from phenomena ① and ③ can be effectively processed. Therefore, the aforementioned problem of this disclosure can be transformed into how to instantiate these two measures into concrete measures.

[0063] One approach is to directly re-annotate a portion of the noisy dataset and train a binary classifier based on this data. In this way, both metrics can be defined based on the classifier's output. However, re-annotating biased data by human annotators is resource-intensive, making it nearly impractical for large datasets. Furthermore, such an annotation process introduces new noise. Additionally, for different input modalities such as images and text, the binary classifier would need to be specifically designed for these different input modalities, introducing additional complexity.

[0064] To address this issue, this disclosure will analyze the formula for the objective function of DPO training to explore instantiation of the aforementioned metric.

[0065] Specifically, consider RLHF, which fine-tunes a large model by aligning it with human preferences through preference optimization. This involves the following two stages:

[0066] 1) Reward Modeling: It applies the Bradley-Terry (BT) model to extract rewards from pairwise data. Learning in China, among which, Indicates the preferred answer. This indicates the second-best answer. This indicates the prompt and context. Therefore, the loss function of the reward model can be expressed as follows:

[0067] (1)

[0068] in, These represent the parameters of the reward model. Represents a preference dataset, Indicates to All of them Calculate the expectation. This represents the sigmoid function, whose output value is between 0 and 1, and is used to convert evaluation values ​​into probabilities.

[0069] 2) RL Fine-tuning: It uses the learned reward model to provide feedback to the large model and applies the Proximal Policy Optimization (PPO) algorithm to fine-tune the model, as follows:

[0070] (2)

[0071] in, The strategy for representing the model, Indicates the strategy parameters, Indicates the reference strategy, express.

[0072] Both DPO and Diffusion DPO are trained based on pairs of human preferences, without the need to train a reward model.

[0073] For example, for DPO, equation (2) above can produce an explicit solution as follows:

[0074] (3)

[0075] in, This represents the adjusted policy distribution. This means:

[0076] (4)

[0077] Therefore, substituting equation (4) into equation (1), we can derive the following DPO optimization objective function:

[0078] (5)

[0079] By minimizing the objective function described above, the model can generate better output given the context. This increases the probability of generating poor output while reducing the likelihood of producing bad output. The probability of . Wherein, the reference strategy As a benchmark, it can provide a standard for feedback on learning quality, thereby helping the model to gradually improve the quality of the generated content.

[0080] For diffusion DPO, its objective function can be similarly expressed as:

[0081] (6)

[0082] in Represents a noise prediction network. This indicates the time step for noise reduction.

[0083] Therefore, for the DPO optimization objective function described above, this disclosure finds that the objective function shown in equation (5) can be rewritten as:

[0084]

[0085] in, This indicates the model's output given the context c. Relative to the generated output The relative probability logarithm measures the relative confidence of a generative model in generating a specific output given a context. This represents the output of the reference model, which, under the same context, measures the confidence that the reference model will produce the same output.

[0086] Furthermore, a similar rewrite can be achieved for the objective function of diffusion DPO as shown in equation (6).

[0087] By minimizing the above The model is actually guided to maximize This mechanism is similar to optimization with binary hinge loss. Therefore, the DPO training process can actually be understood as implicitly guiding the model to be trained to better perform preference classification as a binary classifier.

[0088] Based on this, in embodiments of this disclosure, the aforementioned bias and variance can be instantiated using predictions from multiple fine-tuned models to construct a noise-aware metric indicating the probability of noise, rather than training an additional classifier. These fine-tuned models can be models with the same or different backbone models as the model to be trained.

[0089] According to embodiments of this disclosure, the noise perception metric may include an annotator-built-in reliability term and an annotator-interaction stability term. The annotator-built-in reliability term is used to evaluate the difficulty of the plurality of fine-tuned models in predicting the labels for the preferred data pair, and the annotator-interaction stability term is used to evaluate the stability of the plurality of fine-tuned models in predicting the labels for the preferred data pair.

[0090] Optionally, the aforementioned bias and variance can be instantiated as the annotator-built-in reliability term and the annotator-inter-stability term of this disclosure, respectively. The annotator-built-in reliability term describes the bias between the label predictions of multiple fine-tuned models for the preferred data pair and the label prediction of the reference model; a larger annotator-built-in reliability term corresponds to a larger bias. The annotator-inter-stability term describes the consistency among the label predictions of multiple fine-tuned models for the preferred data pair.

[0091] According to embodiments of this disclosure, determining a noise-perceived metric for each preference data pair in the set of preference data pairs using multiple fine-tuned models may include: for each preference data pair: for each fine-tuned model among the multiple fine-tuned models, determining the confidence level of the label prediction result of the fine-tuned model for the preference data pair, wherein the confidence level of the label prediction result of the fine-tuned model for the preference data pair is expressed as the difference between the label prediction result of the fine-tuned model for the preference data pair and the label prediction result of a predetermined reference model for the preference data pair; determining an annotator-built-in reliability term and an annotator-to-annotator stability term for the preference data pair based on the statistics of the confidence levels of the label prediction results of the multiple fine-tuned models for the preference data pair; constructing the noise-perceived metric based on the annotator-built-in reliability term and the annotator-to-annotator stability term for the preference data pair, wherein the noise-perceived metric is positively correlated with the annotator-built-in reliability term and the annotator-to-annotator stability term.

[0092] Optionally, based on the previous analysis, a quantity can first be defined to measure the difference between the model to be trained and the reference model. Specifically, given a pair of preference data:

[0093] (10)

[0094] in, This can represent the confidence level of the predictions made by the model being trained, because the difference can be seen as the current model generating the same output relative to the reference model. The higher the confidence level, the better the model generates the optimal result. The stronger one's confidence, the higher one's level of confidence.

[0095] According to embodiments of this disclosure, determining the annotator-built-in reliability term and the annotator-inter-annotator stability term of the preference data pair based on the statistical measure of the confidence levels of the label prediction results of the preference data pair by the plurality of fine-tuned models includes: determining the annotator-built-in reliability term of the preference data pair based on the statistical average of the confidence levels of the label prediction results of the preference data pair by the plurality of fine-tuned models, wherein the annotator-built-in reliability term is negatively correlated with the result of the statistical average, and preference data pairs with larger annotator-built-in reliability terms correspond to larger label prediction biases.

[0096] In the embodiments of this disclosure, assuming there are M fine-tuned models, an annotator-built-in confidence term can be defined. as follows:

[0097] (11)

[0098] in, This indicates that the m-th fine-tuned model is for the preference data pair The confidence level of the prediction, where ρ represents the weighting parameter. Therefore, This can correspond to M fine-tuned models for preference data pairs. The larger the deviation of the prediction, the greater the deviation between the prediction of each fine-tuned model for the preference data pair and the prediction of the reference model. Therefore, the probability that the preference data pair contains noise may be higher.

[0099] According to embodiments of this disclosure, determining the annotator-built-in confidence term and the annotator-intermediate stability term of the preferred data pair based on the statistics of the confidence levels of the label prediction results of the multiple fine-tuned models for the preferred data pair includes: determining the annotator-intermediate stability term of the preferred data pair based on the variance between the confidence levels of the label prediction results of the multiple fine-tuned models for the preferred data pair, wherein a preferred data pair with a larger annotator-intermediate stability term corresponds to a larger label prediction instability.

[0100] Alternatively, variance can be instantiated as follows:

[0101] (12)

[0102] Therefore, equation (12) above can describe the M fine-tuned models for preference data pairs. The stability of the prediction is indicated by the value of the model. The larger the value, the greater the fluctuation in the prediction of each fine-tuned model for the preference data pair. Therefore, the probability that the preference data pair contains noise may be higher.

[0103] Based on the above instantiated annotator built-in reliability items Stability between annotators This allows the construction of the noise perception metric disclosed herein.

[0104] Optionally, the constructed noise perception metric can be positively correlated with both the annotator-built-in reliability term and the annotator-to-annotator stability term, since larger bias and larger variance can indicate greater noise.

[0105] According to embodiments of this disclosure, constructing the noise perception metric based on the annotator-built-in reliability term and the inter-annotator stability term of the preference data pair includes: constructing the noise perception metric based on the product of the annotator-built-in reliability term and the inter-annotator stability term of the preference data pair.

[0106] Alternatively, as an example in this disclosure, the built-in annotator reliability term and the inter-annotator stability term can be directly combined by product to obtain the noise-perceived metric. A simple form is as follows:

[0107] (13)

[0108] Regarding the above noise perception metrics , The larger the value, the more positive the preference data is. The higher the probability that the corresponding label is an incorrectly labeled label. Specifically, for the above noise perception metric... When its value is large, the annotator's built-in reliability item Stability between annotators One or both of these will be larger; that is, a larger noise perception metric may indicate the detection of a larger bias or a larger variance. Conversely, for samples with smaller bias and variance, this... The value is usually smaller, and such samples can be considered clean samples during training.

[0109] Therefore, based on the above noise perception metrics It can quantitatively describe the probability that each preference data pair in the preference dataset is noisy, so that when performing DPO, the labeling quality of each preference data pair can be combined for model optimization, so that model training can be free from the influence of noisy data.

[0110] To verify the effectiveness of the noise-perceived metric proposed in this disclosure, experiments were conducted using artificially synthesized noise in the embodiments of this disclosure. Specifically, noise can be added to the preference data by manually adjusting the labels of a portion of the original preference data in the preference dataset (e.g., swapping the labels of preference data pairs), as described above, i.e., synthesizing noisy data. Furthermore, the ratio of altered samples can be visualized using different noise-perceived metric values. Figure 4 This is a graph illustrating the relationship between noise perception metrics and the noise rate of preference data according to embodiments of the present disclosure.

[0111] like Figure 4 As shown, it can be observed that as the noise perception metric increases, the ratio of noisy samples (noise rate) also increases accordingly, thus demonstrating the effectiveness of the noise perception metric in identifying noisy samples.

[0112] Next, return to Figure 3 In step S302, an objective function for direct preference optimization can be determined based on the determined noise perception metric.

[0113] Optionally, after constructing the noise-aware metric of this disclosure, it can be used to enhance the DPO training process. Specifically, the noise-aware metric can be introduced into the objective function of DPO to incorporate consideration of the quality of the preferred data during model training, thereby reducing or even eliminating the negative impact of noisy data on training.

[0114] According to embodiments of this disclosure, the objective function is designed to reduce the impact of the preference data on the pair of preference data with noisy labels in the set on the direct preference optimization, and to increase the impact of the preference data on the pair of preference data without noisy labels in the set on the direct preference optimization.

[0115] Optionally, in the model training of this disclosure, based on the identification of noisy samples, the following two principles can be followed: (1) reduce the influence of noisy data; (2) increase the influence of clean data. That is, in the design of the objective function, the contribution of noisy data to model training can be reduced as much as possible, and the supervision of training by clean data can be enhanced.

[0116] Alternatively, taking into account a larger noise perception metric The value can indicate that the preference data is more likely to be noisy data, that is, the label of the preference data is incorrectly labeled. In the embodiments of this disclosure, optimization terms corresponding to the two principles mentioned above can be introduced into the objective function based on the noise perception metric.

[0117] According to embodiments of this disclosure, determining an objective function for direct preference optimization based on a determined noise-aware metric includes: for each preference data pair in the set of preference data pairs, determining a weighting coefficient corresponding to the preference data pair based on the noise-aware metric, wherein the weighting coefficient is used to weight the label prediction results of the model to be trained for the preference data pair in the objective function, and preference data pairs with larger noise-aware metrics correspond to smaller weighting coefficients.

[0118] Optionally, to reduce the impact of noisy data on DPO training, a weighting coefficient negatively correlated with the noise perception metric can be constructed to weight the prediction results of the model to be trained. In other words, when the noise perception metric is large, the prediction results can be assigned lower weights to weaken the supervision from that sample on model training.

[0119] According to embodiments of this disclosure, determining the objective function for direct preference optimization based on the determined noise-aware metric includes: for each preference data pair in the set of preference data pairs, determining the prediction margin of the label prediction result of the model to be trained for the preference data pair in the objective function based on the noise-aware metric, wherein preference data pairs with smaller noise-aware metrics correspond to larger prediction margins.

[0120] Optionally, to increase the impact of clean data on DPO training, a margin term negatively correlated with the noise-aware metric can be constructed as a supplement to the predictions of the model to be trained, encouraging the model to produce more reliable predictions about clean samples. In other words, when the noise-aware metric is small, a larger margin can be added to the predictions of the model to be trained, encouraging the model to prioritize learning from clean samples and to better utilize reliable annotations.

[0121] As an example, and not a limitation, the above weighting coefficients, prediction margin, and final DPO objective function can be expressed as follows:

[0122]

[0123] Among them, coefficient >0, For data pairs that are more likely to contain noise, It tends to be larger. Therefore, the weighting coefficient and forecast margin It tends to be smaller.

[0124] On the one hand, when When the value is large, it can indicate that preference data pairs may be mislabeled, in which case its corresponding The sample size will be relatively small, thus weakening the supervision from that sample.

[0125] On the other hand, the prediction margin term can improve the model's generalization ability. Specifically, the margin introduced in this disclosure... It is based on noise perception measurement Adaptive. When At that time, smaller Will be accompanied by a large Therefore, it will produce a large In other words, this can encourage models to produce more reliable predictions about clean samples. Conversely, for noisy samples, The margin is relatively large, and the supervision caused by the prediction margin will be negligible; therefore, the objective function will be determined by the weighting coefficients. control.

[0126] To further understand the objective function of this disclosure, the following equation, combined with the gradient, provides some explanation. The gradient of the above objective function has the following form:

[0127]

[0128] Specifically, for data pairs that are more likely to contain noise, Tend to be larger, and The tendency is towards smaller, therefore, the first item The weight of this data pair will be reduced to mitigate its potential negative impact on training. Furthermore, it can be observed that if it is possible to determine... and The appropriate value, then Some parts can be suppressed accordingly, which reduces the role of this data in training.

[0129] In step S303, the model to be trained can be directly optimized using the preference data set based on the determined objective function.

[0130] Based on the target loss function established in step S302, the model to be trained can be trained by DPO by optimizing the loss function.

[0131] As described above, the method provided by the embodiments of this disclosure takes into account that the preference data used in DPO training may have noisy labels that negatively impact training performance. It proposes a noise-aware metric to indicate the probability that the preference label of a pair of preference data is a noisy label, and introduces this noise-aware metric into the DPO training objective function. This incorporates the quality of the preference data into the DPO training process, thereby identifying and mitigating the impact of noisy preference data on DPO training performance. The method of the embodiments of this disclosure can effectively identify noisy data in the preference data used in DPO training and effectively reduce the impact of noisy data on training performance by introducing a noise-aware metric into the DPO objective function, thereby improving DPO training performance. Furthermore, the method provided by the embodiments of this disclosure is widely applicable to various large models, including LLM and DM, to improve the DPO training performance of these large models.

[0132] To verify the effectiveness of the data processing method proposed in this disclosure, experiments were conducted on LLM and DM.

[0133] Specifically, to demonstrate the generalization capability of the data processing method of this disclosure, experiments can be conducted on text and image generation tasks. For example, the generation tasks of this disclosure may include text-to-image generation tasks and text-to-dialogue tasks.

[0134] Optionally, the following experiments of this disclosure may include two main parts.

[0135] First, experiments can be conducted on datasets containing varying proportions of synthetically labeled noisy data to verify that the noise-aware metric disclosed herein is indeed effective for noisy data. For this part, a simple focus can be placed on the text-to-image generation task, and the SD1.5 model can be employed.

[0136] Then, the method of this disclosure can be validated on clean data without artificially labeled noisy data to demonstrate the problem of noisy data in existing preference datasets and to verify the significant practical value of the method of this disclosure in real-world alignment applications. For this part, text-to-image generation tasks and single-turn dialogue tasks can be considered, where SD1.5 and SDXL models can be used for training. The aforementioned PickScore, image reward, Aesthetic, and HPS can be used as evaluation metrics for text-to-image generation tasks. For single-turn dialogue, open-source models (e.g., Qwen2.5-72b, etc.) can be used to evaluate the generated content. Based on this, the performance superiority of each model trained using the method of this disclosure relative to models not trained using the method of this disclosure can be determined.

[0137] Optionally, a simple experiment based on synthetic noise can be conducted. Specifically, similar to the operation described above with reference to Table 1, a specific noise rate can be given, and a subset of training samples can be randomly selected to change their corresponding preference labels. These data can then be used to fine-tune the pre-trained SD1.5 according to the method of this disclosure. As shown in Table 2 below, using the adaptive DPO method of this disclosure, the fine-tuned model exhibits significantly stronger performance at the same noise level than the model fine-tuned using the original DPO. Even compared to a model fine-tuned with real data and without synthetic noise, the model trained with the adaptive DPO of this disclosure still performs better at different noise levels. Furthermore, as the noise rate increases, the method of this disclosure can, to some extent, combat the performance degradation caused by more noisy data. This reflects the effectiveness of the method of this disclosure in combating such synthetic noise.

[0138] Table 2. Training results of models using synthetic noisy data in text-to-image generation tasks.

[0139]

[0140] The aforementioned performance optimization can be attributed to the model's learning process. Typically, during training, the model tends to first learn information from clean labels to continuously improve the predictive power of its implicit reward model, as shown in Equation (4) above. In this way, as the model's performance optimizes, the performance in Equation (11)... Noise-aware metrics will become more reliable and accurate during training, thus improving their reliability. Consequently, noise-aware metrics can function better and help models eliminate the negative impact of noisy data.

[0141] As stated above, the experiments based on synthetic noise cannot fully represent the true value of the method disclosed herein, because the distribution of noisy data generated during the actual annotation process may differ significantly from the distribution of synthetic noise. Therefore, in this disclosure, the proposed adaptive DPO can be used to directly fine-tune the pre-trained model based on these benchmark datasets.

[0142] Tables 3 and 4 show the training results for DM and LLM. To make the training results more intuitive and easier to understand, the win rates of the method disclosed herein relative to the pre-trained model and the model fine-tuned by DPO are presented, respectively.

[0143] Table 3. Win rate of the method disclosed herein relative to benchmark methods on real-world data for text-to-image generation tasks.

[0144]

[0145] Table 4. Win rate of the disclosed method relative to benchmark methods based on real data for a single-turn dialogue.

[0146]

[0147] It can be seen that even without synthetic noise, the model obtained by applying the method of this disclosure still outperforms the original DPO in all these different settings. Specifically, for SD1.5, the method of this disclosure outperforms the original DPO in ImageReward in 66% of the generated results, and also shows consistent results in other settings. These results further validate the present disclosure's view that real data contains noisy samples. Secondly, it can also be shown that the method of this disclosure is effective not only against synthetic noise but also against real noise, demonstrating the generalization ability of the method of this disclosure.

[0148] To more intuitively compare the performance of this disclosure with other methods in generation tasks, this disclosure also... Figure 5 and Figure 6 The image results generated by different methods based on the same backbone model are visualized. Figure 5 This is a diagram illustrating the image results generated by a first model trained by the data processing method according to embodiments of the present disclosure and other methods. Figure 6 This is a diagram illustrating the image results generated by a second model trained using the data processing method according to embodiments of the present disclosure and other methods. Wherein, Figure 5 and Figure 6 Using SD1.5 and SDXL as the backbone models respectively, we fine-tuned them using different methods and generated corresponding image outputs based on the fine-tuned models.

[0149] like Figure 5 and Figure 6As shown, the images generated by the method of this disclosure are generally of higher quality than those generated by the model trained through the original DPO, which is consistent with the quantitative results mentioned above. Specifically, for the SD1.5 model, as... Figure 5 As shown, the model fine-tuned using the method of this disclosure can generate more detail for the human body and background. For the SDXL model, as... Figure 6 As shown, this disclosure finds that pre-trained models and models fine-tuned with the original DPO tend to generate mislocalized limbs, which degrades image quality. The method disclosed herein can help mitigate this problem, thereby producing more detailed portraits.

[0150] Furthermore, to further verify the effectiveness of the method disclosed herein, a series of ablation studies can be conducted regarding the design of the objective function and several hyperparameters of this disclosure. For simplicity, the results of a text-to-image generation task using SD1.5 as the backbone model are presented below.

[0151] First, in order to analyze the design of the objective function proposed in this disclosure, the performance of models with and without prediction margin terms can be compared. Figure 7 This is a diagram comparing the image results generated by the model obtained by the data processing method according to embodiments of the present disclosure with and without using a margin term for optimization.

[0152] like Figure 7 As shown, a significant improvement in image quality can be observed with the prediction margin term. For example, as shown in the third image pair, the image generated by the method with the prediction margin has more face detail compared to the image generated by the method without the prediction margin.

[0153] Furthermore, in the method described above, there are some key hyperparameters; therefore, this disclosure also conducts sensitivity studies on these hyperparameters. Table 5 below shows the results, which demonstrate that the method of this disclosure is relatively robust to changes in hyperparameters. Specifically, as... Increasing from 4 to 10 increases ImageReward by 0.02, while decreasing PickScore. When the value is greater than 10, the performance tends to saturate, which may be attributed to overfitting. In addition to the parameters mentioned above, in equation (12)... It also affects the training results. A larger [percentage] This can make the difference between a given implicit label between noisy and noisy data larger, thus achieving more significant improvements in training.

[0154] Table 5 Ablation studies on hyperparameter values

[0155]

[0156] Finally, to verify the adaptability of the method disclosed herein to other domains, IPO is used as an example to present the results of extending the method of this disclosure to IPO training (i.e., adaptive IPO). Table 6 below shows a comparison of the win rates using adaptive IPO with pre-trained and benchmark IPO.

[0157] Table 6 Win Rate of Adaptive IPO Relative to Benchmark Method

[0158]

[0159] As shown in Table 6, even compared to the benchmark IPO, the method of this disclosure can still achieve a significant improvement in IPO training. This further demonstrates that the method of this disclosure is widely applicable to solving noisy data problems in DPO, IPO, and various other optimization training.

[0160] Figure 8 This is a diagram illustrating the image results generated by a model obtained when the data processing method according to embodiments of the present disclosure is applied to identity preference optimization (IPO). Figure 8 As shown, the image generated by the model obtained through the adaptive IPO of this disclosure has more image details and higher quality than the image generated by the model obtained through the original IPO.

[0161] As can be seen, the adaptive DPO of this disclosure not only demonstrates effectiveness on artificially introduced noisy labels, achieving high fine-tuning performance even at noise levels as high as 30%, but also shows significant improvement when applied to real clean data, effectively addressing the noise problem in naturally labeled datasets. Furthermore, the adaptive DPO of this disclosure can be widely applied to various models and derived methods, such as IPO.

[0162] Therefore, this disclosure investigates the problem of noisy labels in preference datasets, identifying their detrimental impact on model performance during DPO training. Preliminary experiments using a diffusion model demonstrate the negative impact of this noise, highlighting the need for an effective solution. Based on this, a novel noise-aware metric is proposed, incorporating two components—annotator-built-in reliability and annotator-inter-annotator stability—to evaluate and identify potentially noisy data, thereby informing the training process. Next, this disclosure proposes an adaptive DPO loss function that reweights the training objective based on the noise-aware metric and introduces a margin term designed to reduce the impact of noisy samples and encourage the model to prioritize learning from high-quality, reliable annotations. Extensive experiments demonstrate that the method of this disclosure effectively mitigates both synthetic and natural noisy preference data, outperforming the original DPO in both image and text generation.

[0163] Figure 9This is a schematic diagram illustrating a data processing apparatus 900 according to an embodiment of the present disclosure.

[0164] According to embodiments of the present disclosure, the data processing device 900 may include a noise measurement module 901, a target construction module 902, and a model training module 903.

[0165] The noise measurement module 901 can be configured to determine a noise-aware metric for each preference data pair in the set of preference data pairs used for direct preference optimization of the model to be trained, using multiple fine-tuned models. Each preference data pair includes two preference data points with preference labels, each preference data point being either text data or image data. For each preference data pair, the noise-aware metric indicates the probability that the preference label of the preference data pair is a noisy label. Optionally, the noise measurement module 901 can perform the operations described above with reference to step S201, which will not be repeated here.

[0166] The target construction module 902 can be configured to determine an objective function for direct preference optimization based on a determined noise-aware metric. Optionally, the target construction module 902 can perform the operations described above with reference to step S202, which will not be repeated here.

[0167] The model training module 903 can be configured to perform direct preference optimization on the model to be trained using the preference data set based on the determined objective function. Optionally, the model training module 903 can perform the operations described above with reference to step S203, which will not be repeated here.

[0168] According to another aspect of this disclosure, a data processing apparatus is also provided. Figure 10 A schematic diagram of a data processing apparatus 2000 according to an embodiment of the present disclosure is shown.

[0169] like Figure 10 As shown, the data processing device 2000 may include one or more processors 2010 and one or more memories 2020. The memories 2020 store computer-readable code, which, when executed by the one or more processors 2010, can perform the data processing method described above.

[0170] The processor in the embodiments of this disclosure can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x86 architecture or an ARM architecture.

[0171] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0172] For example, the method or apparatus according to embodiments of this disclosure can also be used by means of Figure 11 The architecture of the computing device 3000 shown is used for implementation. For example... Figure 11 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication in the data processing method provided in this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 11 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 11 One or more components in the computing device shown.

[0173] According to another aspect of this disclosure, a computer-readable storage medium is also provided. The computer storage medium stores computer-readable instructions. When the computer-readable instructions are executed by a processor, a data processing method according to embodiments of this disclosure described with reference to the above-drawn figures can be performed. The computer-readable storage medium in the embodiments of this disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0174] Embodiments of this disclosure also provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a data processing method according to embodiments of this disclosure.

[0175] Embodiments of this disclosure provide a data processing method, apparatus, device, computer program product, and computer-readable storage medium.

[0176] The method provided by embodiments of this disclosure considers that the preference data used in DPO training may have noisy labels that negatively impact training performance. It proposes a noise-aware metric to indicate the probability that the preference label of a pair of preference data is a noisy label, and introduces this metric into the DPO training objective function. This incorporates the quality of the preference data into the DPO training process, thereby identifying and mitigating the impact of noisy preference data on DPO training performance. The method of embodiments of this disclosure can effectively identify noisy data in the preference data used in DPO training and effectively reduce the impact of noisy data on training performance by introducing a noise-aware metric into the DPO objective function, thus improving DPO training performance. Furthermore, the method provided by embodiments of this disclosure is widely applicable to various large models, including LLM and DM, to improve the DPO training performance of these large models.

[0177] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing at least one executable instruction for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0178] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0179] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure.

Claims

1. A data processing method, comprising: For a set of preference data pairs used for direct preference optimization of the model to be trained, a noise-aware metric is determined for each preference data pair in the set using multiple fine-tuned models, wherein each preference data pair includes two preference data with preference labels, each preference data being text data or image data, and for each preference data pair, the noise-aware metric is used to indicate the probability that the preference label of the preference data pair is a noisy label; Based on the determined noise perception metric, the objective function for direct preference optimization is determined; Based on the determined objective function, the model to be trained is directly optimized using the set of preference data.

2. The method as described in claim 1, wherein, The noise perception metric includes an annotator-built-in reliability term and an annotator-between stability term. The annotator-built-in reliability term is used to evaluate the difficulty of the multiple fine-tuned models in predicting the labels for the preferred data pairs, and the annotator-between stability term is used to evaluate the stability of the multiple fine-tuned models in predicting the labels for the preferred data pairs.

3. The method as described in claim 2, wherein, Determining the noise-perceived metric for each preference data pair in the set of preference data pairs using multiple fine-tuned models includes: For each pair of preference data: For each of the plurality of fine-tuned models, the confidence level of the fine-tuned model for the label prediction result of the preference data pair is determined, wherein the confidence level of the fine-tuned model for the label prediction result of the preference data pair is expressed as the difference between the label prediction result of the fine-tuned model for the preference data pair and the label prediction result of a predetermined reference model for the preference data pair. Based on the confidence statistics of the label prediction results of the multiple fine-tuned models for the preference data pairs, the annotator built-in confidence term and the annotator-to-annotator stability term of the preference data pairs are determined. Based on the annotator-built-in reliability term and the annotator-interaction stability term of the preference data pair, the noise perception metric is constructed, wherein the noise perception metric is positively correlated with the annotator-built-in reliability term and the annotator-interaction stability term.

4. The method of claim 3, wherein, Based on the statistics of the confidence levels of the label prediction results of the multiple fine-tuned models for the preference data pairs, the annotator-built-in reliability term and the annotator-internal stability term for the preference data pairs are determined as follows: Based on the statistical average of the confidence scores of the label prediction results of the preferred data pairs by the multiple fine-tuned models, the annotator-built-in confidence term of the preferred data pairs is determined, wherein the annotator-built-in confidence term is negatively correlated with the result of the statistical average, and preferred data pairs with larger annotator-built-in confidence terms correspond to larger label prediction biases.

5. The method of claim 3, wherein, Based on the statistics of the confidence levels of the label prediction results of the multiple fine-tuned models for the preference data pairs, the annotator-built-in reliability term and the annotator-internal stability term for the preference data pairs are determined as follows: Based on the variance among the confidence levels of the label prediction results of the multiple fine-tuned models for the preferred data pairs, the inter-annotator stability term of the preferred data pairs is determined, wherein preferred data pairs with a larger inter-annotator stability term correspond to greater label prediction instability.

6. The method of claim 3, wherein, Based on the annotator-built-in reliability term and the annotator-inter-annotator stability term of the preference data pair, the noise-aware metric is constructed as follows: The noise-aware metric is constructed based on the product of the annotator's built-in reliability term and the annotator-to-annotator stability term of the preference data pair.

7. The method of claim 1, wherein, The objective function is designed to reduce the impact of the preference data on the pair of preference data with noisy labels in the set on the direct preference optimization, and to increase the impact of the preference data on the pair of preference data without noisy labels in the set on the direct preference optimization.

8. The method of claim 1, wherein, Based on the determined noise perception metric, the objective function for direct preference optimization is determined as follows: For each preference data pair in the set of preference data pairs, a weighting coefficient corresponding to the preference data pair is determined based on the noise-aware metric, wherein the weighting coefficient is used to weight the label prediction result of the model to be trained for the preference data pair in the objective function, and preference data pairs with larger noise-aware metrics correspond to smaller weighting coefficients.

9. The method of claim 1, wherein, Based on the determined noise perception metric, the objective function for direct preference optimization is determined as follows: For each preference data pair in the set of preference data pairs, the prediction margin of the label prediction result of the model to be trained for the preference data pair in the objective function is determined based on the noise-aware metric, wherein preference data pairs with smaller noise-aware metrics correspond to larger prediction margins.

10. A data processing apparatus, comprising: The noise metric module is configured to determine a noise-aware metric for each preference data pair in the set of preference data pairs used for direct preference optimization of the model to be trained, using multiple fine-tuned models. Each preference data pair includes two preference data with preference labels, each preference data being either text data or image data. For each preference data pair, the noise-aware metric is used to indicate the probability that the preference label of the preference data pair is a noisy label. The objective building module is configured to determine the objective function for direct preference optimization based on the determined noise-aware metric; as well as The model training module is configured to perform direct preference optimization on the model to be trained using the preference data set based on the determined objective function.

11. A data processing device, comprising: One or more processors; as well as One or more memories storing a computer-executable program, which, when executed by the processor, performs the method of any one of claims 1-9.

12. A computer program product stored on a computer-readable storage medium and comprising computer instructions that, when executed by a processor, cause a computer device to perform the method of any one of claims 1-9.

13. A computer-readable storage medium having stored thereon computer-executable instructions, which, when executed by a processor, are used to implement the method of any one of claims 1-9.

Images