Model training method and intelligent question answering method
By constructing triplet data and training a loss function that combines contrastive learning and inverse direct preference optimization, the performance degradation problem of large language models in noisy scenarios is solved, achieving high accuracy and robustness, and is suitable for scenarios such as intelligent question answering systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING SANKUAI ONLINE TECH CO LTD
- Filing Date
- 2026-03-20
- Publication Date
- 2026-06-23
Smart Images
Figure CN122263972A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and more specifically, to a model training method and an intelligent question answering method. Background Technology
[0002] The cue noise problem present in large language models may affect model performance.
[0003] In related technologies, the main approach to solving the noise problem of large language model prompts focuses on prompt preprocessing and external tool repair. Specifically, this involves optimizing input prompts through external components independent of the model, and then inputting clean prompts into the model to ensure output quality.
[0004] The above methods have high computational and deployment costs. The multi-stage operation of the preprocessing process amplifies the error, affects the matching between the model and the requirements, and has low accuracy. Optimizing only the input causes the model to still rely on external components and cannot independently cope with new noise not covered by preprocessing, resulting in poor robustness. Summary of the Invention
[0005] The purpose of this disclosure is to provide a model training method and an intelligent question answering method, thereby overcoming, to at least some extent, the problem of poor quality of generated responses caused by the limitations and defects of related technologies.
[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.
[0007] According to one aspect of this disclosure, a model training method is provided, comprising: determining multiple local sample data from multiple subsets containing deterministic answers in a natural language processing task dataset; generating triples for each local sample data, the triples including a standard hint for label matching, a noise hint, and a label; inputting the standard hint for label matching, the noise hint, the standard hint for label irrelevant information, and the label into a large language model to determine corresponding conditional probability distributions; constructing a loss function that integrates contrastive learning and inverse direct preference optimization based on the conditional probability distributions; inputting paired sample data from the natural language processing task dataset into the large language model, outputting corresponding sample conditional probability distributions, and determining a loss value for the paired sample data based on the sample conditional probability distributions and the loss function; adjusting the model parameters of the large language model according to the loss value to obtain a trained large language model; the trained large language model is used to generate response information for the input data.
[0008] In one exemplary embodiment of this disclosure, the step of inputting the standard prompt matching the tag, the noise prompt, the tag-independent standard prompt, and the tag into a large language model to determine the corresponding conditional probability distribution includes: concatenating the standard prompt matching the tag, the noise prompt, and the tag-independent standard prompt with the tag to obtain multiple concatenated sequences; inputting each of the concatenated sequences into the large language model for forward computation to determine the original score matrix of each concatenated sequence; filtering the original score matrices corresponding to the prompts in the concatenated sequences through a tag mask matrix, retaining the original score matrices corresponding to the tags to obtain a tag score matrix; and converting the tag score matrix into the conditional probability distribution of each concatenated sequence.
[0009] In one exemplary embodiment of this disclosure, the step of filtering the original score matrix corresponding to the prompts in the concatenated sequence through a label mask matrix and retaining the original score matrix of the label position to obtain the label score matrix includes: determining the prompt position and the label position according to the concatenated sequence; marking the prompt position as a first value and the label position as a second value to generate the label mask matrix; and performing element-wise multiplication of the label mask matrix with the original score matrix to retain the original score matrix corresponding to the label to obtain the label score matrix.
[0010] In one exemplary embodiment of this disclosure, constructing a loss function that integrates contrastive learning and inverse direct preference optimization based on the conditional probability distribution includes: determining a first divergence between the noise cue and the label-independent standard cue on the conditional probability distribution; determining a first loss function based on the negative value of the first divergence; determining a second divergence between the noise cue and the label-matching standard cue on the conditional probability distribution; determining a second loss function based on the second divergence; and determining the loss function based on the first loss function and the second loss function.
[0011] In one exemplary embodiment of this disclosure, determining the first divergence between the noise cue and the tag-independent standard cue on the conditional probability distribution includes: determining a first probability distribution for each tag word position in the noise concatenation sequence corresponding to the noise cue, and determining a second probability distribution for each tag word position in the unrelated concatenation sequence corresponding to the tag-independent standard cue; calculating the divergence of the first probability distribution relative to the second probability distribution for each tag word position; and taking the average of the divergences of the tag word positions as the first divergence.
[0012] In one exemplary embodiment of this disclosure, the step of inputting paired sample data of the natural language processing task dataset into a large language model and outputting the corresponding sample conditional probability distribution includes: concatenating the standard hints, noise hints, and label-irrelevant standard hints in each paired sample data with the label, respectively, to generate a label training concatenation sequence, a noise training concatenation sequence, and an irrelevant training concatenation sequence; inputting the noise training concatenation sequence into the large language model for forward computation to determine a first probability distribution; inputting the irrelevant training concatenation sequence into the large language model for forward computation to output a second probability distribution; and inputting the label training concatenation sequence into the large language model for forward computation to output a third probability distribution.
[0013] In one exemplary embodiment of this disclosure, adjusting the model parameters of the large language model according to the loss value to obtain the trained large language model includes: calculating the gradient of the loss value with respect to all model parameters of the large language model; updating the model parameters of the large language model according to the gradient with the goal of minimizing the loss value, until the training termination condition is met, thereby obtaining the trained large language model.
[0014] In one exemplary embodiment of this disclosure, the standard hint for label matching in the triple is obtained according to a standardized template, the noise hint is the noise hint corresponding to the standard hint for label matching, and the label-independent standard hint is the standard hint for other tasks that are unrelated to the label.
[0015] In one exemplary embodiment of this disclosure, the method further includes: randomly selecting a perturbation type and determining the perturbation intensity of the perturbation type; mapping the perturbation intensity to a perturbation ratio; and performing a perturbation operation on the standard prompt for tag matching according to the perturbation ratio based on the perturbation type to determine the noise prompt; wherein the perturbation type includes at least one of character-level perturbation, word-level perturbation, and sentence-level perturbation.
[0016] According to one aspect of this disclosure, an intelligent question-answering method is provided, comprising: acquiring input data; inputting the input data into a trained large language model to generate response information corresponding to the input data; wherein the trained large language model is trained according to any one of the model training methods described above.
[0017] The technical solution provided in this disclosure, on the one hand, directly enhances the model's inherent resistance to noise cues by constructing paired sample data based on standard prompts for label matching, noise cues, label-irrelevant standard prompts, and labels, and training the model based on the paired sample data. Since no external preprocessing components are required, additional computation is reduced, engineering complexity is lowered, and deployment costs are reduced. On the other hand, multi-stage operations in the preprocessing process are avoided, thus eliminating cascading errors and improving accuracy. Furthermore, since the large language model is trained based on the loss value determined by the loss function of fusion contrastive learning and inverse direct preference optimization, the large language model itself is optimized. It utilizes the large language model's own feature learning capabilities, avoiding reliance on external components. This allows the model's potential to be explored to cope with various types of noise disturbances not covered by processing, meeting the requirements of real-time and lightweight scenarios. It guides the model parameters to continuously converge towards enhanced noise resistance and improved semantic discrimination accuracy during iterative updates, solving the problem of easy degradation of model performance in noisy scenarios using traditional loss functions, improving the output quality of the large language model in noisy scenarios, and enhancing accuracy and robustness.
[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0020] Figure 1 The illustration shows an application scenario diagram of a model training method according to an embodiment of the present disclosure.
[0021] Figure 2 The schematic diagram illustrates a flowchart of a model training method according to an embodiment of the present disclosure.
[0022] Figure 3 The schematic diagram illustrates the flowchart for calculating the conditional probability distribution of each spliced sequence in an embodiment of this disclosure.
[0023] Figure 4 The flowchart illustrating the training of a large language model in an embodiment of this disclosure is shown schematically.
[0024] Figure 5 The schematic diagram illustrates a flowchart of an intelligent question-answering method according to an embodiment of the present disclosure. Detailed Implementation
[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.
[0026] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0027] In related technologies, when input prompts are optimized using external components independent of the model, and then clean prompts are input into the model to address the noise problem in large language models, the introduction of external tools or auxiliary LLMs requires additional computing resources such as GPUs, video memory, and computing power, increasing inference time. Furthermore, multi-component deployment requires addressing compatibility issues, increasing engineering complexity. Therefore, these technologies consume significant computing resources and have high deployment costs and complexity. Multi-stage operations in the preprocessing process can amplify errors, and rewriting the auxiliary LLM can deviate from the user's original intent, leading to a disconnect between the final model output and user needs, resulting in poor accuracy. The solutions in related technologies only optimize the model input rather than the model itself, failing to utilize the model's own feature learning capabilities. This causes the model to still rely on external components, unable to independently handle novel noise not covered by preprocessing, resulting in poor robustness and low reliability.
[0028] To address the technical problems existing in related technologies, this disclosure provides a model training method that can be applied to scenarios with high requirements for real-time performance and lightweight design, such as generating response information in open dialogue systems.
[0029] Figure 1 A schematic diagram of a system architecture for which some embodiments of the present disclosure can be applied is shown.
[0030] In some embodiments, the system architecture 100 upon which the model training method and intelligent question answering method rely may include a terminal 110 and a server 120. The terminal 110 may be of any type. Data transmission between the terminal 110 and the server 120 can occur via a network 130. During the model training phase, the server 120 determines local sample data from multiple subsets containing deterministic answers uploaded by the terminal 110, constructs triples for the local sample data, determines a loss function that integrates contrastive learning and inverse direct preference optimization based on the label-matching standard hints, noise hints, labels, and label-independent standard hints in the triples, constructs paired sample data from the label-matching standard hints, noise hints, label-independent standard hints, and labels, processes the paired sample data using a large language model to output a sample conditional probability distribution, substitutes the sample conditional probability distribution into the loss function to determine the loss value of the paired sample data, and adjusts the model parameters of the large language model based on the loss value to obtain the trained large language model. During the model usage phase, the trained large language model can be deployed in any natural language processing task, such as an intelligent question answering system, to analyze the input data to be processed, so that the trained large language model can generate response information corresponding to the input data.
[0031] It should be noted that in the above model training method and intelligent question answering method, server 120 can be any type of server or server cluster. Terminal 110 can be any type of device. The above model training method and intelligent question answering method can be executed entirely by the server; or they can be partially executed by the server and partially by the terminal, without specific limitations here.
[0032] The technical solution in this disclosure avoids the problem of additional computing resources required by introducing external tools or auxiliary LLM (Large Language Model), thereby reducing inference time, engineering complexity, and deployment difficulty.
[0033] This avoids the problems of multi-stage operations in the preprocessing process amplifying errors and deviating from the user's original intent when rewriting the large language model, thus ensuring that the model output matches the requirements, reducing the risk of cascading errors, and improving accuracy.
[0034] By optimizing the model itself and leveraging its inherent feature learning capabilities, the model can independently handle novel noise not covered by preprocessing, increasing its application scope and robustness. An intra-model augmentation approach using contrastive learning fused with a loss function represented by inverse direct preference optimization, combined with a paired training set construction method, achieves the inherent resistance of large language models to cue noise, improving the model's response quality and stability in real-world noise scenarios while reducing computational costs.
[0035] Figure 2 The flowchart illustrating the model training method is shown in the image. (Reference) Figure 2 As shown, the model training method mainly includes the following steps: Step S210: Determine multiple local sample data from multiple subsets containing deterministic answers in the natural language processing task dataset, and generate a triplet for each local sample data. The triplet includes a standard hint for label matching, a noise hint, and a label.
[0036] In this embodiment, the natural language processing (NLP) task dataset can be data from any type of task, such as intelligent question answering data. For example, the NLP task dataset can be the FLAN (Fine-tuned LanguageNet) dataset. The core objective of the FLAN dataset is to fine-tune a pre-trained large language model through diverse instruction tasks, enabling the model to have stronger zero-shot and few-shot learning capabilities, and to better understand and execute various natural language instructions, rather than being limited to specific downstream tasks. It reconstructs the NLP dataset into a unified "instruction-input-output" format through template transformation, adapting to the needs of instruction fine-tuning.
[0037] Intelligent question-answering data can be business data from an intelligent question-answering system. This system can be a customer service system or an open dialogue system in other fields. The data can be single-turn or multi-turn dialogue data, specifically question and response information within a dialogue scenario. Question information can be text-based or voice-based. Response information can be generated through natural language understanding and analysis of the question information. Response information can be associated with the question information. This intelligent question-answering system can be deployed on terminal devices, embedded in an application platform, or as a standalone system. It can also be a standalone interactive device, such as a robot, self-service payment terminal, or hospital consultation terminal, etc., without specific limitations.
[0038] A subset containing deterministic answers refers to a set of subtasks selected from a natural language processing task dataset that share a common task type and produce unique and objective output answers. In a subset containing deterministic answers, the input instructions correspond to a uniquely determined standard answer, rather than open-ended or subjective generated results.
[0039] Multiple local sample data can be determined from multiple subsets containing deterministic answers in the natural language processing task dataset, with each local sample data belonging to a subset. Local sample data can be data pairs consisting of input and output data within a subset. Triples are generated for each local sample data from the multiple subsets containing deterministic answers. Each triple can include a standard hint for label matching, a noise hint, and a label. The standard hint for label matching refers to a hint associated with a label that does not contain noise. The standard hint for label matching in the triple is obtained based on a normalized template, which is used to encapsulate the local sample data into imperative hints that the model can understand. The noise hint in the triple is the noise hint corresponding to the standard hint for label matching. Specifically, it can be obtained by adding redundant information, grammatical errors, typos, etc., to the standard hint for label matching to simulate noise input in a real-world scenario. For example, the noise hint can be determined as follows: randomly selecting a perturbation type and determining the perturbation intensity of the perturbation type; mapping the perturbation intensity to a perturbation ratio; and perturbing the standard hint for label matching according to the perturbation ratio to determine the noise hint.
[0040] In some embodiments, the standard prompts for tag matching are hierarchically split into character, lexical, and sentence layers. One or more perturbation types are randomly selected from a perturbation type pool. These perturbation types can be, for example, at least one of character-level, lexical-level, and sentence-level perturbations. When multiple perturbation types are selected, it is called combined perturbation. During combined perturbation, the priority and probability of each perturbation type can be determined. For example, sentence-level perturbation may be performed first, followed by lexical-level or character-level perturbation. For each perturbation type, the perturbation intensity can be determined. The perturbation intensity is mapped to a perturbation ratio, and a perturbation operation is randomly selected. The perturbation operation is performed on the standard prompts for tag matching corresponding to the perturbation ratio to generate noise prompts. Character-level perturbation operations include replacement, insertion, deletion, and swapping. Lexical-level perturbation operations include synonym replacement, similar-looking word replacement, and low-frequency word replacement. Sentence-level perturbation operations may include sentence rewriting, redundant insertion, and word order adjustment. By generating noise cues through character-level, word-level, and sentence-level perturbations, and randomly selecting perturbation types and intensities, the model simulates real-world noise scenarios. Randomly selecting perturbation types and intensities allows the model to be exposed to various common noise cues encountered in real-world scenarios during training, effectively avoiding training bias caused by a single noise type and providing data support for subsequent enhancement of noise resistance through the model's intrinsic learning.
[0041] In this embodiment, a high-quality, multi-granularity paired training dataset can be constructed for training tasks based on a contrastive learning paradigm. The model needs to be iteratively optimized between standard and noisy prompts for label matching to enhance its intrinsic robustness.
[0042] Step S220: Input the standard prompts for label matching, noise prompts, label-irrelevant standard prompts, and labels into the large language model to determine the corresponding conditional probability distributions; construct a loss function that integrates contrastive learning and inverse direct preference optimization based on the conditional probability distributions; input paired sample data from the natural language processing task dataset into the large language model to output the corresponding sample conditional probability distributions; and determine the loss value of the paired sample data based on the sample conditional probability distributions and the loss function.
[0043] In this embodiment, the loss function is determined by assembling triples from local sample data and label-independent standard cues into input data. First, three core inputs are defined: a label-matching clean cue P1, a noise cue P1' corresponding to the label-matching clean cue P1, and a label-independent standard cue P2. Label-independent standard cues refer to noise-free cues from other tasks unrelated to the label. Multiple inputs, represented by the label-matching standard cue, the noise cue, and the label-independent standard cue, are concatenated with the label y in the triples to generate multiple concatenated sequences.
[0044] Furthermore, multiple concatenated sequences are input into a large language model for forward computation to determine the conditional probability distribution of each concatenated sequence. Figure 3 The flowchart illustrating the calculation of the conditional probability distribution for each spliced sequence is shown in the image. (See reference...) Figure 3 As shown, the main steps include: Step S310: Input the concatenated sequence into the large language model for forward computation to determine the original score matrix of each concatenated sequence; Step S320: Filter the original score matrix corresponding to the prompts in the concatenated sequence through the label mask matrix, and retain the original score matrix of the label position; Step S330: Convert the original score matrix of the label positions into the conditional probability distribution of each spliced sequence.
[0045] In this embodiment, each concatenated sequence can be input into a large language model for forward computation. The large language model can be any type of model, such as Llama, Qwen2.5, etc. Specifically, each concatenated sequence can be segmented into tokens according to the full word list of the large language model by a word segmenter, and mapped to a token ID sequence. After adding a special token, it is converted into an input tensor. The input tensor is input into the large language model, and forward propagation is performed to calculate the original score matrix. The specific process of calculating the original score matrix includes: the embedding layer of the large language model converts the token ID sequence into an embedding vector that integrates semantics and position. After passing through the attention layer, an attention feature tensor is obtained. The attention feature tensor is normalized according to residual connections and layer normalization, and residual superposition is used to obtain intermediate features. The intermediate features are nonlinearly mapped based on the feedforward network to finally output a feature tensor. This feature tensor integrates the global semantic information of all tokens in the concatenated sequence. In the model output layer, the feature tensor is mapped to form the original score matrix of each concatenated sequence.
[0046] During the score matrix filtering process, a binary mask matrix with the same dimensions as the original score matrix is first generated based on the concatenated sequence. This binary mask matrix is the label mask matrix. Specifically, the prompt positions and label positions can be determined based on the concatenated sequence. The prompt positions are marked with a first value, and the label positions are marked with a second value. In addition, markers or redundant positions can also be marked with a first value to generate the label mask matrix. The first value is 0, and the second value is 1.
[0047] The original score matrix of the concatenated sequence is element-wise multiplied with the label mask matrix. After multiplication, the scores of all non-label positions become 0, and only the original scores of the label positions are retained. From the multiplied matrix, all rows with values of 0 are removed to obtain the original score matrix containing only the label positions. This filters out the original score matrix corresponding to the prompts in the concatenated sequence, retaining the original score matrix of the label positions, thus obtaining the label score matrix for each concatenated sequence.
[0048] Based on this, the label score matrix of each concatenated sequence can be converted into probability values in the 0-1 interval, with the sum of the probabilities of each label position being 1, to obtain the conditional probability distribution. The conditional probability distribution represents the conditional probability that a given prompt sequence will generate a corresponding token at a label position. For example, each row of the label score matrix is subjected to an exponential operation using a normalization function, and the sum of the exponential operation results for that row is calculated. Dividing each exponential operation result in that row by the sum yields the probability value of the corresponding token, thus obtaining the conditional probability distribution for each concatenated sequence.
[0049] It should be noted that the concatenated sequence of noise indication and label can be a noise concatenation sequence, the concatenated sequence of label-independent standard hint and label can be an irrelevant concatenation sequence, and the concatenated sequence of label matching standard hint and label can be a label concatenation sequence. Each concatenation sequence can include multiple label term positions, and each label term position can correspond to an independent probability distribution. The label term position refers to the index position or index number corresponding to the token constituting the label in the token sequence corresponding to the concatenation sequence.
[0050] After determining the conditional probability distribution of each concatenated sequence, the loss function for fusion contrastive learning and inverse direct preference optimization can be determined based on the conditional probability distribution. Specifically, the loss function can be the CoIPO loss function for fusion contrastive learning and inverse direct preference optimization. The CoIPO loss function includes a first loss function and a second loss function. Both the first and second loss functions are used to represent the differences in probability distributions between different cues.
[0051] For example, the standard cue P1 for label matching and the noisy cue P1' belong to the same type of sample corresponding to the same task, while the standard cue P2 for label-independent tasks and the noisy cue P1' belong to the opposite type of sample corresponding to different tasks. A first divergence in the conditional probability distribution between the noisy cue and the label-independent standard cue can be determined, and a first loss function is determined based on the negative value of the first divergence. The first loss function is used to amplify the difference in probability distribution between the noisy cue P1' and the label-independent standard cue P2. The amplification of the difference in probability distribution between the noisy cue P1' and the label-independent standard cue P2 is measured by the first divergence between them; the smaller the first divergence, the closer the distributions; the larger the first divergence, the greater the difference in distributions.
[0052] The first divergence of the conditional probability distribution between the noise cue and the label-independent standard cue refers to the first divergence of the conditional probability distribution of the noise concatenation sequence relative to the conditional probability distribution of the label-independent concatenation sequence. For example, a first probability distribution for each label term position in the noise concatenation sequence corresponding to the noise cue can be determined, and a second probability distribution for each label term position in the label-independent standard cue can be determined. For each label term position, the divergence of the first probability distribution relative to the second probability distribution is calculated. Further, the average of the divergences of the label term positions is taken as the first divergence. The divergence of the first probability distribution relative to the second probability distribution can be calculated as follows: determine the logarithm of the ratio of the first probability distribution to the second probability distribution, and multiply the first probability distribution by the logarithm to determine the divergence of each label term position.
[0053] The second divergence of the conditional probability distributions of the noise prompt and the standard prompt for label matching is determined, and a second loss function is determined based on this second divergence. To make the noise prompt P1' as close as possible to the label distribution of the original standard prompt P1 for label matching, the KL divergence between the two is directly calculated as the second divergence, which is then used as the second loss function. The second divergence refers to the second divergence of the conditional probability distribution of the noise concatenation sequence relative to the conditional probability distribution of the label concatenation sequence. For example, the first probability distribution of each label term position in the noise concatenation sequence corresponding to the noise prompt can be determined, and the third probability distribution of each label term position in the label concatenation sequence corresponding to the standard prompt for label matching can be determined. For each label term position, the divergence of the first probability distribution relative to the third probability distribution is calculated. Further, the average value of the divergences of the label term positions is used as the second divergence. When the second divergence decreases, it indicates that the probability distribution of the noise prompt P1' is getting closer and closer to the standard prompt P1 for label matching. It can be considered that even with the addition of noise, the large language model can still output label prediction results that are close to the standard prompt for label matching, thus improving the robustness of the model to prompt noise. Both the first and second divergences can be KL (Kullback–Leibler Divergence, relative entropy) divergences.
[0054] After determining the first loss function and the second loss function, the first loss function and the second loss function can be weighted and summed to obtain the final loss function. If the task prioritizes noise resistance, the weight of the second loss function can be increased.
[0055] In this embodiment, a contrastive learning approach that combines noise resistance and semantic understanding accuracy is employed, utilizing a reverse learning paradigm with different inputs (cues) under the same output (label) and KL divergence to quantify probability distribution similarity. This approach enhances the model's resistance to cue noise directly from within, without relying on any external preprocessing tools or auxiliary components. Even when faced with noisy cuees, the model maintains a label prediction distribution similar to the standard cue that matches the label, ensuring stable output quality. Furthermore, by amplifying the probability distribution difference between the noisy cue P1' and the label-irrelevant standard cue P2, the model can clearly distinguish between semantically different cuees, avoiding misclassifying irrelevant task cuees as target task cuees, thus improving robustness and semantic understanding accuracy.
[0056] In this embodiment, a large language model can be trained based on paired sample data from a natural language processing task dataset. The paired sample data can be constructed from standard hints for label matching, noise hints, label-independent standard hints, and labels from the natural language processing task dataset. Each paired sample data set contains four sets of data: "P1-P1'-P2-label y". The standard hint P1 for label matching, the noise hint P1', and the label-independent standard hint P2 are input to the large language model separately, rather than simultaneously.
[0057] After the paired sample data is input into the large language model, the large language model can perform forward computation on the paired sample data and output the sample conditional probability distribution corresponding to the paired sample data.
[0058] Specifically, the standard cue for label matching, noise cue, and label-irrelevant standard cue in the paired sample data are concatenated with the label to obtain the training concatenation sequence; the training concatenation sequence may include the label training concatenation sequence, the noise training concatenation sequence, and the irrelevant training concatenation sequence.
[0059] The training spliced sequences represented by labeled training spliced sequences, noisy training spliced sequences, and irrelevant training spliced sequences are respectively input into a large language model for forward computation to determine the original score matrix of each training spliced sequence. A binary mask matrix is used to filter the original score matrices corresponding to prompts in the training spliced sequences, retaining the original score matrices of the label positions to obtain the training label score matrix for each training spliced sequence. The training label score matrix is then normalized to convert it into a sample conditional probability distribution for each training spliced sequence. Specifically, the noisy training spliced sequence is input into the large language model for forward computation to determine the first probability distribution; the irrelevant training spliced sequence is input into the large language model for forward computation to output the second probability distribution; and the labeled training spliced sequence is input into the large language model for forward computation to output the third probability distribution.
[0060] Next, the conditional probability distribution of each training concatenation sequence can be input into the expression of the loss function to determine the loss value of the paired sample data based on the conditional probability distribution and the loss function. Specifically, the first probability distribution and the second probability distribution are substituted into the loss function to obtain the first divergence, and the first probability distribution and the third probability distribution are substituted into the loss function to obtain the second divergence. The loss value corresponding to the paired sample data is obtained by weighted summation based on the first divergence and the second divergence.
[0061] In step S230, the model parameters of the large language model are adjusted according to the loss value to obtain the trained large language model; the trained large language model is used to generate response information for the input data.
[0062] Further, with minimizing the loss value as the training objective, the gradient of the loss value with respect to all model parameters of the large language model is calculated based on backpropagation. According to the calculated gradient, all trainable parameters of the model are updated using a preset optimization algorithm (such as AdamW) until the training termination condition is met, resulting in the trained large language model. Minimizing the loss value as the training objective refers to optimizing for reducing the differences in similar prompts and increasing the differences in dissimilar prompts. The training termination condition can be reaching the required number of training iterations or the large language model achieving performance convergence on the validation set. The trained large language model can generate response information corresponding to input data in natural language processing tasks. Input data can be one or more of text, data, or speech.
[0063] This embodiment of the disclosure conducts ablation experiments simultaneously during the training of a large language model, comparing the performance differences of using only Supervised Fine-Tuning (SFT), only Contrastive Learning (CL), only Inverse Direct Preference Optimization (InvDPO), and the complete CoIPO loss function to verify the effectiveness of each core component. Finally, the training and validation process is repeated on models with two different architectures, Llama and Qwen2.5, to ensure stability and universality. The ablation experiments show that the loss function using a fusion of Contrastive Learning and Inverse Direct Preference Optimization performs better.
[0064] In this embodiment, a training framework is proposed that uses standard prompts for label matching, noise prompts, label-irrelevant standard prompts, and labels to construct paired sample data. The training framework uses four sets of data consisting of "P1-P1'-P2-label y" as a unified input unit. Based on the constructed paired sample data, the model can simultaneously capture the semantic relationships between normal samples, noise samples, and contrast samples during the training process. This provides accurate data support for optimization with strong noise resistance and high semantic discriminativeness, breaking through the limitations of traditional single sample or simple paired input that cannot simultaneously achieve both noise resistance and semantic discriminativeness. This paper proposes a CoIPO loss function that integrates contrastive learning and inverse direct preference optimization as the training objective for robust training of large language models. This loss function optimizes the model's forward computation results, guiding the model parameters to converge towards enhanced noise resistance and improved semantic discrimination accuracy during iterative updates. This addresses the issue of performance degradation in noisy scenarios using traditional loss functions. In various noisy scenarios tested in NoisyPromptBench, the model's average accuracy is significantly higher than that of comparative methods such as SFT, CL, and InvDPO, with a performance degradation rate far lower than that of basic large language models. Furthermore, this approach achieves stable robustness improvements across different models, demonstrating its cross-model architecture generalization capability and adaptability to robust enhancement of different types of large language models.
[0065] Figure 4The diagram illustrates the training flowchart for a large language model. (See reference) Figure 4 As shown, the main steps include: Step S410: Construct pair-wise training data pairs. Here, the training data pairs can be paired sample data, including standard cue P1 for label matching, noise cue P1', standard cue P2 for label irrelevant data, and label y.
[0066] In step S420, the standard cue P1 for label matching, the noise cue P1', and the label-irrelevant standard cue P2 are concatenated with the label y to obtain multiple training concatenation sequences. These multiple training concatenation sequences can be represented as [P1,y], [P1',y], and [P2,y].
[0067] Step S430: Perform forward computation on each training concatenation sequence using the large language model to calculate the training label score matrix for each training concatenation sequence; convert the training label score matrix into the sample conditional probability distribution for each training concatenation sequence.
[0068] Step S440: Calculate the first divergence between the sample conditional probability distribution of the label-matching standard cue P1 and the sample conditional probability distribution of the noisy cue P1', and calculate the second divergence between the sample conditional probability distribution of the noisy cue P1' and the sample conditional probability distribution of the label-irrelevant standard cue P2.
[0069] Step S450: Determine the CoIPO loss value of the paired sample data based on the first divergence and the second divergence, calculate the gradient based on the loss value, and update the model parameters based on the gradient.
[0070] Step S460: Determine whether the model training has ended; if yes, proceed to step S470; if no, return to step S410 to continue training.
[0071] Step S470 yields the trained large language model.
[0072] The technical solution in this disclosure requires no external preprocessing components. By training the model with "clean-noise" paired sample data, it directly enhances the model's inherent resistance to cue noise. This avoids additional computational and deployment costs, eliminates cascading errors, and unlocks the model's inherent potential to cope with various types of perturbations, meeting real-time and lightweight requirements and significantly improving the model's output quality in noisy scenarios. It improves the model's response quality and stability in real-world noisy environments while reducing computational costs.
[0073] Figure 5 The flowchart of the intelligent question-answering method in an embodiment of this disclosure is illustrated schematically, illustrating the process of interaction through a natural language processing task. (Refer to...) Figure 5As shown, the main steps include: Step S510: Obtain input data; Step S520: Input the input data into the trained large language model to generate the response information corresponding to the input data.
[0074] In this embodiment, the natural language processing task can be performed based on an intelligent question-answering system, which can be a customer service system or other types of question-and-answer interaction systems. Input data can be text, data, or voice input by the user that is associated with the natural language processing task. The input data can be used to represent the input question. The large language model can be a model used to generate response information.
[0075] In this embodiment, deploying the trained large language model in an actual customer service system can improve the accuracy of the output response information. Furthermore, the large language model can reduce the rate of switching to human agents, improve the overall resolution rate and reliability of the customer service system, and enhance the user experience.
[0076] In some embodiments of this disclosure, a model training apparatus is provided, the model training apparatus comprising: The triplet construction module is used to determine multiple local sample data from multiple subsets containing deterministic answers in a natural language processing task dataset, and generate a triplet for each local sample data. The triplet includes a standard hint for label matching, a noise hint, and a label. The loss value determination module is used to input the standard prompts for label matching, noise prompts, label-irrelevant standard prompts, and labels into the large language model to determine the corresponding conditional probability distributions, and to construct a loss function that integrates contrastive learning and inverse direct preference optimization based on the conditional probability distributions; input the paired sample data of the natural language processing task dataset into the large language model, output the corresponding sample conditional probability distributions, and determine the loss value of the paired sample data based on the sample conditional probability distributions and the loss function; The model parameter update module is used to adjust the model parameters of the large language model according to the loss value to obtain the trained large language model; the trained large language model is used to generate response information for the input data.
[0077] In some embodiments of this disclosure, an intelligent question-answering device is provided, the intelligent question-answering device comprising: The data input module is used to acquire input data; The response information generation module is used to input the input data into the trained large language model and generate response information corresponding to the input data.
[0078] It should be noted that the specific details of each part of the above-mentioned device have been described in detail in some embodiments of the corresponding method. For details that are not disclosed, please refer to the embodiments of the method section, and therefore will not be repeated here.
[0079] Exemplary embodiments of this disclosure also provide an electronic device. This electronic device may be the aforementioned terminal device or server. Generally, the electronic device may include a processor and a memory, the memory for storing executable instructions of the processor, and the processor configured to execute the aforementioned model training method by executing the executable instructions. Furthermore, the electronic device may also include a display for displaying an operating interface.
[0080] The electronic device is described below as an example in the form of a general-purpose computing device. This electronic device is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0081] The components of an electronic device may include, but are not limited to: at least one processing unit, at least one storage unit, a bus connecting different system components (including storage units and processing units), and a display unit.
[0082] The storage unit stores program code that can be executed by the processing unit, causing the processing unit to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit can perform actions such as... Figure 2 The steps are shown in the figure.
[0083] The storage unit may include readable media in the form of volatile storage units, such as random access memory (RAM) and / or cache storage units, and may further include read-only memory (ROM).
[0084] The storage unit may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more applications, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.
[0085] A bus can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus that uses any of the various bus structures.
[0086] The electronic device can also communicate with one or more external devices (such as keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (such as routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (such as local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. As shown in the figure, the network adapter communicates with other modules of the electronic device via a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0087] It should be noted that some embodiments of this disclosure also provide a computer program product, which includes a computer program that implements the above-described method when executed by a processor.
[0088] In one implementation, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. For example, the computer program product can be implemented as a non-volatile storage medium storing a computer program, such as read-only memory, NAND flash memory, etc.
[0089] In one implementation, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.
[0090] Computer program code can be written in one or more programming languages. Examples of programming languages include C, Java, and C++. Program code can execute entirely on the user's computing device, partially on the user's computing device, or as a standalone software package. It can also execute partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).
[0091] Computer programs can be carried or transmitted via signals such as electrical, magnetic, optical, electromagnetic, and infrared rays. Electronic devices can convert signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, to be executed by the processor of the electronic device) the method steps of various exemplary embodiments of this disclosure.
[0092] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0093] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.
[0094] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. A model training method, characterized in that, include: Multiple local sample data are determined from multiple subsets containing deterministic answers in a natural language processing task dataset. Triples are generated for each local sample data, and the triples include standard hints for label matching, noise hints, and labels. The standard prompts for label matching, the noise prompts, the label-irrelevant standard prompts, and the labels are input into a large language model to determine the corresponding conditional probability distributions. Based on the conditional probability distributions, a loss function that integrates contrastive learning and inverse direct preference optimization is constructed. Paired sample data from the natural language processing task dataset are input into the large language model, and the corresponding sample conditional probability distributions are output. Based on the sample conditional probability distributions and the loss function, the loss value of the paired sample data is determined. The model parameters of the large language model are adjusted based on the loss value to obtain the trained large language model; the trained large language model is used to generate response information for the input data.
2. The model training method according to claim 1, characterized in that, The step of inputting the standard prompt for tag matching, the noise prompt, the tag-irrelevant standard prompt, and the tag into a large language model to determine the corresponding conditional probability distribution includes: The standard prompt matching the label, the noise prompt, and the label-irrelevant standard prompt are concatenated with the label to obtain multiple concatenated sequences; Each of the spliced sequences is input into the large language model for forward computation to determine the original score matrix of each spliced sequence; The original score matrix corresponding to the prompts in the concatenated sequence is filtered by the label mask matrix, and the original score matrix corresponding to the label is retained to obtain the label score matrix; The label score matrix is converted into the conditional probability distribution of each of the spliced sequences.
3. The model training method according to claim 2, characterized in that, The step of filtering the original score matrix corresponding to the prompts in the concatenated sequence through the label mask matrix and retaining the original score matrix of the label position to obtain the label score matrix includes: The prompt position and label position are determined based on the splicing sequence; The prompt position is marked with a first value, and the label position is marked with a second value to generate the label mask matrix; The label mask matrix is multiplied element-wise with the original score matrix, and the original score matrix corresponding to the label is retained to obtain the label score matrix.
4. The model training method according to claim 1, characterized in that, The loss function constructed based on the conditional probability distribution, which integrates contrastive learning and inverse direct preference optimization, includes: Determine the first divergence of the noise cue and the label-independent standard cue on the conditional probability distribution, and determine the first loss function based on the negative value of the first divergence; Determine the second divergence of the noise cue and the standard cue matching the label on the conditional probability distribution, and determine the second loss function based on the second divergence; The loss function is determined based on the first loss function and the second loss function.
5. The model training method according to claim 4, characterized in that, Determining the first divergence of the noise cue and the label-independent standard cue on the conditional probability distribution includes: Determine the first probability distribution of each tag word position in the noise splicing sequence corresponding to the noise prompt, and determine the second probability distribution of each tag word position in the irrelevant splicing sequence corresponding to the tag irrelevant standard prompt; For each tag word position, calculate the divergence of the first probability distribution relative to the second probability distribution; The average value of the divergence at the tag word positions is taken as the first divergence.
6. The model training method according to claim 1, characterized in that, The step of inputting paired sample data from the natural language processing task dataset into a large language model and outputting the corresponding sample conditional probability distribution includes: The standard cue, noise cue, and label-irrelevant standard cue in each paired sample data are concatenated with the label to generate the label training concatenation sequence, noise training concatenation sequence, and irrelevant training concatenation sequence, respectively. The noise training spliced sequence is input into the large language model for forward computation to determine the first probability distribution; The unrelated training concatenation sequence is input into the large language model for forward computation, and the second probability distribution is output. The labeled training concatenation sequence is input into the large language model for forward computation, and the third probability distribution is output.
7. The model training method according to claim 1, characterized in that, The step of adjusting the model parameters of the large language model based on the loss value to obtain the trained large language model includes: Calculate the gradient of the loss value with respect to all model parameters of the large language model; With the goal of minimizing the loss value, the model parameters of the large language model are updated according to the gradient until the training termination condition is met, thus obtaining the trained large language model.
8. The model training method according to claim 1, characterized in that, The standard hints for label matching in the triples are obtained based on the standardized template, the noise hints are the noise hints corresponding to the standard hints for label matching, and the label-independent standard hints are the standard hints for other tasks that are unrelated to the labels.
9. The model training method according to claim 1 or 8, characterized in that, The method further includes: Randomly select a disturbance type and determine the disturbance intensity of that disturbance type; Map the disturbance intensity to a disturbance ratio; Based on the perturbation type, the standard prompts for tag matching are perturbed according to the perturbation ratio to determine the noise prompts; the perturbation type includes at least one of character-level perturbation, word-level perturbation, and sentence-level perturbation.
10. An intelligent question-answering method, characterized in that, include: Get the input data; The input data is input into the trained large language model to generate response information corresponding to the input data; wherein the trained large language model is trained according to the model training method according to any one of claims 1-9.