A method and system for training semantic understanding models

Abstract

This invention provides a method and system for training a semantic understanding model. The method includes: acquiring preprocessed training sample data and using the training sample data to perform a two-stage model training on a pre-trained language model, comprising a contrastive instruction fine-tuning stage and a reinforcement learning stage; in the contrastive instruction fine-tuning stage, supervising fine-tuning training of the pre-trained language model based on a hybrid loss function with a supervised contrastive regularization term is performed to obtain a base model that minimizes the hybrid loss function; in the reinforcement learning stage, a policy model and a reference model are initialized using the base model, and the policy model is iteratively trained based on an objective function with dual constraints to obtain a semantic understanding model that minimizes the objective function. This invention improves the classification accuracy of the supervised fine-tuning stage, enhances the stability of the reinforcement learning stage, makes the model decisions align with business risk preferences, and further improves the diversity of model outputs and semantic understanding capabilities.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and system for training a semantic understanding model. Background Technology

[0002] In the field of natural language processing, especially for language models designed for specific business scenarios (such as ride-hailing customer service dialogues), a two-stage training paradigm of "supervised fine-tuning + reinforcement learning" is typically employed. However, existing technologies have significant limitations when processing data with highly semantically ambiguous and long-tailed distribution characteristics.

[0003] During the supervised fine-tuning phase, traditional methods typically aim to maximize the log-likelihood of predicted lexical units. When faced with samples that are semantically similar but have different intentions, such as in the ride-hailing scenario where "poor service attitude" and "verbal harassment" may contain similar negative words, this training method leads to severely overlapping and unclear decision boundaries formed by the model in the feature space. This results in frequent classification confusion in practical applications, reducing the accuracy of semantic understanding.

[0004] In the reinforcement learning phase, algorithms such as group relative policy optimization are often used for fine-tuning to make the model output more in line with human preferences. However, when applied to models with relatively few parameters, these algorithms rely heavily on KL divergence constraints on the output probability distribution to prevent the new policy from deviating too far from the original policy. This constraint is too rigid, often causing the model to sacrifice output diversity in pursuit of high scores in the reward function, falling into the "pattern collapse" dilemma, i.e., continuously generating a few high-scoring but monotonous responses, losing the generalization ability and naturalness of language. At the same time, the semantic representation within the model may also "drift," meaning that although the model has learned to generate text that conforms to the reward, its deep semantic understanding ability has deteriorated, which is fatal in complex business scenarios that require precise logical judgment. Therefore, existing technologies face the dual challenges of blurred classification boundaries and instability in the reinforcement learning process when processing complex dialogue data in specific domains. Summary of the Invention

[0005] To address these issues, this invention provides a semantic understanding model training method and system, aiming to solve the technical problems in existing model training methods when processing data from specific domains, such as blurred classification boundaries caused by the supervised fine-tuning stage, and training instability, poor model output diversity, and decreased semantic understanding ability caused by the reinforcement learning stage.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] According to a first aspect of the present invention, the present invention provides a method for training a semantic understanding model, the method comprising:

[0008] Obtain preprocessed training sample data, and use the training sample data to perform two-stage model training on the pre-trained language model, including a contrast instruction fine-tuning stage and a reinforcement learning stage.

[0009] In the comparison instruction fine-tuning stage, the pre-trained language model is subjected to supervised fine-tuning training based on the hybrid loss function with the introduction of a supervised comparison regularization term, to obtain a base model with the hybrid loss function minimized; the supervised comparison regularization term is used to bring the feature representation vectors of similar labeled samples closer together and push away the feature representation vectors of dissimilar labeled samples in the feature representation space of the pre-trained language model.

[0010] In the reinforcement learning phase, a policy model and a reference model are initialized using the base model, and the policy model is iteratively trained based on an objective function containing dual constraints to obtain a semantic understanding model that minimizes the objective function. The dual constraints include a KL divergence constraint term for constraining the difference in output probability distribution between the policy model and the reference model, and a semantic consistency constraint term for pre-setting the difference in semantic representation between the policy model and the reference model in the hidden layer vector space.

[0011] Furthermore, the hybrid loss function includes a cross-entropy loss term and a supervised contrastive regularization term, the mathematical expression of which is as follows:

[0012]

[0013] Among them, L stage L represents the hybrid loss function; CE L represents the cross-entropy loss term; SCL λ represents the supervised comparison regularization term; λ represents the loss balance coefficient.

[0014] Furthermore, the mathematical expression for the supervised contrastive regularization term is as follows:

[0015]

[0016] Among them, L SCL The supervised contrastive regularization term is represented by I; I represents the set of sample indices in the current training batch; P(i) represents the set of positive samples with the same label as sample i; B(i) represents the set of negative samples with different labels than sample i; z i z p z b τ represents the feature representation vectors of sample i, positive sample p, and negative sample b, respectively; τ represents the temperature coefficient used to control the attention given to difficult samples.

[0017] Furthermore, the objective function also includes a group policy optimization loss term and an entropy regularization term, the mathematical expressions of which are as follows:

[0018]

[0019] Among them, J New (θ) represents the objective function; θ represents the model parameters; E q,o Represents the mathematical expectation; This represents the group policy optimization loss term; This represents the KL divergence constraint term; This represents the semantic consistency constraint term; Represents the entropy regularization term; r t (θ) represents the importance weight; The function represents the group dominance function; the clip function represents the clipping function. Represents the constraint parameter; β t π represents the penalty coefficient of the KL divergence constraint term; θ This represents the current policy output by the policy model; π ref The reference strategy output by the reference model is represented by γ; the weight parameter of the semantic consistency constraint term is represented by h. θ h represents the hidden state vector of the current policy output by the policy model. ref α represents the hidden state vector of the reference strategy output by the reference model; α represents the weight parameter of the entropy regularization term.

[0020] Furthermore, the penalty coefficient of the KL divergence constraint term is configured to be dynamically adjusted based on the divergence values ​​of the policy model and the reference model in the current training iteration, as expressed mathematically below:

[0021]

[0022] Where, β t This represents the penalty coefficient corresponding to the number of training steps t; This represents the average KL divergence value of the current training iteration; This represents the baseline weight.

[0023] Furthermore, the semantic consistency constraint is configured to be determined by calculating the cosine distance between the hidden state vectors of the policy model and the reference model, as shown in the following mathematical expression:

[0024]

[0025] in, h represents semantic consistency constraints; θ h represents the hidden state vector of the current policy output by the policy model.ref This represents the hidden state vector of the reference strategy output by the reference model.

[0026] Furthermore, the within-group advantage function in the objective function is configured as a risk-sensitive weighted function, which is used to apply a larger update weight to candidate answers below a preset reward threshold than to candidate answer samples above the preset reward threshold. The mathematical expression is as follows:

[0027]

[0028] in, The within-group advantage function represents the importance of the i-th candidate answer in the current training batch; r i μ represents the reward value for the i-th candidate answer. r σ represents the average reward value of the current training batch; r δ represents the standard deviation of the reward value for the current training batch; ω represents the correction coefficient; ) represents the weight function.

[0029] Furthermore, the reward value of the candidate answer is configured to be obtained through multi-dimensional evaluation, and the mathematical expression is as follows: the multi-dimensional evaluation includes the evaluation of the accuracy of the generated result, the evaluation of the format, the evaluation of the security, and the evaluation of the logical coherence;

[0030]

[0031] Where, r total This represents the reward value for the candidate answer; r acc The accuracy reward represents the accuracy of the candidate answer; r format This represents a format reward indicating whether the candidate answer conforms to a preset structure format; r safety The security reward represents the content compliance of the candidate answer; I(CoT) logic ) represents the logical indication function characterizing the logical coherence of the candidate answer; w1, w2, w3, and w4 represent the corresponding weights of the accuracy reward, the format reward, the security reward, and the logical indication function, respectively.

[0032] Further, obtaining the preprocessed training sample data includes:

[0033] Obtain the original dialogue data, and manually annotate the original dialogue data to obtain initial sample data;

[0034] The initial sample data is deeply cleaned, expanded, and / or quality-verified using a bidirectional distillation technique that combines the logical long-chain reasoning capability of the DeepSeek-R1 model with the generalization understanding capability of the GPT-5 model, thereby generating the training sample data.

[0035] According to a second aspect of the present invention, the present invention provides a semantic understanding model training system, the system comprising:

[0036] The sample acquisition module is used to acquire preprocessed training sample data;

[0037] The model training module includes a fine-tuning training unit and a reinforcement learning unit, used to perform a two-stage model training on the pre-trained language model using the training sample data, comprising a contrastive instruction fine-tuning stage and a reinforcement learning stage.

[0038] The fine-tuning training unit is used to perform supervised fine-tuning training on the pre-trained language model based on a hybrid loss function with a supervised contrast regularization term introduced during the contrast instruction fine-tuning stage, so as to obtain a base model with the hybrid loss function minimized; the supervised contrast regularization term is used to bring the feature representation vectors of similar labeled samples closer together and push away the feature representation vectors of dissimilar labeled samples in the feature representation space of the pre-trained language model.

[0039] The reinforcement learning unit is used in the reinforcement learning phase to initialize a policy model and a reference model using the base model, and to iteratively train the policy model based on an objective function containing dual constraints to obtain a semantic understanding model that minimizes the objective function; the dual constraints include a KL divergence constraint term for constraining the difference in output probability distribution between the policy model and the reference model, and a semantic consistency constraint term for pre-setting the difference in semantic representation between the policy model and the reference model in the hidden layer vector space.

[0040] The present invention, by adopting the above technical solution, has at least the following beneficial effects:

[0041] This invention provides a method for training a semantic understanding model. It involves acquiring preprocessed training sample data and using this data to perform a two-stage model training process on a pre-trained language model, including a contrastive instruction fine-tuning stage and a reinforcement learning stage. In the contrastive instruction fine-tuning stage, the pre-trained language model is subjected to supervised fine-tuning training based on a hybrid loss function with a supervised contrastive regularization term, resulting in a base model that minimizes the hybrid loss function. The supervised contrastive regularization term is used to bring the feature representation vectors of similar-labeled samples closer together and push away the feature representation vectors of dissimilar-labeled samples in the feature representation space of the pre-trained language model. In the reinforcement learning stage, a policy model and a reference model are initialized using the base model, and the policy model is iteratively trained based on an objective function with dual constraints, resulting in a semantic understanding model that minimizes the objective function. The dual constraints include a KL divergence constraint term to constrain the difference in output probability distributions between the policy model and the reference model, and a semantic consistency constraint term to pre-determine the difference in semantic representations between the policy model and the reference model in the hidden layer vector space. This invention improves the classification accuracy in the supervised fine-tuning stage, enhances the stability in the reinforcement learning stage, makes the model decisions fit the business risk preferences, and further improves the diversity of model outputs and semantic understanding capabilities.

[0042] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A flowchart illustrating a semantic understanding model training method according to an embodiment of the present invention is shown.

[0045] Figure 2 This invention provides a comparative schematic diagram of the feature space topology structure during the fine-tuning stage of the comparison instruction according to an embodiment of the present invention.

[0046] Figure 3 This diagram illustrates the signaling interaction during a single iteration of the reinforcement learning phase according to an embodiment of the present invention.

[0047] Figure 4 A schematic diagram of the structure of a semantic understanding model training system provided in an embodiment of the present invention is shown. Detailed Implementation

[0048] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0049] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0050] This invention provides a method for training a semantic understanding model, such as... Figure 1 As shown, it may include at least the following steps S101~S103:

[0051] Step S101: Obtain preprocessed training sample data, and use the training sample data to perform two-stage model training on the pre-trained language model, including a contrast instruction fine-tuning stage and a reinforcement learning stage.

[0052] In this embodiment of the invention, the training sample data is customer service dialogue data in a ride-hailing scenario. This type of data typically contains a large number of samples with similar semantics but different intent categories. For example, dialogue texts such as "poor driver service attitude" and "driver engages in verbal harassment" may both contain negative emotional words, but correspond to different business liability categories.

[0053] It is understandable that the initial data collection typically involves massive amounts of customer service dialogue data. To improve the quality of training sample data and reduce the burden of manual annotation, this invention proposes a data augmentation strategy centered on "high-quality seed guidance, cross-modal distillation, and logic chain reinforcement." Specifically, the initial dialogue data can be obtained and manually annotated to generate initial sample data. Then, deep business prompts are used, and a bidirectional distillation technique combining the long-chain logical reasoning ability of the DeepSeek-R1 model and the generalization understanding ability of the GPT-5 model is employed to perform deep cleaning, data expansion, and / or quality verification on the initial sample data, generating training sample data.

[0054] In practical applications, by embedding deep business prompts at the model's core (using structured, business-customized system prompts to solidify domain rules and expert perspectives, significantly reducing reliance on labeled data), the amount of original sample data requiring manual annotation is compressed to a minimal scale of less than ten thousand records. Subsequently, a "Gemini" data distillation process can be performed, leveraging the long-chain logical reasoning capabilities of the DeepSeek-R1 model and the generalization understanding capabilities of the GPT-5 model. This involves the DeepSeek-R1 model generating logically rigorous problem-solving paths (i.e., thought processes), while the GPT-5 model verifies semantic diversity. The DeepSeek-R1 model automatically generates new cases with different parameter combinations based on the manually labeled original sample data (seed samples), thus expanding the basic data to be logically rigorous, conforming to business rules, and incorporating thought processes. Furthermore, the GPT-5 model can increase the representational diversity of the basic data and filter out erroneous data without altering the logical results. In this way, the original training data can be expanded into a large amount of high-quality training sample data with diverse semantics and correct logic. These training sample data not only contain the final intent labels, but also the logical steps from the original dialogue to deriving that intent, thus providing richer supervision information for subsequent model training. In this way, the reliance of traditional NLP training on massive amounts of labeled data is broken, reducing manual costs by more than 90% and achieving a lossless transfer from "expert experience" to "machine intelligence".

[0055] In one optional embodiment, in order to construct a semantic understanding model that can accurately understand driver-passenger disputes, service procedures, and trip strategies in the ride-hailing field, contrastive learning based on independent sampling and a balancing strategy based on diversified collection can also be introduced during the data processing process, which will be described in detail below.

[0056] To further improve the performance of the DeepSeek-R1 and GPT-5 models in real-world scenarios, a batch of independently collected online voice dialogue validation data was introduced. Through a contrastive learning mechanism, the sample data generated by these models was forcibly aligned with the multi-turn dialogue seed data from our company's real online "top customer service representatives." This further enhanced the logical analysis capabilities for other samples with the same intent label (positive samples) and other samples with different intent labels (negative samples). This not only expanded the boundaries of the training set but also allowed the models to learn the response strategies and empathetic communication techniques used by frontline customer service representatives when handling "driver-passenger conflicts and fare disputes," achieving a qualitative leap in data processing.

[0057] Furthermore, embodiments of the present invention can also employ diversified data collection and balancing strategies to address the pain point of scarce samples in the "long tail problem" of ride-hailing scenarios. Specifically, in practical applications, "gold medal closed cases" from first-hand voice calls in past historical time periods can be deeply mined and transformed into high-value seed corpora. Based on these refined gold medal seeds, the volume is reduced and efficiency is improved. That is, through refined distillation and manual review, the total amount of high-quality training sample data is controlled to an extremely concise level of less than 50,000 records. Through this "quality-first" strategy, the problem of data scarcity is effectively avoided, while preventing performance dilution of the model in massive amounts of noisy data.

[0058] In another optional embodiment of the present invention, when calling the DeepSeek-R1 model and the GPT-5 model to perform deep cleaning of the original sample data, not only is the model output required, but the endogenization of the Chain of Thought (CoT) must also be emphasized. For the unique liability judgment logic of the ride-hailing industry, this embodiment of the present invention performs mandatory logic injection into the DeepSeek-R1 model and the GPT-5 model. Taking "trip delay" liability judgment as an example, the model is required to follow a rigorous chain of falsification during reasoning. For example, the "trip delay" label is triggered only when the passenger has actually missed a concrete event such as a "plane, high-speed rail, or concert" and provides evidence of delay; if the expression is only subjective anxiety such as "I'm going to miss it" or "I suggest you hurry," it is classified as "user urging." It can be understood that since "trip delay" is directly linked to high compensation, this reasoning process with a chain of thought ensures that the model, when facing complex driver-passenger game dynamics, not only "knows what," but also "knows why," thereby guaranteeing business compliance and financial security from the source.

[0059] It should be noted that in the above model distillation and parameter alignment stage, the embodiments of the present invention select the Qwen-3 (Tongyi Qianwen) full series of open-source architectures as the core base model, that is, the preprocessing language model. Based on the excellent performance of Qwen-3 in Chinese context understanding, long text processing, and complex instruction following, the embodiments of the present invention construct a multi-gradient student model matrix. Specifically, for different online car-hailing service load requirements, a precise "parameter-scenario" matching strategy is set: lightweight (8B / 14B) models focus on scenarios of online intelligent customer service inspection with extremely high real-time requirements and high-frequency driver-passenger responses, ensuring a response speed of milliseconds; medium-weight (32B) models are used to handle tasks that require strong logical deduction, such as complex liability determination and customer complaint damage assessment involving multi-party games; heavyweight (72B) models, as the "knowledge brain" of the large parameter level, are responsible for long-term business report analysis, policy evolution simulation, and handling of ultra-long chain logical disputes, providing a high-precision distillation benchmark for small parameter models.

[0060] After completing the above data preparation stage, it enters a two-stage model training including a contrastive instruction fine-tuning stage and a reinforcement learning stage.

[0061] It can be understood that in order to meet the needs of subsequent contrastive learning, the data preparation stage can also determine the positive sample set (that is, other samples with the same intent label as the anchor sample) and negative sample set (that is, other samples with different intent labels from the anchor sample) for each sample (anchor sample) in the batch according to the intent label of the training sample data.

[0062] Step S102, in the contrastive instruction fine-tuning stage, perform supervised fine-tuning training on the pre-trained language model based on a hybrid loss function introducing a supervised contrastive regularization term to obtain a base model with the minimized hybrid loss function.

[0063] Compared with the traditional supervised fine-tuning SFT, the embodiments of the present invention present a contrastive-regularized SFT based on maximizing the inter-class margin, introducing a supervised contrastive regularization term into the loss function to solve the problem of "fuzzy liability determination" in the online car-hailing scenario, forcing the model to拉近 the dialogue embeddings belonging to the same intent and推远 those with different intents. Among them, the supervised contrastive regularization term is used to拉近 the feature representation vectors of samples with the same label and推远 the feature representation vectors of samples with different labels in the feature representation space of the pre-trained language model.

[0064] Specifically, the hybrid loss function includes a cross-entropy loss term and a supervised contrastive regularization term, and the mathematical expression is as follows:

[0065]

[0066] Among them, L stage L represents the mixed loss function; CE L represents the cross-entropy loss term; λ represents the loss balance coefficient; SCL The supervised comparison regularization term is expressed mathematically as follows:

[0067]

[0068] Among them, L SCL Let z represent the supervised contrastive regularization term; I represent the set of sample indices in the current training batch; P(i) represent the set of positive samples with the same label as sample i (anchor sample); B(i) represent the set of negative samples with different labels than sample i; z i z p z b Let represent the feature representation vectors of sample i, positive sample p, and negative sample b, respectively (the hidden state vectors output by the pre-trained language model encoder layer from the training sample data); τ represents the temperature coefficient used to control the attention given to hard negative samples. Sim(z) i , z p ) is the feature representation vector z of anchor sample i. i The feature vector z of the positive sample p p The similarity function between samples can typically be cosine similarity. Understandably, a smaller τ amplifies similarity differences, causing the model to focus more on distinguishing negative samples that are similar to the anchor sample but have different labels.

[0069] like Figure 2 The diagram shown illustrates a comparison of the feature space topology of a pre-trained language model during the instruction fine-tuning stage. Figure 2 As shown in the left half, traditional supervised fine-tuning methods only optimize the cross-entropy loss, often leading to a mixture of sample points from different classes in the feature space, forming a blurred decision boundary. Correspondingly, as... Figure 2 As shown in the right half, this embodiment of the invention introduces contrastive learning, which makes similar samples aggregate more compactly in the feature space (achieving intra-class compactness), while sample clusters of different categories are significantly pushed apart (achieving inter-class separation), thereby forming a clear decision boundary.

[0070] By analyzing the above-mentioned hybrid loss function L stage Backpropagation is performed to update the parameters of the pre-trained language model. The cross-entropy loss term L... CE The supervised contrastive loss term L enables the model to learn to generate fluent and accurate text. SCLThis prompts the feature representation within the model to bring the vectors of samples with the same intent (such as different expressions of "adjusting the degree difference in service attitude") closer together, while the vectors of samples with different intents (such as "poor service attitude" and "verbal harassment") are pushed apart. After sufficient fine-tuning and training, the resulting pre-trained language model possesses stronger semantic discrimination capabilities, thus forming the base model. The state of this base model is saved as the starting point for subsequent stages.

[0071] In step S103, during the reinforcement learning phase, a policy model and a reference model are initialized using a base model, and the policy model is iteratively trained based on an objective function with dual constraints to obtain a semantic understanding model that minimizes the objective function.

[0072] Compared to traditional reinforcement learning GRPO, this invention presents an improved GRPO (E-DC-GRPO) based on dynamic entropy regularization and semantic consistency constraints. It not only relies on KL divergence but also introduces a "semantic drift penalty term" and "dynamic entropy flow" to address the "reward hacking" problem. In other words, building upon the base model's already strong semantic understanding capabilities, it further enhances the model's output by interacting with the environment and using reward signals, making the model's output more aligned with human preferences and specific business needs (such as logical consistency, security, and format conformity). This also avoids issues like "pattern collapse" and "semantic drift" during the optimization process.

[0073] In the reinforcement learning phase, the parameters of the base model trained during the contrastive instruction fine-tuning phase are first used to simultaneously initialize a policy model and a reference model. The policy model is the one whose parameters are continuously updated during training, while the parameters of the reference model remain fixed throughout the reinforcement learning phase, serving as a stable anchor point for policy model updates. The model then enters an iterative training loop. Figure 3 The diagram shows the signaling interaction in a single iteration. In each iteration of the reinforcement learning phase, the following steps are performed:

[0074] Step 1: Forward propagation and sampling.

[0075] The model draws a batch of inputs from the training sample data (e.g., a batch of user questions or dialogue context). The policy model receives these inputs and performs forward propagation, generating one or more candidate answers for each input. Simultaneously, the reference model also performs forward propagation on the same inputs for subsequent constraint calculation. During this process, both models output their hidden state vectors in the last layer, denoted as h0. θ and h ref .

[0076] Step 2: Evaluation and reward of candidate responses.

[0077] This step evaluates the quality of each candidate answer generated by the strategy model and assigns a quantified reward value. In this embodiment of the invention, the reward model is configured as a multi-dimensional evaluation composite model with a total reward value r. total It consists of multiple weighted components, and the mathematical expression is as follows:

[0078]

[0079] Where, r total This represents the reward value for a candidate answer; r acc The accuracy reward represents the accuracy of the candidate answers; r format The format reward indicates whether a candidate answer conforms to a preset structure format; r safety The security reward represents the content compliance of candidate answers; I(CoT) logic Let represent the logical indicator function characterizing the logical coherence of the candidate answer; w1, w2, w3, and w4 represent the corresponding weights of the accuracy reward, format reward, security reward, and logical indicator function, respectively. In other words, the accuracy reward assesses the accuracy of the generated result, i.e., whether the answer correctly identifies the user's intent or solves the problem; the format reward assesses the format, i.e., whether the answer follows preset format requirements (such as JSON output); the security reward assesses whether the answer contains inappropriate or harmful non-compliant content; and the logical indicator function is used to assess logical coherence. As a preferred implementation, the assessment of logical coherence can be performed by an independent, more powerful discriminator model, specifically designed to evaluate the logical rationality and coherence of the internal thought chain relied upon by the strategy model when generating the answer. By comprehensively evaluating these dimensions, a comprehensive reward value r can be obtained. total This multi-dimensional reward design ensures that the optimization of the model is comprehensive and meets business needs.

[0080] Step 3: Calculate the intra-group dominance function.

[0081] In this embodiment of the invention, the within-group advantage function is configured as a risk-sensitive weighted function. The risk-sensitive weighted function is used to apply a larger update weight to candidate answers below a preset reward threshold than to candidate answer samples above the preset reward threshold. The mathematical expression is as follows:

[0082]

[0083] in, The within-group advantage function represents the importance of the i-th candidate answer in the current training batch; r i μ represents the reward value for the i-th candidate answer.r σ represents the average reward value of the current training batch; r δ represents the standard deviation of the reward value for the current training batch; δ represents the correction factor to prevent the denominator from being zero; ω( ) represents the weighting function, which is designed such that when the reward value r i At lower (e.g., below the mean μ) r ω(r) at time i The value of r is relatively large (e.g., greater than 1); while when r i When it is higher, ω(r) i The value of ) is relatively small (e.g., equal to 1). This asymmetric weighting mechanism makes the model pay more attention to those erroneous outputs that lead to low rewards when updating parameters, and correct them with greater gradient strength. This is crucial for avoiding serious consequences in high-risk business scenarios (such as misjudgment).

[0084] Step 4: Double constraint calculation.

[0085] The dual constraints set in the embodiments of the present invention may include a KL divergence constraint term for constraining the difference in output probability distribution between the policy model and the reference model, and a semantic consistency constraint term for constraining the difference in semantic representation between the preset policy model and the reference model in the hidden layer vector space.

[0086] The KL divergence constraint term is implemented by calculating dynamic KL divergence. It constrains the difference between the output probability distribution of the policy model and the output probability distribution of the reference model, preventing training failure due to excessively rapid policy updates. Its penalty coefficient β... t It is not a fixed constant, but rather based on the average KL divergence value of the current training batch. Dynamic adjustment is performed, and the mathematical expression is:

[0087]

[0088] Where, β t This represents the penalty coefficient corresponding to the number of training steps t; This represents the average KL divergence value of the current training iteration; This represents the baseline weights. It can be understood that this logarithmic dynamic adjustment mechanism allows for a more stable initial training phase when the policy model does not deviate significantly from the reference model. (Smaller), penalty coefficient β t It is also smaller, thus encouraging the model to explore more extensively; while when the policy deviates too much in the later stages of training or due to a certain update ( (relatively large), penalty coefficient β t It will grow rapidly in a logarithmic manner, exerting a strong pull to bring the policy model back to the vicinity of the reference model, thereby ensuring the stability of training.

[0089] Furthermore, the semantic consistency constraint is implemented by calculating the semantic consistency loss. It should be noted that this semantic consistency constraint aims to address the problems of "semantic drift" or "semantic collapse." Traditional KL divergence constraints only focus on the probability distribution at the output level and cannot guarantee that the semantic understanding ability within the model will not degrade. The semantic consistency constraint is implemented by directly constraining the semantic representation differences between the policy model and the reference model in the hidden layer vector space. In this embodiment of the invention, the semantic consistency constraint is achieved by calculating the final hidden layer state vector h of the two models. θ and h ref The semantic consistency constraint is determined by the cosine distance or Wasserstein distance between the parties. Taking the cosine distance as an example, the mathematical expression for the semantic consistency constraint is as follows:

[0090]

[0091] in, h represents semantic consistency constraints; θ h represents the hidden state vector of the current policy output by the policy model; ref This represents the hidden state vector of the reference policy output by the reference model.

[0092] Understandably, the semantic consistency constraint ensures that while the policy model learns to generate high-reward text, its deep semantic understanding of the input remains consistent with that of the comparatively fine-tuned reference model, which possesses strong semantic representation capabilities. This fundamentally prevents the model from "forgetting" its core semantic understanding capabilities in a one-sided pursuit of reward scores.

[0093] Step 5: Update model parameters.

[0094] This step integrates the calculated within-group advantage function, KL divergence constraint, semantic consistency constraint, and an optional entropy regularization term to encourage output diversity, constructing the final E-DC-GRPO objective function, mathematically expressed as follows:

[0095]

[0096] Among them, J New (θ) represents the objective function; θ represents the model parameters; E q,o Indicates mathematical expectation, q represents the question / hint word, and o represents the output / answer; This represents the group policy optimization loss term; Represents the KL divergence constraint term; This represents semantic consistency constraints; Represents the entropy regularization term; r t (θ) represents the importance weight; The within-group dominance function can be expressed as the ratio of the number of samples observed at time step t to the number of samples from the same group. i Seek The result is obtained through averaging; clip represents the clipping function; Represents the constraint parameter; β t π represents the penalty coefficient for the KL divergence constraint term; θ This represents the current policy output by the policy model; π ref The reference strategy is represented by the output of the reference model; γ represents the weight parameter of the semantic consistency constraint term; h θ h represents the hidden state vector of the current policy output by the policy model; ref α represents the hidden state vector of the reference policy output by the reference model; α represents the weight parameter of the entropy regularization term.

[0097] The above iterative process is repeated continuously. By calculating the gradient of the objective function with respect to the corresponding model parameters of the policy model, and using an optimizer (such as AdamW) to update the model parameters, after multiple rounds of iterative training, the model performance converges or reaches the preset number of training steps. The semantic understanding model that minimizes the objective function can improve the accuracy of the ride-hailing semantic understanding task from about 90% of the existing technology to more than 95%, and shows higher stability and robustness when dealing with high-risk liability judgment scenarios.

[0098] This invention provides a method for training a semantic understanding model, which, compared with existing technologies, has at least the following advantages:

[0099] 1) Improve classification accuracy and robustness. By introducing contrastive learning in the supervised fine-tuning stage, the distance between fuzzy categories in the feature space is significantly increased, which solves the classification confusion problem caused by traditional supervised fine-tuning and significantly improves the accuracy and stability of the model on complex tasks.

[0100] 2) Enhance the stability of reinforcement learning. Through the innovative dual constraint mechanism of "KL divergence + semantic consistency", it effectively prevents the pattern collapse and semantic drift caused by the pursuit of rewards during the training of the model in reinforcement learning, and ensures that the model does not forget the core business logic when learning new policies.

[0101] 3) Improve training efficiency and generalization ability. By using a dynamically adaptive KL penalty coefficient, the training process can adaptively balance exploration and utilization, reducing the dependence on manual fine-tuning of parameters, making the training process more stable and automated, thereby improving the model's generalization ability.

[0102] 4) Aligning with business risk preferences, the design of a risk-sensitive advantage function allows the model to focus more on misjudged samples that may lead to serious business consequences during training. This directly addresses the business pain point of "high cost of misjudgment" and optimizes the final model's decision-making behavior to better meet actual business needs.

[0103] Furthermore, as Figure 1 In specific implementation, embodiments of the present invention provide a semantic understanding model training system, such as... Figure 4 As shown, the system may include a sample acquisition module 410 and a model training module 420.

[0104] The sample acquisition module 410 can be used to acquire preprocessed training sample data;

[0105] The model training module 420 includes a fine-tuning training unit 421 and a reinforcement learning unit 422, which can be used to perform two-stage model training on a pre-trained language model using training sample data, including a contrastive instruction fine-tuning stage and a reinforcement learning stage.

[0106] The fine-tuning training unit 421 can be used to perform supervised fine-tuning training on the pre-trained language model based on the hybrid loss function with the introduction of supervised contrast regularization term during the contrast instruction fine-tuning stage, so as to obtain the base model with the minimized hybrid loss function; the supervised contrast regularization term is used to bring the feature representation vectors of similar label samples closer together and push the feature representation vectors of dissimilar label samples further apart in the feature representation space of the pre-trained language model.

[0107] The reinforcement learning unit 422 can be used to initialize a policy model and a reference model using a base model during the reinforcement learning phase, and iteratively train the policy model based on an objective function containing dual constraints to obtain a semantic understanding model that minimizes the objective function. The dual constraints include a KL divergence constraint term used to constrain the difference in output probability distribution between the policy model and the reference model, and a semantic consistency constraint term used to constrain the difference in semantic representation between the pre-defined policy model and the reference model in the hidden layer vector space.

[0108] It should be noted that other corresponding descriptions of the functional modules involved in the semantic understanding model training system provided in this embodiment of the invention can be found in [reference]. Figure 1 The corresponding description of the method shown will not be repeated here.

[0109] Those skilled in the art will clearly understand that the specific working process of the systems, devices, modules and units described above can be referred to the corresponding process in the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0110] Furthermore, the functional units in the various embodiments of the present invention can be physically independent of each other, or two or more functional units can be integrated together, or all functional units can be integrated into one processing unit. The integrated functional units described above can be implemented in hardware, or in software or firmware.

[0111] Those skilled in the art will understand that if the integrated functional unit is implemented in software and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or all or part of it, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computing device (e.g., a personal computer, server, or network device) to execute all or part of the steps of the methods described in the embodiments of the present invention when running the instructions. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] Alternatively, all or part of the steps of the foregoing method embodiments can be implemented by hardware (such as a computing device, personal computer, server, or network device) related to program instructions. The program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by the processor of the computing device, the computing device executes all or part of the steps of the methods described in the various embodiments of the present invention.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principles of the present invention, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the corresponding technical solutions to depart from the protection scope of the present invention.

Claims

1. A method for training a semantic understanding model, characterized in that, The method includes: Obtain preprocessed training sample data, and use the training sample data to perform a two-stage model training on the pre-trained language model, which includes a contrast instruction fine-tuning stage and a reinforcement learning stage; the training sample data is customer service dialogue data in the ride-hailing scenario; In the comparison instruction fine-tuning stage, the pre-trained language model is subjected to supervised fine-tuning training based on the hybrid loss function with the introduction of a supervised comparison regularization term, to obtain a base model with the hybrid loss function minimized; the supervised comparison regularization term is used to bring the feature representation vectors of similar labeled samples closer together and push away the feature representation vectors of dissimilar labeled samples in the feature representation space of the pre-trained language model. In the reinforcement learning phase, a policy model and a reference model are initialized using the base model, and the policy model is iteratively trained based on an objective function containing dual constraints to obtain a semantic understanding model that minimizes the objective function. The dual constraints include a KL divergence constraint term for constraining the difference in output probability distribution between the policy model and the reference model, and a semantic consistency constraint term for pre-setting the difference in semantic representation between the policy model and the reference model in the hidden layer vector space.

2. The method according to claim 1, characterized in that, The hybrid loss function includes a cross-entropy loss term and a supervised contrastive regularization term, and its mathematical expression is as follows: Among them, L stage L represents the hybrid loss function; CE L represents the cross-entropy loss term; SCL λ represents the supervised comparison regularization term; λ represents the loss balance coefficient.

3. The method according to claim 2, characterized in that, The mathematical expression for the supervised contrastive regularization term is as follows: Among them, L SCL The supervised contrastive regularization term is represented by I; I represents the set of sample indices in the current training batch; P(i) represents the set of positive samples with the same label as sample i; B(i) represents the set of negative samples with different labels than sample i; z i z p z b τ represents the feature representation vectors of sample i, positive sample p, and negative sample b, respectively; τ represents the temperature coefficient used to control the attention given to difficult samples.

4. The method according to claim 1, characterized in that, The objective function also includes a group policy optimization loss term and an entropy regularization term, the mathematical expressions of which are as follows: Among them, J New (θ) represents the objective function; θ represents the model parameters; E q,o Represents the mathematical expectation; This represents the group policy optimization loss term; This represents the KL divergence constraint term; This represents the semantic consistency constraint term; The entropy regularization term is represented by rt(θ); the importance weight is represented by rt(θ). The function represents the group dominance function; the clip function represents the clipping function. Represents the constraint parameter; β t π represents the penalty coefficient of the KL divergence constraint term; θ This represents the current policy output by the policy model; π ref The reference strategy output by the reference model is represented by γ; the weight parameter of the semantic consistency constraint term is represented by h. θ h represents the hidden state vector of the current policy output by the policy model. ref α represents the hidden state vector of the reference strategy output by the reference model; α represents the weight parameter of the entropy regularization term.

5. The method according to claim 4, characterized in that, The penalty coefficient of the KL divergence constraint term is configured to be dynamically adjusted based on the divergence values ​​of the policy model and the reference model in the current training iteration, as shown in the following mathematical expression: Where, β t This represents the penalty coefficient corresponding to the number of training steps t; This represents the average KL divergence value of the current training iteration; This represents the baseline weight.

6. The method according to claim 4, characterized in that, The semantic consistency constraint is configured to be determined by calculating the cosine distance between the hidden state vectors of the policy model and the reference model, as shown in the following mathematical expression: in, h represents semantic consistency constraints; θ h represents the hidden state vector of the current policy output by the policy model. ref This represents the hidden state vector of the reference strategy output by the reference model.

7. The method according to claim 4, characterized in that, The within-group advantage function in the objective function is configured as a risk-sensitive weighted function. This risk-sensitive weighted function is used to apply a larger update weight to candidate answers below a preset reward threshold than to candidate answer samples above the preset reward threshold. The mathematical expression is as follows: in, The within-group advantage function represents the importance of the i-th candidate answer in the current training batch; r i μ represents the reward value for the i-th candidate answer. r σ represents the average reward value of the current training batch; r δ represents the standard deviation of the reward value for the current training batch; ω represents the correction coefficient; ) represents the weight function.

8. The method according to claim 7, characterized in that, The reward value for the candidate answer is configured to be obtained through multi-dimensional evaluation, and the mathematical expression is as follows: The multi-dimensional evaluation includes the evaluation of the accuracy of the generated result, the evaluation of the format, the evaluation of the security, and the evaluation of the logical coherence. Where, r total This represents the reward value for the candidate answer; r acc The accuracy reward represents the accuracy of the candidate answer; r format This represents a format reward indicating whether the candidate answer conforms to a preset structure format; r safety The security reward represents the content compliance of the candidate answer; I(CoT) logic ) represents the logical indication function characterizing the logical coherence of the candidate answer; w1, w2, w3, and w4 represent the corresponding weights of the accuracy reward, the format reward, the security reward, and the logical indication function, respectively.

9. The method according to any one of claims 1 to 8, characterized in that, The acquisition of preprocessed training sample data includes: Obtain the original dialogue data, and manually annotate the original dialogue data to obtain initial sample data; The initial sample data is deeply cleaned, expanded, and / or quality-verified using a bidirectional distillation technique that combines the logical long-chain reasoning capability of the DeepSeek-R1 model with the generalization understanding capability of the GPT-5 model, thereby generating the training sample data.

10. A semantic understanding model training system, characterized in that, The system includes: The sample acquisition module is used to acquire preprocessed training sample data; the training sample data is customer service dialogue data in the ride-hailing scenario. The model training module includes a fine-tuning training unit and a reinforcement learning unit, used to perform a two-stage model training on the pre-trained language model using the training sample data, comprising a contrastive instruction fine-tuning stage and a reinforcement learning stage. The fine-tuning training unit is used to perform supervised fine-tuning training on the pre-trained language model based on a hybrid loss function with a supervised contrast regularization term introduced during the contrast instruction fine-tuning stage, so as to obtain a base model with the hybrid loss function minimized; the supervised contrast regularization term is used to bring the feature representation vectors of similar labeled samples closer together and push away the feature representation vectors of dissimilar labeled samples in the feature representation space of the pre-trained language model. The reinforcement learning unit is used in the reinforcement learning phase to initialize a policy model and a reference model using the base model, and to iteratively train the policy model based on an objective function containing dual constraints to obtain a semantic understanding model that minimizes the objective function; the dual constraints include a KL divergence constraint term for constraining the difference in output probability distribution between the policy model and the reference model, and a semantic consistency constraint term for pre-setting the difference in semantic representation between the policy model and the reference model in the hidden layer vector space.

Images