Expert preference alignment business processing method, device, equipment and medium

Abstract

The present invention provides a business processing method, apparatus, device, and medium for expert preference alignment. The method comprises: processing a target business question request using a business response model to obtain a business response result; training the business response model includes training a strategy model and a reward model, wherein the strategy model is pre-trained using knowledge injection, and the trained reward model is used to align and optimize expert preferences in a dynamic knowledge graph based on rewards. The present invention has the beneficial effects of reducing the number of manual calibrations by experts, maintaining the stability of the strategy model, and improving the accuracy and efficiency of business question answering.

Description

Technical Field

[0001] The present invention relates to the field of computer artificial intelligence technology, and in particular to a business processing method, device, equipment and medium for expert preference alignment. Background Art

[0002] Currently, real-time business processing systems in various fields generally face common challenges: in terms of automation capabilities, although routine business can be processed on a basic basis through intelligent tools, manual intervention is still required when encountering new and abnormal business, resulting in limited processing efficiency; and the existing mechanism for handling new and abnormal business has the problem of lengthy processes. Cross-departmental collaboration requires multi-level manual transfers and is handled in sections by different professional teams, which not only causes delayed response time but also leads to duplicate investment in human resources. In the manual operation link, the high-frequency processing of complex tasks is prone to decision-making errors and execution deviations, which directly affect the quality of handling. To meet these challenges, it is urgent to build an intelligent collaborative processing system, optimize the real-time data processing link through a dynamic diversion mechanism, and establish a knowledge management system with self-evolution capabilities to achieve the continuous transformation of abnormal handling experience into system capabilities.

[0003] In the field of artificial intelligence, especially in machine learning and reinforcement learning, human preference labels have become the core elements guiding the behavioral decisions of intelligent agents.

[0004] Back-end problem-solving by salespeople often faces time-consuming, labor-intensive, and inefficient tasks. Due to poor information flow and cumbersome system operations, the problem-solving process becomes extremely complex, forcing salespeople to spend significant time collecting data, verifying information, and conducting multiple communications. This not only impacts work efficiency but can also delay responses to customer requests, ultimately impacting customer satisfaction. Therefore, optimizing back-end processing and enhancing system intelligence are crucial to reduce the burden on salespeople and improve overall work efficiency. Summary of the Invention

[0005] The main purpose of the embodiments of the present invention is to propose a business processing method, device, equipment and medium for expert preference alignment, thereby improving the accuracy and efficiency of business question answering.

[0006] One aspect of the present invention provides a business processing method for expert preference alignment, characterized by comprising:

[0007] According to the query request of the target business, the business response model is used to process it and obtain the business response result;

[0008] The training steps of the business response model include:

[0009] Based on the first data set and the first base, pre-training is performed using knowledge injection, and then two-stage fine-tuning is performed to obtain an initial policy model;

[0010] Determine a dynamic knowledge graph using the initial strategy model and the first data set;

[0011] Performing expert-labeled answer and suggestion training using the second dataset and the second base to obtain a reward model;

[0012] Determine the question-answering results based on the initial strategy model and the dynamic knowledge graph, evaluate the question-answering results using a reward model, and perform strategy optimization and knowledge graph update based on the evaluation results;

[0013] The initial strategy model and the dynamic knowledge graph are aligned using a double-loop online iteration to obtain a business response model.

[0014] According to the business processing method for expert preference alignment, the step of training the business response model further includes performing preprocessing and data formatting on the first data set and the second data set:

[0015] The preprocessing of the first data set includes filtering samples of sensitive words using rule matching and deduplication using the MinHash LSH algorithm; the preprocessing of the second data set includes deleting samples where the length difference between the preferred response and the rejected response is greater than a preset multiple;

[0016] Formatting the first data set includes converting the sample data into a response-answer format; formatting the second data set includes converting the sample data into triplets including prompts, choices, and rejections, and generating adversarial samples based on the triplets.

[0017] According to the business processing method for expert preference alignment, pre-training is performed using knowledge injection based on the first data set and the first base, and then two-stage fine-tuning is performed to obtain an initial strategy model, including:

[0018] LLaMA-3-8B is used as the first base, and LoRA is used to update the self-attention layer of the first base. The first dataset is historical question-and-answer data without expert labels.

[0019] A contrastive learning strategy is used to initialize the parameters of the policy model in the embedding space of the domain knowledge graph;

[0020] Two-stage fine-tuning is performed through graph entity annotation and full-scale business data fine-tuning, where graph entity annotation includes inserting knowledge enhancement samples including graph entity annotation into question and answer data, and full-scale business data fine-tuning includes synchronously integrating the semantic parsing features of the dynamic knowledge graph.

[0021] According to the business processing method of expert preference alignment, the comparison learning strategy includes character-level comparison, semantic-level comparison, and logical-level comparison, wherein:

[0022] Character-level comparison The formula is:

[0023]

[0024] in, Indicates the calculation of the edit distance similarity between the response y and the sample s, where S is the total sample set;

[0025] Semantic level comparison The formula includes:

[0026]

[0027] in, Represented as the semantic embedding vector of the response text, Graph attention embedding represented as a standard solution in dynamic knowledge graphs, represented as randomly sampled embeddings of unrelated solutions, is the boundary margin;

[0028] Logic level comparison The formula is:

[0029]

[0030] in, is an indicator function that takes 1 when the response y violates the business logic and 0 otherwise; is the sequence of operation steps extracted from the response y, and the correct order is the standard operation process defined in the dynamic knowledge graph;

[0031] The total loss function of the ratio learning strategy for:

[0032]

[0033] in , , is the weight parameter, and , , Set it according to the business type of the target business.

[0034] According to the business processing method for expert preference alignment, the second data set and the second base are used to perform answer and suggestion training with expert labels to obtain a reward model, including:

[0035] The SFT model is used as the second base to replace the last layer as the linear head, perform answering and suggestions, and perform adversarial training through expert labels of answer answers and suggested answers to obtain a reward model, where the reward model is used to output a scalar reward value, where the scalar reward value includes basic semantics, knowledge matching, logical consistency and dynamic fusion.

[0036] According to the business processing method for expert preference alignment, the question-answering results are determined based on the initial strategy model and the dynamic knowledge graph, the question-answering results are evaluated using a reward model, and strategy optimization and knowledge graph update are performed based on the evaluation results, including:

[0037] The initial strategy model automatically associates nodes of the dynamic knowledge graph during each question and answer, obtaining a question and answer result that includes knowledge verification tags.

[0038] Sort and score the Q&A results, and add labels to the Q&A results based on the sorting and scoring;

[0039] Based on the labeled question-answering results, the initial strategy model is optimized based on expert preferences. , the formula is:

[0040]

[0041] in Ask questions for users, To respond correctly, For error response, is the initial strategy model, is the strategy model to be optimized, Negative expectations, Represents the distribution as expectations, is a hyperparameter;

[0042] Dynamic weight adjustment using deep semantic compensation reward function for:

[0043]

[0044] in To provide basic scoring; is a dynamic weight, and , is knowledge compensation, which is used to indicate the semantic matching degree between the response and the Top-N related nodes in the dynamic knowledge graph. N is the total number of nodes, i is the node identification number, len() is the length, and prompt is the input text.

[0045] Calculate knowledge matching:

[0046]

[0047] in, Representing graph attention networks; Node embedding, pre-trained using the TransE algorithm; To generate the response embedding, it is generated by the last hidden state of the initial policy model;

[0048] A hierarchical dynamic KL constraint mechanism, a dynamic KL-PID controller and a differentiated weight allocation method are adopted. The semantic space of the core entities in the embedded layer constraint domain is aligned with the initial policy model. The representation strength of the business rule association is maintained through the middle layer constraints. Moreover, new fault combinations are determined under the premise of meeting the knowledge boundary through the output layer constraints.

[0049] According to the business processing method for expert preference alignment, a double-loop online iteration is used to align the initial strategy model and the dynamic knowledge graph to obtain a business response model, including:

[0050] The parameters of the initial policy model are updated through an outer loop and KL divergence fluctuations are monitored. When the offset exceeds the preset offset value, the knowledge distillation protection mechanism is triggered.

[0051] The responses generated by the questions and answers are updated to the knowledge cache in real time through the inner loop, and independent knowledge partitions are established in the knowledge cache for high-frequency questions and answers;

[0052] Visually monitor knowledge coverage, policy offset, and knowledge validity, and use cross-version knowledge tracing to roll back knowledge history.

[0053] Another aspect of an embodiment of the present invention provides a business processing device for expert preference alignment, including:

[0054] The first module is used to process the target business's question request using the business response model to obtain a business response result;

[0055] The training of the business response model includes:

[0056] The second module is used to perform pre-training using knowledge injection based on the first data set and the first base, and then perform two-stage fine-tuning to obtain an initial policy model;

[0057] A third module is configured to determine a dynamic knowledge graph using the initial strategy model and the first data set;

[0058] A fourth module is configured to perform answer and suggestion training with expert labels using the second dataset and the second base to obtain a reward model;

[0059] The fifth module is used to determine the question-answering results based on the initial strategy model and the dynamic knowledge graph, evaluate the question-answering results using a reward model, and perform strategy optimization and knowledge graph update based on the evaluation results;

[0060] The sixth module is used to align the initial strategy model and the dynamic knowledge graph using a double-loop online iteration to obtain a business response model.

[0061] Another aspect of an embodiment of the present invention provides an electronic device, including a processor and a memory;

[0062] The memory is used to store programs;

[0063] The processor executes the program to implement the method described above.

[0064] Embodiments of the present invention further disclose a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the method described above.

[0065] The beneficial effects of the present invention are: low-quality historical data is used to train the initial policy model and build a dynamic knowledge graph library, and then the reward model is trained using expert-labeled data. The reward is fed back into the training process, and the semantic verification of the dynamic knowledge graph is used to compensate for the misjudgment of the reward model on multi-step logic. The hierarchical dynamic KL constraint mechanism is used to reduce the distortion rate of knowledge representation, while allowing innovative solutions generated by the output layer. The system reduces the number of expert manual calibrations during the iteration process, maintains the stability of the policy model, and improves the accuracy and efficiency of business questions and answers. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0067] Figure 1 4 is a flowchart of a business processing method for expert preference alignment according to an embodiment of the present invention.

[0068] Figure 2 It is a schematic diagram of the business question-and-answer process of an embodiment of the present invention.

[0069] Figure 3 This is a schematic diagram of the initial strategy model learning process of an embodiment of the present invention.

[0070] Figure 4 It is a schematic diagram of the strategy optimization and knowledge graph update process of an embodiment of the present invention.

[0071] Figure 5 4 is a schematic diagram of a business processing device for expert preference alignment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. In the subsequent description, suffixes such as "module," "component," or "unit" used to represent elements are used solely to facilitate the description of the present invention and have no specific meaning in themselves. Therefore, "module," "component," or "unit" may be used interchangeably. "First," "second," and the like are used solely to distinguish technical features and should not be construed as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In this subsequent description, the consecutive numbering of method steps is for ease of review and understanding. In conjunction with the overall technical solution of the present invention and the logical relationship between the various steps, adjusting the order of implementation of the steps does not affect the technical effects achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and should not be construed as limiting the present invention.

[0073] refer to Figure 1 ,in Figure 1 This is a flowchart of a business processing method for expert preference alignment according to an embodiment of the present invention. It includes but is not limited to steps S100 to S600:

[0074] S100: Based on the question request of the target business, a business response model is used to process the request and obtain a business response result.

[0075] In some embodiments, reference Figure 2 The business question and answer process diagram shown is a business that requires questions and answers, such as questions in the field of telecommunications card orders or medical response questions.

[0076] The training steps of the business response model include:

[0077] In some embodiments, the preprocessing of the first data set includes filtering samples of sensitive words using rule matching and performing deduplication using the MinHash LSH algorithm; the preprocessing of the second data set includes deleting samples whose length difference between the preferred response and the rejected response is greater than a preset multiple;

[0078] Formatting the first data set includes converting the sample data into a response-answer format; formatting the second data set includes converting the sample data into triplets including prompts, choices, and rejections, and generating adversarial samples based on the triplets.

[0079] In some embodiments, the first dataset is low-quality (i.e., does not include expert data) historical question-and-answer data. For example, for a single-domain question on a telecom card, preprocessing includes data collection, data cleaning, and data formatting and enhancement.

[0080] For data collection, the historical data for the initial strategy model comes from previously processed questions, but does not include expert judgment labels. This is one-question-one-answer data. The reward model then uses data from expert-judged questions, consisting of one question and two answers. When the expert label is 1, both answers are correct, and one answer complements the other. When the expert label is 0, one answer is correct and the other is incorrect.

[0081] Data cleaning includes:

[0082] (1) Initial strategy model training data: filter low-quality samples and directly filter invalid question-answer pairs. For example, if the question is: Mobile phone_Pre-disassembled machine in different places within the province|RES_Cloud network procurement and control ZTE SMS account opening_New_API|subService, the parameter cannot be empty, please contact the upstream system to reallocate the parameter. The answer is: 1+1=2, then it is a low-quality sample, and the question-answer pairs must correspond, otherwise it will be filtered; use rule matching to filter samples including sensitive words; deduplication processing: delete duplicate instruction pairs based on the MinHash LSH algorithm (Jaccard similarity>0.9).

[0083] (2) Reward model training data: consistency check, delete samples with inter-annotator consistency less than 80%. Length balancing, delete samples with a length difference of more than 4 times between the preferred response and the rejected response to avoid length bias dominating the sorting.

[0084] For data formatting and enhancement, it includes:

[0085] Data formatting and enhancement:

[0086] (1) Initial strategy model: uniformly converted into response-answer format

[0087] For example:

[0088] {

[0089] "input":"Mobile Phone|Subscription|RES_Cloud Network Procurement and Control Activation IMS Basic User Data_API|Execution Failed!ThisMSISDNalreadyexists.",

[0090] "output": "First verify the core network data platform. If there is junk data, reverse the issue first, delete the junk data in the reverse order, reissue the reverse order, and then re-execute the forward order. If the issue persists, consult the on-duty maintenance expert. The root cause analysis is junk data on the platform."

[0091] }

[0092] (2) Reward model: The original data is converted into the format of (prompt, chosen, rejected), where the suggestion corresponds to chosen and the answer corresponds to rejected. Here, the data with the expert label 0 is pre-converted as follows. The data with the label 1 rejected is given by similar matching of historical data.

[0093] For example:

[0094] {

[0095] "prompt": "RES_Cloud Network Acquisition and Control Delete IMS User Data_API|This IMSI does not exist",

[0096] "chosen": "Execute the reverse operation first, then the forward operation. If it succeeds, it's ok. If the reverse operation fails, check the core network data. If it is possibly occupied by another number, cancel the order and regenerate the short code." #Preferred response

[0097] "rejected": "Cancel order and report that the IMSI does not exist" # Reject response

[0098] }

[0099] The training data randomly swaps the chosen and rejected positions of 40% of the samples to prevent the model from favoring the responses that appear first, enhances negative samples, and uses the weaknesses of the current reward model to generate adversarial samples (such as deliberately generating grammatically correct but logically incorrect responses as rejected).

[0100] In some embodiments, the first dataset used for training the policy model is configured as follows:

[0101] Training set: 95%

[0102] Validation set: 5%.

[0103] Test set: Use an independent evaluation set that does not participate in training. Packing and chunking: Use packing technology to concatenate short samples to a fixed length (such as 4096 tokens) to reduce padding calculation waste.

[0104] The second dataset configuration for training the reward model is as follows:

[0105] Training set: 70%

[0106] Validation set: 20%

[0107] Test set: 10%.

[0108] S200 , performing pre-training using a knowledge injection method based on the first data set and the first base, and then performing two-stage fine-tuning to obtain an initial strategy model.

[0109] In some embodiments, reference Figure 3 The initial strategy model learning process diagram shown in FIG. 1 includes but is not limited to steps S310 to S330:

[0110] S310, using LLaMA-3-8B as the first base, and using LoRA to update the self-attention layer of the first base, the first dataset is the historical question-and-answer data without expert labels;

[0111] S320, using a contrastive learning strategy to initialize the parameters of the strategy model in the embedding space of the domain knowledge graph;

[0112] S330: Perform two-stage fine-tuning through graph entity annotation and full-scale business data fine-tuning. Graph entity annotation includes inserting knowledge-enhanced samples containing graph entity annotations into the question-answer data, and full-scale business data fine-tuning includes synchronously integrating semantic parsing features of the dynamic knowledge graph.

[0113] It is understandable that the model of the first base can be customized according to the business type.

[0114] In some embodiments, the comparison learning strategy includes character-level comparison, semantic-level comparison, and logical-level comparison, wherein:

[0115] Character-level comparison The formula is:

[0116]

[0117] in, Indicates the calculation of the edit distance similarity between the response y and the sample s, where S is the total sample set;

[0118] Semantic level comparison The formula includes:

[0119]

[0120] in, Represented as the semantic embedding vector of the response text, Graph attention embedding represented as a standard solution in dynamic knowledge graphs, represented as randomly sampled embeddings of unrelated solutions, is the boundary margin;

[0121] Logic level comparison The formula is:

[0122]

[0123] in, is an indicator function that takes 1 when the response y violates the business logic and 0 otherwise; is the sequence of operation steps extracted from the response y, and the correct order is the standard operation process defined in the dynamic knowledge graph;

[0124] Than the total loss function of the learning strategy for:

[0125]

[0126] in , , is the weight parameter, and , , Set it according to the business type of the target business, for example , , They are 0.3, 0.5, and 0.2 respectively.

[0127] Among them, the base model is: LLaMA-3-8B, the fine-tuning method adopts LoRA (r=64, alpha=16), only the attention layer is updated, and the three-layer contrastive learning objective as shown in the example above is constructed during the initial policy model fine-tuning stage.

[0128] In some embodiments, the knowledge-enhanced SFT initial policy model is constructed using an SFT (Supervised Fine-Tuning) initial policy model: This model is fine-tuned using a large amount of low-quality data without expert labels (i.e., only one question and one answer without expert judgment). The model base can be selected as needed to serve as the initialization strategy, providing a starting point for policy optimization, referred to herein as the initial strategy. This initial policy model has online question-and-answer functionality and can receive and answer real-time questions online. The resulting new questions and answers are then uploaded to an external reward model for evaluation.

[0129] Among them, knowledge injection pre-training adopts a contrastive learning strategy to initialize the parameters of the strategy model in the embedding space of the domain knowledge graph, thereby enhancing the ability to understand domain concepts.

[0130] S300, determining a dynamic knowledge graph through an initial strategy model and a first data set.

[0131] In some embodiments, the construction of a domain knowledge graph (implemented through full-scale business fine-tuning) takes the telecommunications work order problem domain as an example, as follows:

[0132] (1) Domain knowledge modeling and data preprocessing: First, the knowledge system is defined based on the characteristics of the telecommunications work order domain, and a three-layer knowledge architecture is constructed: the core concept layer defines basic entities such as fault code, business type, equipment model, error type, and solution; the business rule layer establishes business relationship networks such as "equipment model-generation-error type" and "solution-repair-fault code"; and the dynamic instance layer records historical work order processing cases and their contextual characteristics.

[0133] Multi-source data collection: structured data sources, such as work order records, equipment parameter tables, and fault code libraries in the business system database; unstructured data sources, such as historical work order text records, solution documents, and expert experience manuals; and real-time data streams, including high-quality responses and user feedback generated during online iterations.

[0134] (2) Knowledge extraction and fusion (hybrid extraction technology): rule-driven extraction, for structured data, using regular expressions to match key fields; deep learning extraction, based on the BERT-BiLSTM-CRF model, extracting entities and relationships from work order text; context-aware extraction, building a dependency syntax tree to parse complex semantics (such as "API call failed due to parameter errors").

[0135] (3) Knowledge fusion strategy: Entity alignment, using a fuzzy matching algorithm based on MinHash to eliminate the differences in expressions such as "IMS user data deletion" and "delete IMS user data"; conflict resolution, establishing an expert review queue, and manually verifying candidate triples with a confidence level lower than 85%; weight allocation, setting knowledge weights based on the credibility of the data source (expert annotation > system log > user generation).

[0136] (4) Dynamic voting mechanism for knowledge conflict resolution: A confidence-based multi-expert voting system is introduced in the entity alignment stage. When a similar conflict is detected in the "question-solution" matching, the following verification is initiated:

[0137] Rule engine verification (40% weight) checks compliance with the hard constraints of the business rule library;

[0138] Semantic similarity verification (35% weight), calculating the BERT-Whitening similarity between the candidate solution and the dynamic knowledge graph;

[0139] Historical adoption rate verification (25% weight), which counts the frequency of successful application of the solution in past work orders;

[0140] Set the dynamic threshold τ=0.7+0.1 log(Nconflict+1), when the comprehensive score is lower than the threshold, expert intervention is triggered.

[0141] (5) Graph construction and quality verification (layered storage architecture): The basic graph library uses the Neo4j graph database to store core concepts and relationships in the field to ensure the efficiency of complex queries; the case knowledge base uses Elasticsearch to build vector indexes to support semantic retrieval of similar work orders; the real-time cache layer uses Redis to store frequently accessed knowledge, with a response delay of less than 10ms.

[0142] S400: Using the second data set and the second base to perform answer and suggestion training with expert labels, to obtain a reward model.

[0143] In some embodiments, the SFT model is used as the second base to replace the last layer as the linear head, and answers and suggestions are performed. Adversarial training is performed through expert labels of the answer and the suggestion answer to obtain a reward model, wherein the reward model is used to output a scalar reward value, wherein the scalar reward value includes basic semantics, knowledge matching, logical consistency and dynamic fusion.

[0144] In some embodiments, a multi-dimensional external reward model is trained: a small amount of high-quality expert-labeled data (containing a question and its corresponding two answers: an answer and a suggestion, with expert labels. When the label is 1, both answers are correct, and when the label is 0, only the suggestion is the correct answer) is used to train an external reward model. The SFT (Supervised Fine-Tuning) model is used as the base, and the last layer is replaced with a linear head. The scalar reward value is output and used after passing the accuracy test. After training, the reward model can be used to provide an external reward signal for our initial policy model.

[0145] The reward model includes: basic semantic reward, which is the scoring function of the original reward model for answer quality; knowledge matching evaluation, which calculates the semantic similarity between the response and the dynamic knowledge graph and captures fine-grained associations through the graph attention network; logical consistency verification, which uses a built-in business rule engine to perform logical verification; a dynamic fusion mechanism, which sets adaptive weights (such as 60% for basic scoring, 30% for knowledge matching, and 10% for logical verification) to automatically increase the knowledge matching weight to 50% for complex questions; and the introduction of adversarial sample training, which mixes 10% of artificially constructed knowledge conflict samples into each batch to enhance robustness.

[0146] S500 determines the question-answering results based on the initial strategy model and the dynamic knowledge graph, uses the reward model to evaluate the question-answering results, and performs strategy optimization and knowledge graph update based on the evaluation results.

[0147] refer to Figure 4 The schematic diagram of the strategy optimization and knowledge graph update process shown in the figure adopts a dual-loop online iteration engine to perform strategy optimization, knowledge graph update and business response model generation.

[0148] In some embodiments, determining a question-answering result based on an initial strategy model and a dynamic knowledge graph, evaluating the question-answering result using a reward model, and performing strategy optimization and knowledge graph update based on the evaluation result include the following steps:

[0149] The initial strategy model automatically associates the nodes of the knowledge graph each time a question is asked and answered, and obtains a question and answer result including a knowledge verification mark. For example, the prompt (question text) pool is sampled, all of which are derived from online questions asked by real users, and diversified responses (answers) are generated. In this embodiment of the present invention, 4 pairs of responses are generated for each question.

[0150] The Q&A results are sorted and scored, and labeled based on the sorting and scoring. For example, the reward model will sort and score the four pairs of responses, output a best answer and a worst answer after sorting, and label them accordingly. (During this process, if it is the initial iteration stage, a small number of experts can be added for sampling inspection to determine whether the output meets the expert level.) Specifically, the VLLM engine is used for parallel generation, with a throughput of 600 tokens / sec / GPU. Cache optimization is also performed, and prompt codes are cached in Redis to reduce repeated calculations.

[0151] Based on the labeled question-answering results, the initial strategy model is optimized based on expert preferences. , the formula is:

[0152]

[0153] in Ask questions for users, To respond correctly, For error response, is the initial strategy model, is the strategy model to be optimized, Negative expectations, Represents the distribution as expectations, is a hyperparameter.

[0154] The deep semantic compensation reward function of the embodiment of the invention adopts a knowledge graph embedding compensation mechanism, adds a semantic verification module on the basis of the original reward model, and dynamically adjusts the weight formula:

[0155]

[0156] in The original quality score (0-1 range) output by the reward model is used as the basic score to evaluate basic indicators such as the fluency and logic of the answer; (Dynamic weight) Dynamically calculated based on the complexity of the problem ; For knowledge compensation, the response is the semantic matching degree (cosine similarity weighted) of the Top-N related nodes in the dynamic knowledge graph, N is the total number of nodes, i is the node identification number, len() is the length, and prompt is the input text.

[0157] Based on the traditional reward model, a semantic verification module of dynamic knowledge graph is introduced to meet the special needs of aligning preferences of experts in vertical fields. In , when the problem complexity (calculated by input length and entity density) increases, the reward model weight is automatically reduced ( ), improve the knowledge matching weight (1- For example, in a telecom work order scenario, when a user asks a complex question involving "cross-provincial IMS user data deletion," the system automatically increases the knowledge matching weight to 50% and performs semantic verification through the "cross-provincial business processing rules" node in the dynamic knowledge graph, avoiding policy deviations caused by the reward model's misjudgment of complex logic.

[0158] Calculate knowledge matching:

[0159]

[0160] in, Representing graph attention networks; Node embedding, pre-trained using the TransE algorithm; To generate the response embedding, it is generated by the last hidden state of the initial policy model;

[0161] A hierarchical dynamic KL constraint mechanism, a dynamic KL-PID controller and a differentiated weight allocation method are adopted. The semantic space of the core entities in the embedded layer constraint domain is aligned with the initial policy model. The representation strength of the business rule association is maintained through the middle layer constraints. Moreover, new fault combinations are determined under the premise of meeting the knowledge boundary through the output layer constraints.

[0162] In some embodiments, the hierarchical dynamic KL constraint mechanism The formula is:

[0163]

[0164] stratification weights, ;

[0165] The constraint levels include:

[0166] Embedding layer constraints: ensure that the semantic space of core domain entities (such as "IMS user data") is aligned with the initial policy model to prevent concept drift.

[0167] Middle-layer constraints: Maintain the representation strength of business rule associations (such as the causal chain from "error type to solution") to avoid the destruction of the logical relationship of the dynamic knowledge graph.

[0168] Output layer constraints: Allow the generation layer to explore new solutions (such as new and unseen fault combinations) while adhering to knowledge boundaries.

[0169] The traditional KL constraint only acts on the output layer distribution and is difficult to constrain the knowledge representation within the model. Through differentiated weight distribution (embedding layer 0.5, middle layer 0.3, output layer 0.2), a balance between domain knowledge stability and generation flexibility is achieved.

[0170] Through the dynamic KL-PID controller, when the intermediate layer KL divergence is detected to exceed the threshold, the knowledge distillation protection is automatically triggered, forcing the policy model to relearn the key business rules in the dynamic knowledge graph. Dynamic KL-PID controller (hierarchical KL constraints, design of hierarchical KL control strategy):

[0171] Level 1 (KL < threshold 1): Keep the learning rate and batch size unchanged and train normally.

[0172] Level 2 (KL∈[threshold 1, threshold 2]): Proportional control (PID algorithm) is introduced to dynamically reduce the learning rate and increase the batch size (e.g., 512→1024) to suppress gradient variance.

[0173] Level 3 (KL > Threshold 2): Pause parameter updates and start the policy distillation phase. Use the output of the current policy to relabel some preference pairs and force alignment of the initial model distribution.

[0174] In some embodiments, the optimizer configuration uses AdamW, and the learning rate is adjusted from initial to dynamic.

[0175] In some embodiments, the dynamic training strategy includes:

[0176] Real-time gradient monitoring, calculating the moving average of the gradient variance of the first 200 steps. When the variance is lower than the threshold, the warm-up is terminated early and the cosine decay phase is entered.

[0177] Segmented cycle restart, splitting the 10,000-step main cycle into 5 sub-cycles (T = 2,000 steps each), resetting the learning rate to 50% of the current value at the end of each sub-cycle, and then continuing cosine decay;

[0178] Dynamic cycle length, adjusting the sub-cycle length based on the rate of change of verification loss - if the loss decreases below a threshold, the next sub-cycle is automatically shortened to 1500 steps, otherwise it is extended to 2500 steps;

[0179] Real-time memory monitoring: dynamically detects GPU memory usage during training and automatically adjusts the number of sub-batches (e.g., 512 → 256 or 768) to maximize throughput.

[0180] Adaptive label smoothing noise, dynamically adjusted based on training progress or KL divergence, allowing for more noise, such as 20%, in the early stages of training and reducing it later, to prevent overfitting while not hindering learning.

[0181] Rollback mechanism: if the KL divergence exceeds the threshold for three consecutive evaluations, it will roll back to the previous checkpoint (KL divergence constraints are used to prevent the policy model from deviating too far from the initial model);

[0182] Early stopping mechanism: stop after three consecutive rounds of verification when the loss decreases;

[0183] Model snapshot, saving EMA (exponential moving average) parameters for each round.

[0184] S600 uses a double-loop online iteration to align the initial strategy model and the dynamic knowledge graph to obtain a business response model.

[0185] In some embodiments, the double-loop online iteration includes:

[0186] The outer loop (policy iteration) aggregates 5,000 high-quality preference pairs every 2 hours to update model parameters, monitors KL divergence fluctuations in real time, and triggers the knowledge distillation protection mechanism when the offset is greater than 0.3;

[0187] Inner loop (knowledge iteration): After each response is generated, the knowledge cache is updated immediately. Independent knowledge partitions are established for high-frequency questions (>50 times / day) to improve retrieval efficiency.

[0188] Stability is ensured through a three-dimensional monitoring dashboard (knowledge coverage, strategy offset, and knowledge validity), a cross-version knowledge traceability mechanism is established, historical status rollback is supported at any time, and the reward value increase of the model in each iteration can be dynamically monitored on the Wandb platform. The optimal strategy can be selected as the final model strategy based on needs.

[0189] In some embodiments, in actual tests on telecom card applications, compared with the traditional RLHF method (single reward model + global KL constraint), the improvements of the present invention are as follows:

[0190] The deep semantic compensation reward function improves the accuracy of expert preference alignment for complex problems by 23.5% (from 71.2% to 94.7%), mainly due to the semantic verification of the dynamic knowledge graph compensating for the reward model's misjudgment of multi-step logic.

[0191] The hierarchical dynamic KL constraint mechanism reduces the knowledge representation distortion rate to 1.8% (compared to 12.3% for traditional methods), while allowing the output layer to generate 16.7% of innovative solutions (verified effective by experts).

[0192] The system reduced the number of expert manual calibrations by 78% during the iteration process, proving that the dynamic KL constraint and knowledge compensation mechanism effectively maintained the stability of the policy model.

[0193] Figure 5 FIG. 5 is a schematic diagram of a business processing device for expert preference alignment according to an embodiment of the present invention. The device includes a first module 510 , a second module 520 , a third module 530 , a fourth module 540 , a fifth module 550 , and a sixth module 560 .

[0194] Among them, the first module is used to use the business response model to process the question request of the target business and obtain the business response result; the training of the business response model includes: the second module is used to use the knowledge injection method to pre-train according to the first data set and the first base, and then perform two-stage fine-tuning to obtain the initial strategy model; the third module is used to determine the dynamic knowledge graph through the initial strategy model and the first data set; the fourth module is used to use the second data set and the second base to perform answer and suggestion training with expert labels to obtain the reward model; the fifth module is used to determine the question and answer results based on the initial strategy model and the dynamic knowledge graph, use the reward model to evaluate the question and answer results, and perform strategy optimization and knowledge graph update according to the evaluation results; the sixth module is used to align the initial strategy model and the dynamic knowledge graph using a double-loop online iteration to obtain the business response model.

[0195] Exemplarily, with the cooperation of the first to sixth modules in the device, the embodiment device can implement any of the aforementioned expert preference alignment business processing methods, that is, according to the question request of the target business, the business response model is used for processing to obtain a business response result; the training steps of the business response model include: pre-training is performed according to the first data set and the first base using a knowledge injection method, and then two-stage fine-tuning is performed to obtain an initial strategy model; the dynamic knowledge graph is determined through the initial strategy model and the first data set; the answer and suggestion training with expert labels is performed using the second data set and the second base to obtain a reward model; the question and answer results are determined according to the initial strategy model and the dynamic knowledge graph, the question and answer results are evaluated using the reward model, and strategy optimization and knowledge graph update are performed according to the evaluation results; the initial strategy model and the dynamic knowledge graph are aligned using a double-loop online iteration to obtain a business response model. The beneficial effects of the present invention are: low-quality historical data is used to train the initial policy model and build a dynamic knowledge graph library, and then the reward model is trained using expert-labeled data. The reward is fed back into the training process, and the semantic verification of the dynamic knowledge graph is used to compensate for the misjudgment of the reward model on multi-step logic. The hierarchical dynamic KL constraint mechanism is used to reduce the distortion rate of knowledge representation, while allowing innovative solutions generated by the output layer. The system reduces the number of expert manual calibrations during the iteration process, maintains the stability of the policy model, and improves the accuracy and efficiency of business questions and answers.

[0196] An embodiment of the present invention further provides an electronic device, the electronic device including a processor and a memory;

[0197] The memory stores a program;

[0198] The processor executes a program to perform the aforementioned business processing method for expert preference alignment; the electronic device has the function of carrying and running the software system for the business processing of expert preference alignment provided by an embodiment of the present invention, such as a personal computer, a minicomputer, a main frame, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or communicating with a charged particle tool or other imaging device, etc.

[0199] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the business processing method for expert preference alignment as described above.

[0200] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0201] Embodiments of the present invention also disclose a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned business processing method for expert preference alignment.

[0202] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0203] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0204] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0205] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting, or processing it in another suitable manner as necessary, and then storing it in a computer memory.

[0206] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0207] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0208] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0209] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A business processing method for expert preference alignment, characterized in that: include: According to the query request of the target business, the business response model is used to process it and obtain the business response result; The training steps of the business response model include: Based on the first data set and the first base, pre-training is performed using knowledge injection, and then two-stage fine-tuning is performed to obtain an initial policy model; Determine a dynamic knowledge graph using the initial strategy model and the first data set; Performing expert-labeled answer and suggestion training using the second dataset and the second base to obtain a reward model; Determine the question-answering results based on the initial strategy model and the dynamic knowledge graph, evaluate the question-answering results using a reward model, and perform strategy optimization and knowledge graph update based on the evaluation results; The initial strategy model and the dynamic knowledge graph are aligned using a double-loop online iteration to obtain a business response model, including: The parameters of the initial policy model are updated through an outer loop and KL divergence fluctuations are monitored. When the offset exceeds the preset offset value, the knowledge distillation protection mechanism is triggered. The responses generated by the questions and answers are updated to the knowledge cache in real time through the inner loop, and independent knowledge partitions are established in the knowledge cache for high-frequency questions and answers; Visually monitor knowledge coverage, policy offset, and knowledge validity, and use cross-version knowledge tracing to roll back knowledge history.

2. The business processing method for expert preference alignment according to claim 1, characterized in that: The step of training the business answer model further includes performing preprocessing and data formatting on the first data set and the second data set: The preprocessing of the first data set includes filtering sensitive word samples using rule matching and deduplication using the MinHashLSH algorithm; the preprocessing of the second data set includes deleting samples where the length difference between the preferred response and the rejected response is greater than a preset multiple; The formatting of the first data set includes converting the sample data into a response-answer format; the formatting of the second data set includes converting the sample data into triplets including prompts, choices, and rejections, and generating adversarial samples based on the triplets.

3. The business processing method for expert preference alignment according to claim 1, characterized in that: The method of pre-training using knowledge injection based on the first data set and the first base, and then performing two-stage fine-tuning to obtain an initial strategy model includes: LLaMA-3-8B is used as the first base, and LoRA is used to update the self-attention layer of the first base. The first dataset is historical question-and-answer data without expert labels. A contrastive learning strategy is used to initialize the parameters of the policy model in the embedding space of the domain knowledge graph; Two-stage fine-tuning is performed through graph entity annotation and full-scale business data fine-tuning, where graph entity annotation includes inserting knowledge enhancement samples including graph entity annotation into question and answer data, and full-scale business data fine-tuning includes synchronously integrating the semantic parsing features of the domain knowledge graph.

4. The business processing method for expert preference alignment according to claim 3, characterized in that: The contrastive learning strategy includes character-level contrast, semantic-level contrast, and logical-level contrast, where: Character-level comparison The formula is: in, Indicates the calculation of the edit distance similarity between the response y and the sample s, where S is the total sample set; Semantic level comparison The formula includes: in, Represented as the semantic embedding vector of the response text, Graph attention embedding represented as a standard solution in domain knowledge graph, represented as randomly sampled embeddings of unrelated solutions, is the boundary margin; Logic level comparison The formula is: in, is an indicator function that takes 1 when the response y violates the business logic and 0 otherwise; is the sequence of operation steps extracted from the response y, and the correct order is the standard operation process defined in the domain knowledge graph; The total loss function of the contrastive learning strategy for: in , , is the weight parameter, and , , Set it according to the business type of the target business.

5. The business processing method for expert preference alignment according to claim 1, characterized in that: The second data set and the second base are used to perform answer and suggestion training with expert labels to obtain a reward model, including: The SFT model is used as the second base to replace the last layer as the linear head, perform answering and suggestions, and perform adversarial training through expert labels of answer answers and suggested answers to obtain a reward model, where the reward model is used to output a scalar reward value, where the scalar reward value includes basic semantics, knowledge matching, logical consistency and dynamic fusion.

6. The business processing method for expert preference alignment according to claim 5, characterized in that: The step of determining the question-answering result based on the initial strategy model and the dynamic knowledge graph, evaluating the question-answering result using the reward model, and performing strategy optimization and knowledge graph updating based on the evaluation result includes: The initial strategy model automatically associates nodes of the dynamic knowledge graph during each question and answer, obtaining a question and answer result that includes knowledge verification tags. Sort and score the Q&A results, and add labels to the Q&A results based on the sorting and scoring; Based on the labeled question-answering results, the initial strategy model is optimized based on expert preferences. , the formula is: in Ask questions for users, To respond correctly, For error response, is the initial strategy model, is the strategy model to be optimized, Negative expectations, Represents the distribution as expectations, is a hyperparameter; Dynamic weight adjustment using deep semantic compensation reward function for: in To provide basic scoring; is a dynamic weight, and , is knowledge compensation, which is used to indicate the semantic matching degree between the response and the Top-N related nodes in the dynamic knowledge graph. N is the total number of nodes, i is the node identification number, len() is the length, and prompt is the input text. Calculate knowledge matching: in, Representing graph attention networks; Node embedding, pre-trained using the TransE algorithm; To generate the response embedding, it is generated by the last hidden state of the initial policy model; A hierarchical dynamic KL constraint mechanism, a dynamic KL-PID controller and a differentiated weight allocation method are adopted. The semantic space of the core entities in the embedded layer constraint domain is aligned with the initial policy model. The representation strength of the business rule association is maintained through the middle layer constraints. Moreover, new fault combinations are determined under the premise of meeting the knowledge boundary through the output layer constraints.

7. A business processing device for expert preference alignment, characterized in that: include: The first module is used to process the target business's question request using the business response model to obtain a business response result; The training of the business response model includes: The second module is used to perform pre-training using knowledge injection based on the first data set and the first base, and then perform two-stage fine-tuning to obtain an initial policy model; A third module is configured to determine a knowledge graph using the initial strategy model and the first data set; A fourth module is configured to perform answer and suggestion training with expert labels using the second dataset and the second base to obtain a reward model; The fifth module is used to determine the question-answering results based on the initial strategy model and the dynamic knowledge graph, evaluate the question-answering results using a reward model, and perform strategy optimization and knowledge graph update based on the evaluation results; The sixth module is used to align the initial strategy model and the knowledge graph using a double-loop online iteration to obtain a business response model, including: The parameters of the initial policy model are updated through an outer loop and KL divergence fluctuations are monitored. When the offset exceeds the preset offset value, the knowledge distillation protection mechanism is triggered. The responses generated by the questions and answers are updated to the knowledge cache in real time through the inner loop, and independent knowledge partitions are established in the knowledge cache for high-frequency questions and answers; Visually monitor knowledge coverage, policy offset, and knowledge validity, and use cross-version knowledge tracing to roll back knowledge history.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the business processing method for expert preference alignment according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the business processing method for expert preference alignment according to any one of claims 1 to 6.

Images