Expert preference alignment service processing method and device, equipment and medium

Abstract

The invention provides a business processing method and device for expert preference alignment, equipment and a medium, and the method comprises the steps: carrying out the processing through employing a business reply model according to a question request of a target business, and obtaining a business reply result; the training step of the business reply model comprises training of a strategy model and training of a reward model, the strategy model is pre-trained in a knowledge injection mode, and expert preference alignment and optimization of the dynamic knowledge graph are carried out based on rewards through the trained reward model. The method has the advantages that the number of manual calibration times of experts is reduced, the stability of the strategy model is maintained, and the accuracy and efficiency of business questions and answers are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer artificial intelligence technology, and particularly to a business processing method, device, equipment and medium for aligning expert preferences. Background Art

[0002] Currently, real-time business processing systems in various fields generally face common challenges: at the level of automation capabilities, although routine operations can be basically processed through intelligent tools, when encountering new abnormal operations, manual intervention is still required, resulting in limited processing efficiency; and in the existing mechanism for processing new abnormal operations, there is a problem of long processes. Cross-departmental collaboration requires multi-level manual transfer and segmented processing by different professional teams, which not only causes a lag in response time but also leads to repeated investment in human resources. In the link of manual operation, high-frequency processing of complex tasks is prone to decision errors and execution deviations, directly affecting the disposal quality. To address these challenges, it is urgent to build an intelligent collaborative processing system, optimize the real-time data processing link through a dynamic diversion mechanism, and at the same time establish a knowledge management system with self-evolution ability to realize the continuous transformation of abnormal disposal experience into system capabilities.

[0003] In the field of artificial intelligence, especially in machine learning and reinforcement learning, human preference labels have become the core elements guiding the behavior decision-making of agents.

[0004] When business personnel handle problems in the background, they often face the dilemma of time-consuming, laborious and inefficient. Due to poor information transmission and cumbersome system operations, the problem-solving process becomes extremely complicated, resulting in business personnel spending a lot of time collecting materials, checking information and communicating multiple times. This not only affects work efficiency but also may delay the response to customer needs, thereby affecting customer satisfaction. Therefore, it is particularly important to optimize the background processing process and improve the intelligence level of the system to reduce the burden on business personnel and improve the overall work efficiency. Summary of the Invention

[0005] The main purpose of the embodiments of the present invention is to propose a business processing method, device, equipment and medium for aligning expert preferences, which improves the accuracy and efficiency of business question answering.

[0006] One aspect of the present invention provides a business processing method for aligning expert preferences, which is characterized by including: Processing according to the question request of the target business by using a business reply model to obtain a business reply result; The training steps of the business reply model include: Performing pre-training in a knowledge injection manner according to a first data set and a first base, and then performing two-stage fine-tuning to obtain an initial policy model; Determining a dynamic knowledge graph through the initial policy model and the first data set; Use the second dataset and the second base to perform training on answers and suggestions with expert labels to obtain a reward model; Determine the Q&A result according to the initial policy model and the dynamic knowledge graph, evaluate the Q&A result using the reward model, and perform policy optimization and knowledge graph update according to the evaluation result; Use double-loop online iteration to align the initial policy model and the dynamic knowledge graph to obtain a business reply model.

[0007] According to the business processing method aligned with the expert preference, the steps of training the business reply model further include performing preprocessing and data formatting on the first dataset and the second dataset: The preprocessing of the first dataset includes performing sensitive word sample filtering using a rule matching method and performing deduplication processing using the MinHash LSH algorithm; the preprocessing of the second dataset includes deleting samples where the length difference between the preferred response and the rejection response is greater than a preset multiple; The formatting of the first dataset includes converting the sample data into a response-answer format; the formatting of the second dataset includes converting the sample data into a triple including a prompt, a selection, and a rejection, and generating adversarial samples according to the triple.

[0008] According to the business processing method aligned with the expert preference, where pre-training is performed using a knowledge injection method based on the first dataset and the first base, and then two-stage fine-tuning is performed to obtain an initial policy model, including: Using LLaMA-3-8B as the first base, updating the self-attention layer of the first base using LoRA, and the first dataset is the question-and-answer historical data without expert labels; Adopt a contrastive learning strategy to initialize the parameters of the embedding space of the domain knowledge graph during the pre-training of the initial policy model; Perform two-stage fine-tuning through graph entity annotation and full-scale business data fine-tuning. The graph entity annotation includes inserting knowledge-enhanced samples including graph entity annotation into the Q&A data, and the full-scale business data fine-tuning includes synchronously integrating the semantic parsing features of the dynamic knowledge graph.

[0009] According to the business processing method aligned with the expert preference, the contrastive learning strategy includes character-level contrast, semantic-level contrast, and logic-level contrast, where: Character-level contrast The formula is: Where represents calculating the edit distance similarity between the response y and the positive sample s, and S is the set of positive samples; Semantic-level contrast The formula includes: Wherein, is expressed as the semantic embedding vector of the response text, is expressed as the graph attention embedding of the standard solution in the dynamic knowledge graph, is expressed as the embedding of randomly sampled irrelevant solutions, is the boundary margin; Logical-level contrast The formula is: Wherein, is the indicator function, taking 1 when the response y violates the business logic and 0 otherwise; is the sequence of operation steps extracted from the response y, and the correct order is the canonical operation process defined in the dynamic knowledge graph; The total loss function of the ratio learning strategy is: Where , , are weight parameters, and , , are set according to the business type of the target business.

[0010] According to the business processing method aligned with the expert preferences, wherein the second dataset and the second base are used to perform the training of answers and suggestions with expert labels to obtain a reward model, including: Taking the SFT model as the second base to replace the last layer with a linear head, performing answers and suggestions, and performing adversarial training through the expert labels of the answer and the suggested answer to obtain a reward model, wherein the reward model is used to output a scalar reward value, and the scalar reward value includes basic semantics, knowledge matching degree, logical consistency and dynamic fusion.

[0011] According to the business processing method aligned with the expert preferences, wherein the Q&A result is determined according to the initial policy model and the dynamic knowledge graph, the reward model is used to evaluate the Q&A result, and the policy optimization and knowledge graph update are performed according to the evaluation result, including: During each Q&A through the initial policy model, the nodes of the dynamic knowledge graph are automatically associated to obtain a Q&A result including a knowledge verification mark; Sort and score the Q&A results, and add labels to the Q&A results according to the sorting and scoring; Perform policy optimization on the initial policy model according to expert preferences based on the Q&A results with added labels , the formula is: where is the user's question, is the correct response, is the wrong response, is the initial policy model, is the policy model to be optimized, represents negative expectation, represents the expectation with the distribution of ; is a hyperparameter; Adopt a deep semantic compensation reward function to perform dynamic weight adjustment which is: where is the basic score; is the dynamic weight, and , is the knowledge compensation, which is used to represent the semantic matching degree between the response and the Top-N related nodes in the dynamic knowledge graph. N is the total number of nodes, i is the node identification number, len() is the length, and prompt is the input text; Calculate the knowledge matching degree: where, represents the graph attention network; is the node embedding, which is pre-trained by the TransE algorithm; is the response embedding, which is generated by the last hidden state of the initial policy model; Adopt a hierarchical dynamic KL constraint mechanism, a dynamic KL-PID controller and a differential weight allocation method, and align the semantic space of the domain core entity with the initial policy model through the embedding layer, maintain the representation strength related to the business rules through the middle layer, and, determine new fault combinations through the output layer under the premise of conforming to the knowledge boundary.

[0012] According to the business processing method aligned with the expert preferences, in which a double-loop online iteration is used to align the initial policy model and the dynamic knowledge graph to obtain a business reply model, including: Aggregate the parameters of the initial policy model through the outer loop and monitor the KL divergence fluctuation. When the offset is higher than the preset offset value, trigger the knowledge distillation protection mechanism; Store the dynamic knowledge graph generated by the question and answer in the knowledge cache through the outer loop, and establish an independent knowledge partition for high-frequency questions and answers in the knowledge cache; Visualize and monitor the knowledge coverage, policy deviation, and knowledge effectiveness, and roll back the knowledge historical state using cross-version knowledge tracing.

[0013] Another aspect of the embodiments of the present invention provides a business processing device for aligning expert preferences, including: A first module for processing a question request of a target business using a business reply model to obtain a business reply result; The training of the business reply model includes: A second module for pre-training using a knowledge injection method based on a first data set and a first base, and then performing two-stage fine-tuning to obtain an initial policy model; A third module for determining a dynamic knowledge graph through the initial policy model and the first data set; A fourth module for performing training on answers and suggestions with expert labels using a second data set and a second base to obtain a reward model; A fifth module for determining a question-and-answer result based on the initial policy model and the dynamic knowledge graph, evaluating the question-and-answer result using the reward model, and performing policy optimization and knowledge graph update according to the evaluation result; A sixth module for aligning the initial policy model and the dynamic knowledge graph using double-loop online iteration to obtain a business reply model.

[0014] Another aspect of the embodiments of the present invention provides an electronic device, including a processor and a memory; The memory is used to store a program; The processor executes the program to implement the method described above.

[0015] The embodiments of the present invention also disclose a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the method described above.

[0016] The beneficial effects of the present invention are as follows: The initial policy model is trained using low-quality historical data to construct a dynamic knowledge graph library, and then the reward model is trained using data with expert labels. The reward feedback is fed back into the training process, and the semantic verification of the dynamic knowledge graph compensates for the misjudgment of the multi-step logic by the reward model. The hierarchical dynamic KL constraint mechanism is used to reduce the knowledge representation distortion rate, and at the same time, innovative solutions generated by the output layer are allowed. The system reduces the number of expert manual calibrations during the iteration process, maintains the stability of the policy model, and improves the accuracy and efficiency of business question answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of embodiments in conjunction with the accompanying drawings, where: Figure 1 is a schematic flow chart of a business processing method for aligning expert preferences according to an embodiment of the present invention.

[0018] Figure 2 is a schematic flow chart of a business Q&A process according to an embodiment of the present invention.

[0019] Figure 3 is a schematic flow chart of an initial policy model learning process according to an embodiment of the present invention.

[0020] Figure 4 is a schematic flow chart of a policy optimization and knowledge graph update process according to an embodiment of the present invention.

[0021] Figure 5 is a schematic diagram of a business processing device for aligning expert preferences according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. In the following description, suffixes such as "module", "component" or "unit" used to denote elements are only for the convenience of describing the present invention and have no specific meaning per se. Therefore, "module", "component" or "unit" can be used interchangeably. "First", "second", etc. are only used for the purpose of distinguishing technical features and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence of the indicated technical features. In the following description, the consecutive numbering of method steps is for the convenience of review and understanding. Combining the overall technical solution of the present invention and the logical relationship between each step, adjusting the implementation order between steps will not affect the technical effect achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as limiting the present invention.

[0023] Refer to Figure 1 , where Figure 1 is a schematic flow chart of a business processing method for aligning expert preferences according to an embodiment of the present invention. It includes but is not limited to steps S100 to S600: S100. According to the question request of the target business, use the business reply model for processing to obtain the business reply result.

[0024] In some embodiments, refer to Figure 2The schematic diagram of the business Q&A process shown, where the target business is the business that requires Q&A, such as questions in the field of telecom card orders or medical responses, etc.

[0025] The training steps of the business response model include: In some embodiments, it also includes preprocessing of the first dataset, including performing sensitive word sample filtering by using a rule matching method and performing deduplication processing by using the MinHash LSH algorithm; preprocessing of the second dataset includes deleting samples where the length difference between the preferred response and the rejection response is greater than a preset multiple. Formatting of the first dataset includes converting the sample data into a response-answer format; formatting of the second dataset includes converting the sample data into a triple including a prompt, a choice, and a rejection, and generating adversarial samples according to the triple.

[0026] In some embodiments, for the first dataset, it is historical question-and-answer data of low quality (i.e., not including expert data). Exemplarily, for questions in the field of telecom card orders, the preprocessing includes data collection, data cleaning, and data formatting and enhancement.

[0027] For data collection, first, the historical data of the initial policy model all come from historical processed questions, but do not contain expert judgment labels, and are question-and-answer data. Then, the data used by the reward model comes from those judged by experts, including one question and two answers. When the expert label is 1, both answers are correct, and one answer is a supplement to the other. When the expert label is 0, one answer is correct and the other is wrong.

[0028] For data cleaning, it includes: (1) Initial policy model training data: Filter low-quality samples, directly filter invalid question-and-answer pairs. For example, for the question: Mobile phone_Intra-provincial off-site pre-disassembly|RES_Cloud network acquisition and control ZTE SMS account opening_New_API|subService, the parameter cannot be empty, please contact the upstream system to reassign the parameter, and the answer: 1 + 1 = 2, it is a low-quality sample, and both the question and the answer must correspond, otherwise filter; use rule matching to filter samples including sensitive words; perform deduplication processing: Delete duplicate instruction pairs based on the MinHash LSH algorithm (Jaccard similarity > 0.9).

[0029] (2) Reward model training data: Consistency check, delete samples with an inter-annotator consistency less than 80%. Length balance, delete samples where the length difference between the preferred response and the rejection response is greater than 4 times to avoid length bias dominating the sorting.

[0030] For data formatting and enhancement, it includes: Data formatting and enhancement: (1)Initial policy model: uniformly converted to the response-answer format For example: { "input": "Mobile phone|Order|RES_Cloud Network Procurement and Control to Open IMS Basic User Data_API|Execution failed! This MSISDN already exists.", "output": "First verify the core network data platform. If there is garbage data, reverse it first. Delete the garbage data in the reverse order form, then re-dispatch the reverse order form, and then re-execute the forward order form. If it is not solved, consult the on-duty experts in the maintenance group; the root cause analysis is that there is garbage data in the platform." } (2)Reward model: Convert the original data to the (prompt, chosen, rejected) format, where the suggestion corresponds to chosen and the answer corresponds to rejected. Here, the data with the expert label of 0 is pre-converted as follows, and the rejected with the label of 1 is given by similar matching of historical data.

[0031] For example: { "prompt": "RES_Cloud Network Procurement and Control to Delete IMS User Data_API|This IMSI does not exist", "chosen": "Execute in reverse first and then forward. If successful, it's okay. If the reverse is not successful, check the core network data. If it may be occupied by other numbers, cancel the order and regenerate the short code", # Preferred response "rejected": "Cancel the order form and report that the IMSI does not exist to pass the order" # Rejected response } The positions of chosen and rejected of 40% of the samples in this training data are randomly swapped to prevent the model from being biased towards the response that appears first. Negative sample enhancement is used to generate adversarial samples using the weaknesses of the current reward model (such as deliberately generating responses with correct grammar but incorrect logic as rejected).

[0032] In some embodiments, the first data set for training the policy model is configured as follows: Training set: 95% Validation set: 5%.

[0033] Test set: Use an independent evaluation set and do not participate in training. Packing and chunking: Use the packing technology to splice short samples to a fixed length (such as 4096 tokens) to reduce the waste of padding calculation.

[0034] The second data set for training the reward model is configured as follows: Training set: 70% Validation set: 20% Test set: 10%.

[0035] S200. Based on the first data set and the first base, perform pre-training using the knowledge injection method, and then perform two-stage fine-tuning to obtain an initial policy model.

[0036] In some embodiments, refer to Figure 3 the schematic diagram of the initial policy model learning process shown, which includes but is not limited to steps S310 to S330: S310. Use LLaMA-3-8B as the first base, and use LoRA to update the self-attention layer of the first base. The first data set is the question-and-answer historical data without expert labels; S320. Adopt a contrastive learning strategy to initialize the parameters of the embedding space of the domain knowledge graph during the pre-training of the initial policy model; S330. Perform two-stage fine-tuning through graph entity annotation and full-scale business data fine-tuning. Among them, graph entity annotation includes inserting knowledge-enhanced samples including graph entity annotation in the question-and-answer data, and full-scale business data fine-tuning includes synchronously fusing the semantic parsing features of the dynamic knowledge graph.

[0037] It can be understood that the model of the first base can be selected according to the business type.

[0038] In some embodiments, the contrastive learning strategy includes character-level contrast, semantic-level contrast, and logical-level contrast, where: Character-level contrast The formula is: Among them, represents calculating the edit distance similarity between the response y and the positive sample s, and S is the set of positive samples; Semantic-level contrast The formula includes: Among them, represents the semantic embedding vector of the response text, represents the graph attention embedding of the standard solution in the dynamic knowledge graph, represents the embedding of randomly sampled irrelevant solutions, is the margin; Logical-level contrast The formula is: Among them, is the indicator function, which takes 1 when the response y violates the business logic and 0 otherwise; is the sequence of operation steps extracted from the response y, and the correct order is the standard operation process defined in the dynamic knowledge graph; The total loss function of the ratio learning strategy is: where , , are weight parameters, and , , are set according to the business type of the target business. For example , , are 0.3, 0.5, and 0.2 in sequence.

[0039] Among them, the base model: LLaMA-3-8B, the fine-tuning method uses LoRA (r = 64, alpha = 16), only updates the attention layer, and constructs a three-layer contrastive learning objective as shown in the above example during the fine-tuning stage of the initial policy model.

[0040] In some embodiments, the construction of the knowledge-enhanced SFT initial policy model is based on the construction of the SFT (Supervised Fine-Tuning) initial policy model: a large amount of data without expert labels (i.e., low-quality data with only question-and-answer and no expert evaluation) is used for fine-tuning. The model base can be selected according to requirements as the initial policy, providing a starting point for policy optimization, which is called the initial policy here. This initial policy model has an online Q&A function, can receive real-time questions online, and give answers. The new Q&A obtained will be uploaded to the external reward model for evaluation.

[0041] Among them, knowledge injection pre-training adopts a contrastive learning strategy to initialize the parameters of the policy model in the embedding space of the domain knowledge graph, enhancing the domain concept understanding ability.

[0042] S300, determine the dynamic knowledge graph through the initial policy model and the first dataset.

[0043] In some embodiments, for the construction of the domain knowledge graph (realized through full-scale business fine-tuning), taking the telecommunications work order problem domain as an example, the example is as follows: (1) Domain knowledge modeling and data preprocessing: First, the knowledge system definition is based on the characteristics of the telecommunications work order domain, and a three-layer knowledge architecture is constructed: the core concept layer defines basic entities such as fault codes, business types, device models, error types, and solutions; the business rule layer establishes business relationship networks such as "device model - generates - error type", "solution - repairs - fault code"; the dynamic instance layer records historical work order processing cases and their context features.

[0044] Multi-source data collection: Structured data sources, such as work order record tables, equipment parameter tables, and fault code libraries in business system databases; unstructured data sources, such as historical work order text records, solution documents, and expert experience manuals; real-time data streams, such as high-quality responses and user feedback generated during the online iteration process.

[0045] (2) Knowledge extraction and fusion (hybrid extraction technology): Rule-driven extraction, for structured data, using regular expressions to match key fields; deep learning extraction, based on the BERT-BiLSTM-CRF model, to extract entities and relationships from work order texts; context-aware extraction, constructing dependency syntax trees to parse complex semantics (such as "API call failed due to parameter error").

[0046] (3) Knowledge fusion strategies: Entity alignment, using a fuzzy matching algorithm based on MinHash to eliminate expression differences such as "IMS user data deletion" and "delete IMS user data"; conflict resolution, establishing an expert review queue to manually verify candidate triples with a confidence level lower than 85%; weight assignment, setting knowledge weights according to the credibility of data sources (expert annotation > system log > user-generated).

[0047] (4) Dynamic voting mechanism for knowledge conflict resolution: Introduce a multi-expert voting system based on confidence during the entity alignment stage. When detecting similar conflicts in "question - solution" matching, start the following verification: Rule engine verification (40% weight), checking whether it conforms to the hard constraints of the business rule library; Semantic similarity verification (35% weight), calculating the BERT-Whitening similarity between the candidate solution and the dynamic knowledge graph; Historical adoption rate verification (25% weight), counting the successful application frequency of this solution in past work orders; Set the dynamic threshold τ = 0.7 + 0.1 log(Nconflict + 1), when the comprehensive score is lower than the threshold, trigger expert intervention.

[0048] (5) Graph construction and quality verification (hierarchical storage architecture): Basic graph library, using the Neo4j graph database to store core domain concepts and relationships to ensure the efficiency of complex queries; case knowledge base, using Elasticsearch to build vector indexes to support semantic retrieval of similar work orders; real-time cache layer, storing frequently accessed knowledge based on Redis, with a response latency < 10ms.

[0049] S400, using the second dataset and the second base to perform training on answers and suggestions with expert labels to obtain a reward model.

[0050] In some embodiments, the SFT model is used as the second base to replace the last layer with a linear head to execute answers and suggestions, and adversarial training is performed through the expert labels of the answer and the suggested answer to obtain a reward model, where the reward model is used to output a scalar reward value, and the scalar reward value includes basic semantics, knowledge matching degree, logical consistency, and dynamic fusion.

[0051] In some embodiments, a multi-dimensional training external reward model is adopted: a small amount of high-quality data with expert labels (including questions and their corresponding two answers: answers and suggestions, and with expert labels, when the label is 1, both answers are correct, and when the label is 0, only the suggestion is the correct answer) is used to train an external reward model. Taking the SFT (Supervised Fine-Tuning) model as the base, the last layer is replaced with a linear head (Linear Head) to output a scalar reward value, and it is used after passing the accuracy test. After the reward model is trained, it can be used to provide an external reward signal for our initial policy model.

[0052] The reward model includes: basic semantic reward, which is the scoring function of the original reward model for the answer quality; knowledge matching degree evaluation, which calculates the semantic similarity between the response and the dynamic knowledge graph, and captures fine-grained associations through a graph attention network; logical consistency verification, which builds a built-in business rule engine for logical verification; dynamic fusion mechanism, which sets adaptive weights (such as 60% for basic scoring, 30% for knowledge matching, and 10% for logical verification) to automatically increase the knowledge matching weight to 50% for complex questions; introducing adversarial sample training, and mixing 10% of artificially constructed knowledge conflict samples in each batch to enhance robustness.

[0053] S500, determine the question-and-answer result according to the initial policy model and the dynamic knowledge graph, evaluate the question-and-answer result by using the reward model, and perform policy optimization and knowledge graph update according to the evaluation result.

[0054] Reference Figure 4 As shown in the schematic diagram of the policy optimization and knowledge graph update process, a double-loop online iterative engine is used for policy optimization, knowledge graph update, and generation of the business reply model.

[0055] In some embodiments, determining the question-and-answer result according to the initial policy model and the dynamic knowledge graph, evaluating the question-and-answer result by using the reward model, and performing policy optimization and knowledge graph update according to the evaluation result include the following steps: During each question and answer session, the initial policy model automatically associates with the nodes of the knowledge graph to obtain a question and answer result including a knowledge verification flag. For example, by sampling from the Prompt (question text) pool, all of which are from the online questions of real users, diverse responses (answers) are generated. In the embodiments of the present invention, 4 pairs of responses are generated for each question.

[0056] Sort and score the question and answer results, and add labels to the question and answer results according to the sorting and scoring. For example, the reward model will sort and score the 4 pairs of responses. After sorting, it will output a best answer and a worst answer, and add corresponding labels. (During this process, in the initial iteration stage, a small number of experts can be added for sampling inspection to determine whether the output meets the expert level). Specifically, use the vllm engine to generate in parallel, with a throughput of 600 tokens / sec / GPU; cache optimization, cache the prompt encoding to Redis to reduce repeated calculations.

[0057] Perform policy optimization on the initial policy model according to the expert preferences based on the question and answer results with added labels , the formula is: where is the user's question, is the correct response, is the wrong response, is the initial policy model, is the policy model to be optimized, represents the negative expectation, represents the expectation with the distribution of and is a hyperparameter.

[0058] The deep semantic compensation reward function of the invention embodiment adopts a knowledge graph embedding compensation mechanism, adding a semantic verification module on the basis of the original reward model. The dynamic weight adjustment formula: where is the basic score, the original quality score (in the range of 0-1) output by the reward model, which evaluates basic indicators such as the fluency and logic of the answer; (dynamic weight) is calculated dynamically according to the question complexity , is the knowledge compensation, the semantic matching degree (cosine similarity weighted) between the response and the Top-N relevant nodes in the dynamic knowledge graph, N is the total number of nodes, i is the node identification number, len() is the length, and prompt is the input text.

[0059] Based on the traditional reward model, a semantic verification module of dynamic knowledge graph is introduced to meet the special needs of aligning preferences of experts in vertical fields. In , when the problem complexity (calculated by input length and entity density) increases, the reward model weight is automatically reduced ( ), improve the knowledge matching weight (1- For example, in a telecom work order scenario, when a user asks a complex question involving "cross-provincial IMS user data deletion," the system automatically increases the knowledge matching weight to 50% and performs semantic verification through the "cross-provincial business processing rules" node in the dynamic knowledge graph, avoiding policy deviations caused by the reward model's misjudgment of complex logic.

[0060] Calculate knowledge matching: in, Representing graph attention networks; Node embedding, pre-trained using the TransE algorithm; To generate the response embedding, it is generated by the last hidden state of the initial policy model; A hierarchical dynamic KL constraint mechanism, a dynamic KL-PID controller and a differentiated weight allocation method are adopted. The semantic space of the core entities in the embedded layer constraint domain is aligned with the initial policy model. The representation strength of the business rule association is maintained through the middle layer constraints. Moreover, new fault combinations are determined under the premise of meeting the knowledge boundary through the output layer constraints.

[0061] In some embodiments, the hierarchical dynamic KL constraint mechanism The formula is: stratification weights,

[0062] Constraint levels include: Embedding layer constraints: ensure that the semantic space of core domain entities (such as "IMS user data") is aligned with the initial policy model to prevent concept drift.

[0063] Middle-layer constraints: Maintain the representation strength of business rule associations (such as the causal chain from "error type to solution") to avoid the destruction of the logical relationship of the dynamic knowledge graph.

[0064] Output layer constraints: Allow the generation layer to explore new solutions (such as new and unseen fault combinations) while adhering to knowledge boundaries.

[0065] The traditional KL constraint only acts on the output layer distribution and is difficult to constrain the knowledge representation within the model. Through differentiated weight distribution (embedding layer 0.5, middle layer 0.3, output layer 0.2), a balance between domain knowledge stability and generation flexibility is achieved.

[0066] Through the dynamic KL-PID controller, when the intermediate layer KL divergence is detected to exceed the threshold, the knowledge distillation protection is automatically triggered, forcing the policy model to relearn the key business rules in the dynamic knowledge graph. Dynamic KL-PID controller (hierarchical KL constraints, design of hierarchical KL control strategy): Level 1 (KL < threshold 1): Keep the learning rate and batch size unchanged and train normally.

[0067] Level 2 (KL∈[threshold 1, threshold 2]): Proportional control (PID algorithm) is introduced to dynamically reduce the learning rate and increase the batch size (e.g., 512→1024) to suppress gradient variance.

[0068] Level 3 (KL > Threshold 2): Pause parameter updates and start the policy distillation phase. Use the output of the current policy to relabel some preference pairs and force alignment of the initial model distribution.

[0069] In some embodiments, the optimizer configuration uses AdamW, and the learning rate is adjusted from initial to dynamic.

[0070] In some embodiments, the dynamic training strategy includes: Real-time gradient monitoring, calculating the moving average of the gradient variance of the first 200 steps. When the variance is lower than the threshold, the warm-up is terminated early and the cosine decay phase is entered. Segmented cycle restart, splitting the 10,000-step main cycle into 5 sub-cycles (T = 2,000 steps each), resetting the learning rate to 50% of the current value at the end of each sub-cycle, and then continuing cosine decay; Dynamic cycle length, adjusting the sub-cycle length based on the rate of change of verification loss - if the loss decreases below a threshold, the next sub-cycle is automatically shortened to 1500 steps, otherwise it is extended to 2500 steps; Real-time memory monitoring: dynamically detects GPU memory usage during training and automatically adjusts the number of sub-batches (e.g., 512 → 256 or 768) to maximize throughput. Adaptive label smoothing noise, dynamically adjusted based on training progress or KL divergence, allowing for more noise, such as 20%, in the early stages of training and reducing it later, to prevent overfitting while not hindering learning. Rollback mechanism: if the KL divergence exceeds the threshold for three consecutive evaluations, it will roll back to the previous checkpoint (KL divergence constraints are used to prevent the policy model from deviating too far from the initial model); Early stopping mechanism: stop after three consecutive rounds of verification when the loss decreases; Model snapshots, saving EMA (Exponential Moving Average) parameters in each round.

[0071] S600, which uses a double-loop online iteration to align the initial policy model and the dynamic knowledge graph to obtain a business reply model.

[0072] In some embodiments, the double-loop online iteration includes: The outer loop (policy iteration), which aggregates 5000 high-quality preference pairs every 2 hours to update the model parameters, monitors the KL divergence fluctuation in real time, and triggers the knowledge distillation protection mechanism when the offset > 0.3; The inner loop (knowledge iteration), which instantaneously updates the knowledge cache after each response generation, establishes an independent knowledge partition for high-frequency questions (>50 times / day) to improve the retrieval efficiency; Stability guarantee, through a three-dimensional monitoring dashboard (knowledge coverage, policy offset, knowledge validity), establishing a cross-version knowledge traceability mechanism to support the rollback of historical states at any time, and on the wandb platform, the reward value improvement of the model in each round of iteration can be dynamically monitored, and the best policy can be selected as the policy of the final model according to requirements.

[0073] In some embodiments, in the field test of telecom cards, compared with the traditional RLHF method (single reward model + global KL constraint), the improvement effects of the present invention are as follows: The deep semantic compensation reward function improves the expert preference alignment accuracy of complex problems by 23.5% (from 71.2% to 94.7%), mainly because the semantic verification of the dynamic knowledge graph compensates for the misjudgment of the multi-step logic by the reward model.

[0074] The hierarchical dynamic KL constraint mechanism reduces the knowledge representation distortion rate to 1.8% (12.3% for the traditional method), and at the same time allows the output layer to generate 16.7% of innovative solutions (verified effective by experts).

[0075] The system reduces the number of expert manual calibrations by 78% during the iteration process, proving that the dynamic KL constraint and the knowledge compensation mechanism effectively maintain the stability of the policy model.

[0076] Figure 5 It is a schematic diagram of the business processing device for expert preference alignment in the embodiments of the present invention. The device includes a first module 510, a second module 520, a third module 530, a fourth module 540, a fifth module 550 and a sixth module 560.

[0077] Among them, the first module is used to process the question request of the target business by using the business reply model to obtain the business reply result; the training of the business reply model includes: the second module is used to perform pre-training in a knowledge injection manner according to the first data set and the first base, and then perform two-stage fine-tuning to obtain the initial policy model; the third module is used to determine the dynamic knowledge graph through the initial policy model and the first data set; the fourth module is used to perform training on answers and suggestions with expert labels by using the second data set and the second base to obtain the reward model; the fifth module is used to determine the question-and-answer result according to the initial policy model and the dynamic knowledge graph, evaluate the question-and-answer result by using the reward model, and perform policy optimization and knowledge graph update according to the evaluation result; the sixth module is used to align the initial policy model and the dynamic knowledge graph by using double-loop online iteration to obtain the business reply model.

[0078] Exemplarily, with the cooperation of the first module to the sixth module in the device, the embodiment device can implement any one of the foregoing business processing methods for aligning expert preferences, that is, process the question request of the target business by using the business reply model to obtain the business reply result; the training steps of the business reply model include: perform pre-training in a knowledge injection manner according to the first data set and the first base, and then perform two-stage fine-tuning to obtain the initial policy model; determine the dynamic knowledge graph through the initial policy model and the first data set; perform training on answers and suggestions with expert labels by using the second data set and the second base to obtain the reward model; determine the question-and-answer result according to the initial policy model and the dynamic knowledge graph, evaluate the question-and-answer result by using the reward model, and perform policy optimization and knowledge graph update according to the evaluation result; align the initial policy model and the dynamic knowledge graph by using double-loop online iteration to obtain the business reply model. The beneficial effects of the present invention are: use low-quality historical data to train the initial policy model and construct a dynamic knowledge graph library, then use the data with expert labels to train the reward model, feedback according to the reward into the training process, use the semantic verification of the dynamic knowledge graph to compensate for the misjudgment of the multi-step logic by the reward model, adopt a hierarchical dynamic KL constraint mechanism to reduce the knowledge representation distortion rate, and at the same time allow innovative solutions generated by the output layer. The system reduces the number of expert manual calibrations during the iteration process, maintains the stability of the policy model, and improves the accuracy and efficiency of business question answering.

[0079] An embodiment of the present invention further provides an electronic device, which includes a processor and a memory; The memory stores a program; The processor executes a program to perform the foregoing business processing method for aligning expert preferences; the electronic device has the function of carrying and running the software system for the business processing of aligning expert preferences provided by the embodiments of the present invention. For example, a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or communicating with charged particle tools or other imaging devices, etc.

[0080] The embodiments of the present invention also provide a computer-readable storage medium, and the storage medium stores a program, and the program is executed by a processor to implement the business processing method for aligning expert preferences as described above.

[0081] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order mentioned in the operation diagrams. For example, depending on the functions / operations involved, two consecutive blocks shown may actually be executed substantially simultaneously or the blocks can sometimes be executed in the reverse order. In addition, the embodiments presented and described in the flowcharts of the present invention are provided by way of example for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of a larger operation are executed independently.

[0082] The embodiments of the present invention also disclose a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions so that the computer device executes the foregoing business processing method for aligning expert preferences.

[0083] In addition, although the present invention has been described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features described may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It can also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More precisely, considering the attributes, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the modules will be understood within the ordinary skill of an engineer. Therefore, those skilled in the art can implement the present invention as set forth in the claims without undue experimentation. It can also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, and the scope of the present invention is determined by the full scope of the appended claims and their equivalents.

[0084] When the above-mentioned functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs, etc., all kinds of media that can store program codes.

[0085] The logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a predefined sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device.

[0086] More specific examples (nonexhaustive list) of computer-readable media include the following: electrical connection parts with one or more wirings (electronic devices), portable computer disk cartridges (magnetic devices), random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fiber devices, and portable compact disc read-only memories (CDROMs). Additionally, a computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other appropriate processing when necessary, and then storing it in a computer memory.

[0087] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0088] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0089] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the claims and their equivalents.

[0090] The above is a specific description of the preferred embodiments of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A business processing method for aligning expert preferences, characterized in that, Including: Processing according to the question request of the target business using a business reply model to obtain a business reply result; The training steps of the business reply model include: Performing pre-training using a knowledge injection method based on a first dataset and a first base, and then performing two-stage fine-tuning to obtain an initial policy model; Determining a dynamic knowledge graph through the initial policy model and the first dataset; Performing answer and advice training with expert labels using a second dataset and a second base to obtain a reward model; Determining a question-and-answer result based on the initial policy model and the dynamic knowledge graph, evaluating the question-and-answer result using the reward model, and performing policy optimization and knowledge graph update according to the evaluation result; Aligning the initial policy model and the dynamic knowledge graph using a double-loop online iteration to obtain a business reply model.

2. The business processing method for aligning with expert preferences according to claim 1, wherein The training steps of the business reply model further include performing preprocessing and data formatting on the first dataset and the second dataset: The preprocessing of the first dataset includes performing sensitive word sample filtering using a rule matching method and performing deduplication processing using the MinHashLSH algorithm; the preprocessing of the second dataset includes deleting samples with a difference in length between the preferred response and the rejection response greater than a preset multiple; The formatting process of the first dataset includes converting the sample data into a response-answer format; the formatting of the second dataset includes converting the sample data into a triple including a prompt, a choice, and a rejection, and generating adversarial samples according to the triple.

3. The business processing method for aligning with expert preferences according to claim 1, wherein The performing pre-training using a knowledge injection method based on a first dataset and a first base, and then performing two-stage fine-tuning to obtain an initial policy model includes: Using LLaMA-3-8B as the first base, updating the self-attention layer of the first base using LoRA, and the first dataset is the question-and-answer historical data without expert labels; Adopting a contrastive learning strategy to initialize the parameters of the embedding space of the domain knowledge graph during the pre-training of the initial policy model; Performing two-stage fine-tuning through graph entity annotation and full-scale business data fine-tuning. Among them, graph entity annotation includes inserting knowledge enhancement samples including graph entity annotation into the question-and-answer data, and full-scale business data fine-tuning includes synchronously integrating the semantic parsing features of the dynamic knowledge graph.

4. The business processing method for expert preference alignment according to claim 3, wherein The contrastive learning strategy includes character-level contrast, semantic-level contrast, and logical-level contrast, where: Character-level comparison The formula is: Among them, represents calculating the edit distance similarity between the computed response y and the positive sample s, where S is the set of positive samples; Semantic-level comparison The formula includes: Among them, is represented as the semantic embedding vector of the response text, is represented as the graph attention embedding of the standard solution in the dynamic knowledge graph, is represented as the embedding of randomly sampled irrelevant solutions, is the boundary margin; Logical level comparison The formula is: Among them, is an indicator function, taking 1 when the response y violates the business logic and 0 otherwise; is the sequence of operation steps extracted from the response y, and the correct order is the standard operation process defined in the dynamic knowledge graph; The total loss function of the contrastive learning strategy is as follows: Among them , , are weight parameters, and , , are set according to the business type of the target business.

5. The business processing method for expert preference alignment according to claim 1, wherein The performing answer and advice training with expert labels using a second dataset and a second base to obtain a reward model includes: Using the SFT model as the second base to replace the last layer with a linear head, performing answers and advice, and performing adversarial training through the expert labels of the answer and advice answers to obtain a reward model. The reward model is used to output a scalar reward value, and the scalar reward value includes basic semantics, knowledge matching degree, logical consistency, and dynamic fusion.

6. The business processing method for aligning with expert preferences according to claim 5, wherein The determining a question-and-answer result based on the initial policy model and the dynamic knowledge graph, evaluating the question-and-answer result using the reward model, and performing policy optimization and knowledge graph update according to the evaluation result includes: When answering questions each time through the initial policy model, automatically associate the nodes of the dynamic knowledge graph to obtain the question-and-answer results including knowledge verification marks; Sort and score the question-and-answer results, and add labels to the question-and-answer results according to the sorting and scoring; Perform policy optimization on the initial policy model according to the expert's preference based on the Q&A results with added tags , and the formula is: Among them is the user's question, is the correct response, is the wrong response, is the initial policy model, is the policy model to be optimized, represents negative expectation, represents the expectation with distribution and is a hyperparameter; Execute dynamic weight adjustment using a deep semantic compensation reward function It is: wherein is the basic score; is the dynamic weight, and , is the knowledge compensation, used to represent the semantic matching degree between the response and the Top-N relevant nodes in the dynamic knowledge graph, N is the total number of nodes, i is the node identification serial number, len() is the length, and prompt is the input text; Calculate the knowledge matching degree: Among them, represents the graph attention network; is the node embedding, pre-trained using the TransE algorithm; is the response embedding, generated by the last hidden state of the initial policy model; Adopt a hierarchical dynamic KL constraint mechanism, a dynamic KL-PID controller and a differential weight allocation method, and align the semantic space of the domain core entity with the initial policy model through the embedding layer, maintain the representation strength associated with the business rules through the middle layer constraint, and determine new fault combinations through the output layer constraint under the premise of conforming to the knowledge boundary.

7. The business processing method for expert preference alignment according to claim 6, wherein The above-mentioned use of double-loop online iteration to align the initial policy model and the knowledge graph to obtain a business reply model, including: Through the outer loop, aggregate the parameters of the initial policy model and monitor the KL divergence fluctuation. When the offset is higher than the preset offset value, trigger the knowledge distillation protection mechanism; Through the outer loop, store the knowledge graph generated by the question-and-answer into the knowledge cache library, and establish an independent knowledge partition for high-frequency questions in the knowledge cache library; Visually monitor the knowledge coverage, policy offset and knowledge validity, and adopt a cross-version knowledge tracing method to roll back the knowledge historical state.

8. A business processing device with expert preference alignment, characterized in that Including: The first module is used to process the question request of the target business by using the business reply model to obtain the business reply result; The training of the business reply model includes: The second module is used to perform pre-training by using the knowledge injection method according to the first data set and the first base, and then perform two-stage fine-tuning to obtain the initial policy model; The third module is used to determine the knowledge graph through the initial policy model and the first data set; The fourth module is used to perform the training of answers and suggestions with expert labels by using the second data set and the second base to obtain the reward model; The fifth module is used to determine the question-and-answer results according to the initial policy model and the dynamic knowledge graph, evaluate the question-and-answer results by using the reward model, and perform policy optimization and knowledge graph update according to the evaluation results; The sixth module is used to align the initial policy model and the knowledge graph by using double-loop online iteration to obtain the business reply model.

9. An electronic device, characterized in that, Including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the business processing method for expert preference alignment as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by the processor to implement the business processing method for expert preference alignment as described in any one of claims 1-7.

Images