Large model training method and device, equipment and storage medium

Abstract

The invention provides a large model training method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of natural language processing, information retrieval and large models. According to the specific implementation scheme, a knowledge conflict training set is constructed, wherein the knowledge conflict training set comprises structured data which corresponds to sample input and is provided with knowledge conflict processing flow labels; utilizing the knowledge conflict training set to supervise and finely adjust the first model to obtain an intermediate model; based on a preset reward mechanism, performing quality scoring on a knowledge conflict processing flow generated by the intermediate model for sample input; and training the intermediate model by adopting a reinforcement learning mode based on the quality score to obtain a target model. According to the scheme of the embodiment of the invention, preliminary training is carried out by utilizing the data set with conflict processing flow labels, and deep optimization is carried out through a reward mechanism and reinforcement learning, so that the obtained target model can systematically solve knowledge conflicts.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to natural language processing, information retrieval, and large model technology. Specifically, it relates to a training method, training device, equipment, and storage medium for a large model that solves the knowledge conflict problem in a retrieval-enhanced generation system. Background Technology

[0002] In recent years, generative artificial intelligence technologies, represented by Large Language Models (LLMs), have made significant progress and demonstrated powerful capabilities in various tasks such as knowledge question answering and content creation. However, LLMs also have some inherent limitations, such as the untimely updating of their internal knowledge and the potential to generate "illusionary" content that does not conform to the facts when answering knowledge-intensive or factually demanding questions.

[0003] To alleviate the above problems, retrieval-augmented generation (RAG) is proposed. RAG (Related Aspects of Knowledge) technology is widely used. The core idea of ​​this technology is to retrieve information fragments related to the user's question from one or more external knowledge bases (such as internal enterprise documents, the Internet, etc.) before the large model generates an answer, and provide this information as context to the large model, thereby helping the model generate a more accurate and timely answer. Summary of the Invention This disclosure provides a method, apparatus, device, and storage medium for training large models.

[0004] According to one aspect of this disclosure, a method for training a large model is provided, comprising: Construct a knowledge conflict training set, which contains structured data corresponding to the sample input and labeled with knowledge conflict handling procedures; The first model was fine-tuned under supervision using a knowledge conflict training set to obtain an intermediate model. Based on a preset reward mechanism, the quality score of the knowledge conflict handling process generated by the intermediate model in response to sample input is evaluated. Based on the quality score, reinforcement learning is used to train the intermediate model to obtain the target model.

[0005] According to another aspect of this disclosure, a training apparatus for a large model is provided, comprising: The building module is used to build a knowledge conflict training set, which contains structured data corresponding to the sample input and labeled with the knowledge conflict handling process. The supervised fine-tuning module is used to perform supervised fine-tuning of the first model using a knowledge conflict training set to obtain an intermediate model. The scoring module is used to score the quality of the knowledge conflict handling process generated by the intermediate model in response to the input question, based on a preset reward mechanism. The reinforcement learning module is used to train the intermediate model using reinforcement learning based on quality scores to obtain the target model.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in any one of the embodiments of this disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method according to any one of the embodiments of this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method according to any one of the embodiments of this disclosure.

[0009] According to the scheme of this disclosure embodiment, a dataset with conflict resolution process annotations is used for initial training, and then deep optimization is performed through reward mechanism and reinforcement learning, so that the target model can systematically resolve knowledge conflicts.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating a training method for a large model according to an embodiment of this disclosure; Figure 2 This is a flowchart illustrating the data construction phase provided in an embodiment of this disclosure; Figure 3 This is a flowchart illustrating the two-stage training phase provided in an embodiment of this disclosure; Figure 4This is a schematic diagram of the structure of a training device for a large model according to an embodiment of the present disclosure; Figure 5 This is a block diagram of an electronic device used to implement embodiments of the present disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] In related technologies, the introduction of external knowledge into RAG systems also brings new technical challenges, namely the knowledge conflict problem. When a model integrates its internal parameterized knowledge with retrieved external knowledge, inconsistencies or contradictions may arise. These knowledge conflicts may occur between internal and external knowledge, or between different external knowledge sources. This conflicting information can interfere with the model's judgment, thereby affecting the accuracy and reliability of the generated content.

[0014] To address knowledge conflicts and related issues in RAG systems, a common technical approach is conflict detection and handling. For example, after obtaining reference knowledge points related to the information to be detected, the two are compared to identify factual, rule, or logical conflicts. For conflicts between model internal parameters and external context in single-hop question answering, confidence levels are calculated using methods such as information entropy, external evaluator scoring, self-sampling scoring, context scoring, query relevance scoring, and self-assessed confidence. Based on the confidence level, decisions are made regarding whether to use the context retrieved by RAG or to assign weights to internal and external knowledge. However, such one-off conflict resolution strategies are difficult to apply to complex scenarios with multiple iterations.

[0015] Some studies have shown that there is no clear correlation between a model’s confidence and its ability. Some large models are powerful and confident, tending to trust their internal knowledge, while others are powerful but not confident, tending to trust external context. Therefore, the same confidence method may perform very differently on different models.

[0016] In RAG retrieval within enterprise office scenarios (e.g., document question-and-answer scenarios in super assistants), especially in multi-hop question-and-answer scenarios, conflicts arising from multiple knowledge sources are frequently encountered. These conflicts include those caused by knowledge illusions within the model, conflicts between internal model knowledge and external context, and conflicts between different external contexts. In different scenarios, it is necessary to identify specific types of knowledge conflicts and adaptively generate resolution paths to reduce the harmful impact of erroneous or incomplete knowledge on the results. Therefore, an efficient and universally applicable knowledge conflict resolution framework can effectively improve the accuracy of knowledge question answering.

[0017] In order to at least partially solve one or more of the above-mentioned problems and other potential problems, the embodiments of this disclosure provide a method for training a large model. By utilizing the technical solutions of the embodiments of this disclosure, the obtained target model can systematically resolve knowledge conflicts and improve the accuracy and consistency of knowledge question answering.

[0018] This disclosure provides a method for training a large model, which can be run on electronic devices with computing capabilities, such as servers, cloud computing platforms, or distributed computing clusters. Figure 1 This is a flowchart illustrating the training method of the model provided according to embodiments of this disclosure. Figure 1 As shown, the method includes the following steps: S110: Construct a knowledge conflict training set. This set contains structured data corresponding to the sample inputs, labeled with knowledge conflict handling procedures.

[0019] In this embodiment of the disclosure, the knowledge conflict training set can be understood as a specific dataset used for model training and fine-tuning. This training set consists of a large number of "sample input-structured data" pairs. Wherein: Sample input can refer to a natural language query submitted by a user to the system, such as "When did Company X acquire Company Y?". Sample input can be a pre-set question with a standard answer.

[0020] Structured data can be a complete text sequence that details the key parts of the process from receiving a question to generating the final answer, and is formatted using a predefined knowledge conflict resolution process annotation. These annotations enable the model to clearly identify different stages such as thinking, retrieval, and conflict analysis during training.

[0021] Knowledge conflict resolution process annotations can be a set of predefined special labels or tags used to describe the steps a model takes to solve a problem.

[0022] Knowledge conflict can refer to factual contradictions, inconsistencies, or logical conflicts that exist between information from any source.

[0023] The goal of this step is to create a set of high-quality teaching examples. This involves not only showing the model the correct answer to a question, but also demonstrating the standard process for solving that problem, especially how to handle inconsistencies or knowledge conflicts. Specifically, a powerful model can be used to solve a complex problem using its complete process. This process includes: attempting to answer using internal knowledge, actively searching for knowledge when insufficient or uncertain, acquiring external knowledge, comparing internal and external knowledge to identify conflicts, analyzing and resolving these conflicts, and finally drawing a conclusion. This complete thought process is recorded as raw text and then formatted using predefined tags to form structured data.

[0024] Example: For the sample input: "What are the different types of knowledge conflicts in RAG technology?", a corresponding structured data example is shown below: “ <think> The user asked about the classification of RAG knowledge conflicts. Based on my internal knowledge, conflicts mainly fall into two categories: conflicts between internal and external knowledge.< / think> <internal> There is a conflict between the knowledge inside the model and the knowledge retrieved from external sources.< / internal> <think> This category may be incomplete; I need to search it.< / think> <search> RAG knowledge conflict types< / search> <information> Document A retrieved indicates that RAG conflicts include: 1. Contextual and parameterized knowledge conflicts; 2. Contextual knowledge conflicts; 3. Parameterized knowledge conflicts.< / information> <think> The retrieved information is more comprehensive.< / think> <conflict> Internal knowledge (only 1 type) is inconsistent with external knowledge (3 types), and external knowledge is more complete.< / conflict> <answer> There are three main types of knowledge conflicts in RAGs: context-parameter knowledge conflict, context-internal knowledge conflict, and parameter-internal knowledge conflict.< / answer> " Pairing and combining a large number of such "sample inputs" and "structured data" creates a knowledge conflict training set.

[0025] S120: Use the knowledge conflict training set to perform supervised fine-tuning of the first model to obtain the intermediate model.

[0026] In this embodiment of the disclosure, the first model can be a pre-trained basic generative model with general content generation capabilities, such as a large language model. This model is not limited to processing text, but can also be a model capable of understanding and generating multi-modal data (such as text, images, audio, etc.), such as a multimodal large model.

[0027] Supervised fine-tuning (SFT) is a common model training technique that teaches a model how to generate answers in a specific format for a particular type of question by showing it a pair of "questions" and "standard answers".

[0028] An intermediate model refers to a model that, after being trained by SFT, has initially mastered the ability to resolve knowledge conflicts according to a specific process and format (i.e., the format of structured data).

[0029] In this step, the knowledge conflict training set constructed in S110 is used as training data. Sample inputs are used as model inputs, and the corresponding structured data is used as the expected output. Through SFT training, the first model learns this complete chain of "thinking-retrieval-conflict detection-response" and the specific label format.

[0030] Example: During training, the model is fed the sample input: "What are the different types of knowledge conflicts in RAG technology?", and trained to generate the complete structured data segment shown in example S110. After training on a large amount of such data, the first model becomes an intermediate model. When it encounters a new problem, it will attempt to solve it using a similar structured process.

[0031] S130: Based on a preset reward mechanism, the quality score is given to the knowledge conflict handling process generated by the intermediate model in response to sample input.

[0032] In this embodiment, the preset reward mechanism can be one or a set of rules, functions, or models used to evaluate the quality of the model's output. The purpose of this mechanism is to provide a clear optimization objective for reinforcement learning. The knowledge conflict resolution process refers to the (possibly labeled) response process that the intermediate model autonomously generates when faced with sample input. The quality score is a quantitative score (e.g., a value between 0 and 1) given by the preset reward mechanism for the knowledge conflict resolution process generated by the model; the higher the score, the better the quality of the process.

[0033] In this step, the intermediate model generates answers for a batch of sample inputs. Then, a pre-defined reward mechanism evaluates and scores these generated responses.

[0034] Example: The intermediate model generates process A for the sample input: "When did Company X acquire Company Y?": <internal> X acquired Y in 2020< / internal> <answer> 2020< / answer> (This is an incorrect and incomplete answer.) A pre-set reward mechanism evaluates it, potentially awarding a low score (e.g., quality score = 0.1) due to lack of retrieval or an incorrect answer. The model then generates process B: “ <internal> ...< / internal> <search> ...< / search> <conflict> ...< / conflict> <answer> 2021< / answer> (This is a correct and complete answer.) A pre-set reward mechanism evaluates it, potentially awarding a high score for thorough retrieval, proper conflict handling, and correctness, such as a quality score of 0.9.

[0035] S140: Based on the quality score, reinforcement learning is used to train the intermediate model to obtain the target model.

[0036] In this embodiment, reinforcement learning (RL) is a machine learning paradigm. In this scenario, the intermediate model acts as an "agent," its generated knowledge conflict resolution process is its "action," and the quality score obtained in S130 is the "reward." The target model is the final output of this method, optimized through reinforcement learning, capable of resolving knowledge conflict problems with high quality.

[0037] In this step, a reinforcement learning algorithm is employed, using the quality score obtained in S130 as a reward signal. The algorithm adjusts the internal parameters of the intermediate model so that, when generating answers in the future, it is more inclined to generate knowledge conflict resolution processes that can obtain high-quality scores.

[0038] According to the scheme of this disclosure, a two-stage training approach—Supervised Fine-tuning using SFT and Reinforcement Learning (RL)—solves the problem that SFT alone cannot fully optimize the model's decision-making ability. The SFT stage teaches the model the basic process and format for handling conflicts, while the RL stage uses quality scoring as guidance to enable the model to execute the knowledge conflict resolution process more effectively and reliably, thereby learning a robust decision-making strategy rather than relying on unstable confidence levels. This results in a final target model with better accuracy and robustness when facing complex knowledge conflict problems.

[0039] It should be noted that related techniques often rely on the model's confidence score to determine whether to accept internal or external knowledge. However, this method has weak generalization capabilities because different models perform differently on confidence scores, leading to varying results for the same method on different models. Furthermore, compared to the one-off conflict resolution strategies in related techniques that struggle to handle complex multi-turn question-answering, the RL phase of this disclosure optimizes the entire decision chain, reduces the risk of cascading errors, and improves scalability for handling complex problems.

[0040] In one possible implementation, the step of constructing the knowledge conflict training set in S110 is specified in detail. The method includes: S201: Using a second model, based on internal and / or external knowledge, perform at least one round of processing on the sample input to obtain iteratively processed data.

[0041] In this embodiment, the second model can be an auxiliary model used to generate training data. To ensure the quality and richness of the generated data, the second model is typically a very powerful large model, which may be stronger than the first model. Iterative data processing refers to the text records of the original thought process, retrieval process, and response process simulated by the second model without labeling.

[0042] In this step, a sophisticated prompt is designed to guide the second model to act as a problem-solving expert. When faced with sample input, this expert thinks and provides answers, such as first invoking internal knowledge, then deciding whether to use external knowledge (e.g., a search engine), and analyzing and iterating on the retrieved information until a final answer is reached. This complete multi-round thinking and action process is fully recorded, forming iterative processing data.

[0043] S202: Perform structured annotation on the iteratively processed data to obtain structured data.

[0044] In this embodiment of the disclosure, structured annotation refers to formatting the iterative processing data generated in S201 according to a predefined label specification.

[0045] This step can be performed in several ways. For example, regular expressions or parsing scripts can be written to automatically identify keywords such as "step X" or "[execute search...]" in the iterative data and convert them into corresponding labels. Alternatively, another model (such as the second model itself) can be called, providing it with the iterative data and label specifications, allowing it to automatically complete the labeling.

[0046] According to the scheme of this disclosure embodiment, in S201, a powerful second model is used to generate iterative processing data rich in detailed steps, and then in S202, it is structured and annotated, thereby realizing the automated and batch construction of high-quality, high-complexity knowledge conflict training sets. This not only avoids the expensive cost of manual annotation, but also ensures that the training data contains a complete and high-quality reasoning and conflict resolution logic chain.

[0047] In one possible implementation, S201 utilizes a second model, based on internal and / or external knowledge, to perform at least one round of processing on the sample input to obtain iteratively processed data, further including the following steps: S301: Obtain the internal knowledge response generated by the second model in response to the sample input.

[0048] In this embodiment of the disclosure, internal knowledge response refers to the output result or answer obtained by the model when processing sample inputs without the aid of any external tools (such as a search engine), relying solely on the knowledge stored in its parameters (i.e., the knowledge learned during the pre-training phase).

[0049] S302: Based on the sample input or internal knowledge response, perform external knowledge retrieval to obtain external knowledge.

[0050] In this embodiment of the disclosure, the second model evaluates the internal knowledge response in S301. If the model determines that the internal knowledge is uncertain, incomplete, or that the question itself is time-sensitive (such as "What's the weather like today?"), it decides to initiate an external knowledge retrieval. External knowledge refers to information obtained from external knowledge sources such as search engines, databases, or APIs.

[0051] S303: Detect whether there is a knowledge conflict between internal knowledge response and external knowledge, or between multiple external knowledge sources.

[0052] In this embodiment of the disclosure, knowledge conflict refers to factual contradictions, inconsistencies, or logical conflicts that exist between information from different sources. In this step, a conflict detection agent can be used to compare the internal knowledge response of S301 with the external knowledge of S302. The conflict detection agent can be implemented using a second model or other external models.

[0053] Example: Internal knowledge response: "Liu XX is from Shanxi." External knowledge: "Liu XX was born in Beijing, his ancestral home is Henan, and he grew up in Yangquan, Shanxi..." The model detected a knowledge conflict: "The internal knowledge (from Shanxi) and the external knowledge (born in Beijing, ancestral home in Henan, grew up in Shanxi) are not completely consistent in their descriptions of place of origin, and there is a conflict." S304: In the event of a knowledge conflict, perform a new external knowledge retrieval to update the external knowledge.

[0054] Specific implementation process: When S303 detects a conflict, the model initiates a conflict resolution sub-process. For example, a key resolution method is to re-perform external knowledge retrieval to obtain more authoritative and detailed information to adjudicate the conflict.

[0055] S305: Re-answer the sample input based on the updated external knowledge to obtain iterative processing data.

[0056] In this embodiment, the second model now aggregates all information (internal knowledge, external knowledge retrieved in the first search, updated knowledge retrieved in the second search, etc.), performs a final comprehensive analysis and decision, and generates the final answer. The iterative processing data is a complete textual record of the entire process from S301 to S305.

[0057] Example: The iterative processing data (original text) generated after model synthesis analysis can be: Step 1: Internal knowledge suggests that Liu XX is from Shanxi.

[0058] Step 2: The first search revealed that he was born in Beijing, his ancestral home was Henan, and he grew up in Shanxi, which conflicted with internal knowledge.

[0059] Step 3: To resolve the conflict, perform a second search.

[0060] Step 4: Search and confirm that Liu XX's ancestral home is Henan, he was born in Beijing, and grew up in Yangquan, Shanxi.

[0061] Step 5: Conclusion: The statement "Shanxi native" is inaccurate but widely circulated because it refers to someone who grew up in Shanxi. However, a more accurate answer is that their ancestral home is Henan, and they were born in Beijing.

[0062] Step 6: Final Answer: Liu XX's ancestral home is Henan, he was born in Beijing, and grew up in Shanxi. It should be noted that if the model cannot answer based on internal knowledge, it can skip S301, directly retrieve external knowledge, and answer based on the external knowledge.

[0063] According to the scheme of this disclosure embodiment, a complete closed loop including "internal response - retrieval - conflict detection - re-retrieval - comprehensive answer" is defined through detailed steps S301 to S305. In particular, by actively executing the step of re-retrieving external knowledge to update external knowledge when a knowledge conflict is detected, and proactively seeking to supplement the contextual information, the problem of forcibly making incorrect decisions when information is insufficient is avoided, significantly improving the effectiveness of the final answer. This ensures that the generated iterative processing data (SFT data) is a complex cognitive process involving multi-step reasoning, fact checking, and conflict resolution, greatly improving the depth and quality of the training data.

[0064] In one possible implementation, the step of structuring the iterative processing data in S202 is defined in detail. This process involves converting the raw iterative processing data generated in S201 into structured data, including: S401: Use internal knowledge tags to label the internal knowledge responses in the iteratively processed data.

[0065] In this embodiment of the disclosure, the internal knowledge tag is a predefined marker (e.g.: <internal> and< / internal> This step aims to identify content generated by the model solely based on its own parameterized knowledge during iterative data processing.

[0066] Example: In the iterative processing data of S305, the part "Internal knowledge believes Liu XX is from Shanxi" will be labeled as: <internal> Liu XX is from Shanxi.< / internal> S402: Use retrieval information tags to annotate the external knowledge retrieval process in the iteratively processed data.

[0067] Specific implementation process: Retrieve information tags, which can be a set of tags (e.g.: <search> and< / search> Used to annotate query statements. <information> and< / information> (Used to label the returned knowledge). This step aims to identify the interaction process between the model and external knowledge bases (such as search engines).

[0068] Example: In the iterative processing data of S305, the section "The first search revealed that he was born in Beijing..." will be labeled as: <search> Liu XX's place of origin< / search> <information> Liu XX was born in Beijing in June 1963, with ancestral roots in Henan Province, and grew up in Yangquan, Shanxi Province...< / information> S403: Use conflict content tags to annotate the knowledge conflict analysis process in the iterative data processing.

[0069] In this embodiment of the disclosure, the conflicting content tag can be a set of tags (e.g.: <conflict> and< / conflict> Conclusions used to indicate conflicts <context> and< / context> (Analysis context used to label conflicts). This step aims to identify and analyze inconsistencies found by the model.

[0070] Example: <conflict> The internal knowledge (Shanxi native) and the external knowledge (born in Beijing, ancestral home in Henan) are inconsistent in their descriptions of place of origin.< / conflict> S404: Use answer labels to annotate the final answer in the iteratively processed data.

[0071] In this embodiment of the disclosure, the answer label is a predefined tag (e.g.: <answer> and< / answer> This step aims to clearly identify the final conclusions reached after the model has gone through the entire process.

[0072] Example: In the iterative processing of data in S305, the final answer, "Liu XX's ancestral home is Henan...", will be labeled as: <answer> Liu XX's ancestral home is Henan, but he was born in Beijing and grew up in Shanxi.< / answer> When dealing with knowledge conflicts, related technologies often employ a "black box" decision-making process, failing to explicitly represent conflicting knowledge. This results in many conflicts remaining unidentified and unresolved, leading to poor interpretability. According to the solution of this disclosure, through refined annotation in steps S401 to S404, unstructured iterative processing data is transformed into highly structured, machine-readable structured data. This annotation method enables the first model to clearly learn the specific syntax and boundaries corresponding to different operations (such as internal thinking, external search, conflict detection, and answer output) during the SFT stage. This forms the basis for the model to autonomously execute and optimize these complex processes during the subsequent RL stage.

[0073] In one possible implementation, a further quality control step is added to the process of constructing the knowledge conflict training set. Following S202, the method also includes: S203: Automatically filter structured data to retain structured data that simultaneously meets the following preset conditions.

[0074] In this embodiment, automatic filtering can be a script or program running after S202 to filter out low-quality or non-compliant structured data generated in stages S201-S202. Preset conditions are hard rules used for filtering and may include: Condition 1: The iterative data processing includes at least one knowledge conflict identification and resolution process.

[0075] Implementation: The filtering script will automatically detect whether the structured data contains at least one conflicting content tag (e.g., <conflict>If no conflict is detected during the generation of S201, the data entry will be discarded. This is to ensure that the training set focuses on the core task of "knowledge conflict".

[0076] Condition 2: The number of tool calls included in the iteratively processed data is not less than the preset number.

[0077] Implementation method: Tool calls can refer to external knowledge retrieval (i.e. <search>(The occurrence of the tag). The preset number of occurrences is a configurable threshold, such as 2 times. The script will count the occurrences. <search>If a tag appears less than twice (e.g., the answer is obtained after only one retrieval), the data may be considered too simple and will be discarded. This ensures that the model learns "multi-hop" or "complex" retrieval processes.

[0078] Condition 3: The final answer labeled in the structured data is consistent with the standard answer preset for the sample input.

[0079] Implementation: When constructing the SFT dataset, in addition to sample input, a "ground truth" is also required. The filtering script extracts the answer labels from the structured data. <answer>The model processes the text within the specified range and compares it to the standard answer (e.g., exact match or semantic similarity match such as BERTScore). If the final answer obtained by the model after a complex process in S201 is incorrect, the data will be discarded.

[0080] Example: Sample input: "When will X acquire Y?", Standard answer: "2021".

[0081] Data A: Contains <conflict>Tags, containing 2 times <search>Label, <answer>The content is "2021". This data will be retained.

[0082] Data B: Not included <conflict>Tag. This data is discarded (does not meet condition 1).

[0083] Data C: Contains <conflict>Tags, but only once. <search>Label (assuming a preset count of 2). This data is discarded (does not meet condition 2).

[0084] Data D: Contains <conflict>Tags, containing 2 times <search>Tags, but <answer>The content is "2020". This data is discarded (it does not meet condition 3).

[0085] According to the scheme of this disclosure embodiment, the automatic screening step in S203 greatly improves the data purity and quality of the knowledge conflict training set. This step ensures that all data used for SFT training simultaneously meets three key criteria: "contains conflict," "complex process" (multiple tool calls), and "correct answer." This allows the intermediate model to focus on learning how to resolve knowledge conflicts through complex and correct paths, avoiding the learning of incorrect or overly simplistic processing patterns. This lays a solid foundation for subsequent models to learn correct decision logic and reduce cascading errors during reasoning.

[0086] In one possible implementation, the steps of S130, which perform quality scoring on the knowledge conflict handling process generated by the intermediate model based on sample input, according to a preset reward mechanism, are specified in detail. Specifically, the steps include: S601: Generate multiple responses for sample input using an intermediate model.

[0087] In this embodiment of the disclosure, multiple answers refer to the intermediate model generating multiple different knowledge conflict handling processes for the same sample input by introducing randomness (e.g., setting the temperature parameter to be greater than 0, or using top-p sampling).

[0088] S602: Based on the preset reward mechanism, a comprehensive score is obtained by evaluating the knowledge conflict handling process in the generation of multiple answers.

[0089] In this embodiment of the disclosure, a preset reward mechanism (i.e., a reward model) evaluates the complete knowledge conflict handling process of each answer generated in S601 and gives a quality score.

[0090] The pre-set reward mechanism is not based on a single standard, but rather evaluates from multiple dimensions. These dimensions may include, but are not limited to: Search performance dimensions: Evaluate whether the query generated by the search tag is efficient, and whether the information in the information tag is relevant to the question and authoritative.

[0091] Conflict handling dimension: Evaluate whether the conflict label accurately identifies the conflict and whether the logic for resolving the conflict is reasonable.

[0092] Contextual sufficiency dimension: assesses whether the response provides sufficient contextual information.

[0093] Formatting specification dimension: Evaluate whether the generated text strictly adheres to the label format learned in the SFT stage.

[0094] Example: The reward model scores the three responses in S601: Answer A: Search performance (0.8), conflict resolution (NA), context (0.3), format (1.0) -> Quality score = 0.7 Answer B: Search performance (0.9), conflict resolution (NA), context (0.8), format (1.0) -> Quality score = 0.9 Answer C: Search performance (0.8), conflict resolution (NA), context (0.3), format (1.0), answer correctness (0) -> Quality score = 0.3 According to the scheme of this disclosure embodiment, multiple answers are generated in S601 and scored using the multi-dimensional reward mechanism in S602, providing a rich exploration space and refined reward signals for reinforcement learning. This enables the model not only to know which answer is better, but also to know why it is better (B has a more sufficient context), thereby enabling targeted optimization of its specific capabilities in search, conflict resolution, context supplementation, etc.

[0095] In one possible implementation, the steps for comprehensive scoring in S602 are specified in detail, particularly for dimensions that are difficult to quantify, and specifically include the following steps: S701: For abstract metrics in the conflict resolution or context adequacy dimensions, call external models for automated scoring.

[0096] In this embodiment of the disclosure, abstract metrics refer to evaluation criteria that cannot be measured by simple rules (such as keyword matching or whether the answer is correct), such as "the logicality of reasoning," "the rationality of the solution," and "whether the context is rich and relevant." The external model can be a more powerful and accurate "judge" model (such as GPT-4 or an equivalent model), which is independent of the intermediate model being trained.

[0097] In this step, when the reward mechanism needs to evaluate an abstract metric, it will not calculate it itself, but will package the corresponding text fragments generated by the intermediate model and send them to the external model (referee) for the referee to score.

[0098] Example: The intermediate model generates a knowledge conflict resolution process, which includes a conflict analysis text: " <conflict> Internal knowledge states that Company X acquired Company Y in 2020; external search A states 2021; and external search B (official announcement) also states 2021. The internal knowledge is outdated; B is the most credible source.< / conflict> "The reward mechanism needs to evaluate the 'conflict handling logic' of the content tagged with 'conflict.' Therefore, this conflict analysis text is scored using an external model."

[0099] According to the solution of this disclosure embodiment, by introducing a powerful external model as a "referee" in S701, the major problem of quantifying and automatically evaluating abstract metrics (such as logic and rationality) in reinforcement learning is solved. This enables intermediate models to be effectively trained and optimized on these high-level cognitive abilities without relying on expensive, slow, and subjective manual evaluation.

[0100] In one possible implementation, the steps for calling an external model for automated scoring in S701 are specified in detail, including the following steps: S801: Build a prompt message containing preset scoring criteria.

[0101] In this step, the prompt is an instruction sent to the external model (judge). To ensure that the judges give consistent and accurate scores, the instruction must include a clear "role definition" and "scoring criteria".

[0102] Example: The prompt message is as follows: "Role: You are a rigorous logic analyst. Task: Please evaluate the 'logic' of the following text in resolving knowledge conflicts. Scoring criteria:" 1 point (Logical confusion): Unidentified conflict or incorrect solution.

[0103] 3 points (Logic is acceptable): The conflict was identified, but the evidence was not fully utilized or the reasoning was too simplistic.

[0104] 5 points (Logically Sound): Accurately identifies conflicts and provides a clear reasoning process based on evidence (such as source credibility and timeliness). Requirement: Please return only a numerical score between 1 and 5. S802: Input the text content and prompts generated by the intermediate model that correspond to the abstract indicators into the external model.

[0105] Specific implementation process: This step involves combining the prompt message from S801 with the text content from the S701 example (i.e., " <conflict>The internal knowledge says... (that paragraph) is pieced together to form a complete input, and then sent to the external model (referee) via an API call.

[0106] Example: The complete input sent to the external model is: "Role: You are a rigorous logic analyst... (Full hints from S801)... Text to be evaluated: Internal knowledge says Company X acquired Y in 2020, external retrieval A says 2021, external retrieval B (official announcement) says 2021. Internal knowledge is outdated, B has the highest credibility." S803: Obtain the scoring results returned by the external model as the score of the abstract indicator.

[0107] Specific implementation process: After receiving and processing the input from S802, the external model (referee) will return a score according to the requirements in the prompt message.

[0108] Example: An external model analyzes the text to be evaluated and concludes that it "accurately identifies conflicts" and provides clear reasoning "based on evidence," meeting the "5-point" standard. Therefore, it returns a score of "5". This number "5" is then captured by the reward mechanism, used as the score for this abstract indicator, and included in the total quality score of S602.

[0109] According to the scheme of this disclosure embodiment, through the fine operation of S801~S803, a reproducible and automated "referee scoring" process is defined, so that the scoring of the external model is based on evidence, thereby providing a stable, high-quality and scalable reward signal for reinforcement learning.

[0110] In one possible implementation, the knowledge conflict resolution process annotation includes: Internal knowledge tags used to identify the knowledge within a model; Search query tags used to identify the search query and search information tags used to identify the search content; Conflict tags used to identify knowledge conflict phenomena and conflict content tags with related background information; Answer labels are used to identify the final answer in the output.

[0111] In this embodiment of the disclosure, the knowledge conflict resolution process annotation is a specific labeling system used for SFT training and RL stage format specification dimension evaluation. This system may specifically include: Internal knowledge tags: for example <internal> ...< / internal> This tag is used to identify the "thinking" or "answer" generated by the model solely from its parameterized knowledge base without any external retrieval.

[0112] Search query tags and search information tags: for example <search> ...< / search> and <information> ...< / information> The retrieval query tag is used to identify the query statement generated by the model and about to be sent to the search engine. The retrieval information tag is used to identify the knowledge fragments returned from the search engine and adopted by the model as context.

[0113] Conflicting tags and conflicting content tags: for example <conflict> ...< / conflict> and <context> ...< / context> Conflict labels are used to identify conclusions where the model explicitly identifies inconsistencies in information (e.g., "conflict between internal and external knowledge"). Conflict content labels are used to identify a detailed description of the model's specific analysis, background, and context regarding the conflict.

[0114] Answer tags: for example <answer> ...< / answer> This label is used to identify the final answer output by the model after it has gone through the complete "think-retrieve-conflict analysis" process.

[0115] According to the scheme of this disclosure embodiment, through this complete annotation system consisting of tags such as internal knowledge, retrieval, conflict, and answer, the first model can systematically learn the standard operating procedures for solving complex problems in the SFT stage. Each tag corresponds to an operable and evaluable step, which not only provides a clear learning objective for SFT, but also provides a verifiable basis for the multi-dimensional rewards (such as format specification dimension) in the RL stage.

[0116] In one possible implementation, reinforcement learning is used for training in S140, specifically including the following steps: S141: Using a group relative strategy optimization algorithm, the parameters of the intermediate model are updated based on the quality score to obtain the target model.

[0117] In this embodiment, the Group-wise Relative Policy Optimization (GRPO) algorithm is a specific reinforcement learning algorithm used in this embodiment. It can be understood as an implementation of a policy optimization algorithm.

[0118] In this step, S601 is first executed, which allows the intermediate model to generate a "group" of answers for the same sample input, such as [answer A, answer B, answer C]. Then S602 is executed, which calculates a quality score for each answer in this group, such as [0.7, 0.9, 0.3].

[0119] The core idea of ​​GRPO is to optimize by leveraging the "relative" goodness or badness of a set of answers and their scores, rather than looking at each answer in isolation. The algorithm identifies the highest-scoring answer (answer B, 0.9 points) and the lower-scoring answers (answers A and C) within this "set." The algorithm then calculates an optimization objective that aims to increase the probability of the model generating answer B while decreasing the probability of generating answers A and C.

[0120] GRPO can be implemented by constructing preference pairs, such as (answer B > answer A) and (answer B > answer C). Algorithms (such as DPO or similar relative ranking-based algorithms) then use these preference pairs to update the parameters of intermediate models. In this way, the model is trained to be more inclined to generate answers that are relatively better within the group (i.e., answers that receive higher quality scores).

[0121] According to the scheme of this disclosure embodiment, by employing the group relative policy optimization algorithm in S141, the training process of reinforcement learning becomes more efficient and stable. Compared with traditional RL algorithms (which may only rely on the absolute score of a single answer), GRPO, by utilizing the "relative ranking" information of a group of answers, can more robustly learn the preferences of the reward model, effectively avoiding overfitting to the absolute value of the reward score, thereby converging faster and training a target model with better performance.

[0122] In one possible implementation, embodiments of this disclosure provide a method for training a multimodal generative model. This method is suitable for processing inputs containing multiple information modalities (e.g., images and text) and resolving cross-modal knowledge conflicts. The method can run on a server or dedicated hardware capable of processing multimodal data. The method includes the following steps: S901: Obtain multimodal sample input.

[0123] In this step, multimodal sample input refers to an input combination containing at least two different modalities of data. The sample input can consist of an image and a text question associated with that image.

[0124] Example of sample input: Image modality: An image showing "a cat lying on a keyboard".

[0125] Text modality: User question: "What are the animals in this picture doing?" S902: Construct structured data that includes a cross-modal knowledge conflict resolution process.

[0126] This step aims to generate multimodal structured data for SFT training, simulating the process by which the model handles cross-modal information conflicts.

[0127] Specific implementation process: Utilizing a powerful multimodal second model (capable of understanding images and text), iterative processing is performed on the sample input to generate iterative processing data containing a cross-modal knowledge conflict detection and resolution process. This process may include: Image understanding (internal knowledge output - modality 1): The model analyzes the image content and generates an internal description: "The image shows a cat." Text understanding and preliminary answer (internal knowledge output - modality 2): The model understands the text question and, combined with image understanding, provides a preliminary answer: "The cat is resting." External knowledge retrieval: The model may trigger retrieval based on image content or questions, such as searching for "cat keyboard behavior". The retrieved external knowledge (text) is: "Research shows that cats like warm places, and keyboards attract cats to lie on them to sleep or keep warm when they are dissipating heat." Cross-modal conflict detection: The model compares the image content (cat lying down) with the retrieved text information (cat may be sleeping or warming itself). Suppose the model also retrieves another conflicting piece of information: "Common sense base: Cats are nocturnal animals and usually sleep during the day." The model may detect a potential conflict or ambiguity: the image itself cannot completely determine whether the cat is "resting," "sleeping," or "warming itself."

[0128] Conflict resolution / information integration: The model needs to integrate visual and textual information. It may determine that the visual evidence of "lying on the keyboard" is the most direct, combine it with the retrieved explanations of "warming up / sleeping," choose a more precise or conservative description, and may point out uncertainties.

[0129] Generate final output: The model generates the final multimodal output (in this case, mainly text responses).

[0130] The above process is recorded and structured, resulting in structured data.

[0131] S903: Use a knowledge conflict training set containing multimodal structured data for model training. The large amount of multimodal structured data generated in S902 is aggregated into a knowledge conflict training set. The subsequent supervised fine-tuning and reinforcement learning stages are similar to S120-S140, but the trained models (the first model and intermediate models) are multimodal generative models capable of simultaneously processing image and text inputs and learning to generate structured outputs containing cross-modal conflict handling processes. In the RL stage, the reward mechanism also needs to adapt to the multimodal scenario; for example, evaluating whether the generated text response is consistent with the image content and whether the retrieved text can reasonably explain the image phenomenon.

[0132] According to the embodiments of this disclosure, by applying the training method to multimodal scenarios, the target generative model can not only handle knowledge conflicts within a single modality, but also effectively identify and resolve contradictions from different modalities (such as visual information and textual information). By learning the process of handling cross-modal conflicts, the model can generate more accurate, reliable, and multimodal content that can simultaneously rely on evidence from multiple modalities, greatly expanding the scope of application of the method and improving its performance in complex multimodal applications such as image question answering and video understanding.

[0133] In one possible implementation, embodiments of this disclosure provide a method for resolving knowledge conflict issues in retrieval enhancement generation, which is implemented through two core stages: data construction and two-stage training. Figure 2 The flowchart illustrating the data construction phase provided in this embodiment of the disclosure specifically includes the following steps: Phase 1: Data Construction S210: Obtain sample input, which can be a user query. In this step, natural language queries from users can be collected and organized as sample input. This query is the starting point of the entire process and can be a question, a command, or any input that requires the model to generate content.

[0134] S220: Generate Structured Supervised Fine-Tuning (SFT) Data. This step aims to automatically build a high-quality dataset for the first stage of training. For the query obtained in S210, an auxiliary model or system simulates a complete thought process and problem-solving process. This process may include: first, the model performs initial thinking and answers based on its internal knowledge; then, it proactively initiates external searches to obtain supplementary information as needed; next, it systematically detects and identifies potential conflicts between internal knowledge, external knowledge, and different external knowledge sources; finally, based on a comprehensive analysis of all information and conflict resolution, it generates the final answer. The entire process is recorded and structured using predefined labels to form SFT data.

[0135] The SFT data can contain the following types of structured labels: <think>Tags: Used to record the thought process or reasoning chain of a model.

[0136] <internal>Tags: Used to record the answers generated by the model from its internal parameterized knowledge.

[0137] <search>and <information>Tags: These are used to record the search queries generated by the model and the relevant information obtained from external sources, respectively.

[0138] <conflict>and <context>Tags: These are used to explicitly identify detected knowledge conflicts and their related background information.

[0139] <answer>Tags: Used to identify the final answer generated by the model.

[0140] Figure 3 The flowchart illustrating the two-stage training phase provided in this embodiment of the disclosure specifically includes the following steps: Phase Two: Two-Stage Training S310: Perform Supervised Fine-Tuning (SFT) Training. In this step, a base model is trained using the SFT dataset built in the first stage. The model learns to take user queries as input and generate structured data that conforms to the format of the SFT data and contains a complete thought process and conflict resolution process. Through this stage of training, the model initially masters the standardized process of identifying and representing knowledge conflicts.

[0141] S320: Perform reinforcement learning (RL) training. To further optimize the model's decision-making and generation capabilities in complex scenarios, this step performs reinforcement learning on the model trained by SFT. This process is iterative and may include the following sub-steps: S321: Generate multiple responses (Rollout).

[0142] For a given query, a model trained with SFT will generate multiple distinct response sequences. Each response represents a complete knowledge conflict resolution process with structured labels.

[0143] S322: Use a diversified reward mechanism for evaluation.

[0144] A pre-defined Diverse Reward mechanism evaluates each response generated in S321 from multiple dimensions and assigns a comprehensive quantitative score. This reward mechanism aims to guide the model's optimization direction from multiple aspects, ensuring the overall quality of the generated content.

[0145] S323: Computational Advantages.

[0146] Based on the reward score obtained for each response in S322, the reinforcement learning algorithm calculates the advantage value of each response relative to the average level. Responses with high scores receive a positive advantage, while responses with low scores receive a negative advantage.

[0147] S324: Update Param.

[0148] Finally, based on the advantage value calculated in S323, the algorithm updates the model's parameters. This update process encourages the model to generate outputs more similar to high-scoring responses in the future, while suppressing the generation of outputs similar to low-scoring responses. By continuously repeating the loop from S321 to S324, the model's ability to resolve knowledge conflicts is continuously enhanced.

[0149] According to the embodiments of this disclosure, the above implementation steps can bring about significant technical improvements. First, by introducing systematic knowledge conflict detection and localization into data construction and diversified reward mechanisms, the model can effectively handle the contradictions between internal knowledge and multi-source external information, thereby improving the accuracy and consistency of the generated content.

[0150] Secondly, the multi-step thinking chain and self-reflection mechanism in the two-stage training enable the model to perform multiple rounds of reasoning and error correction in complex information environments, which enhances the model's adaptability and robustness to complex knowledge scenarios.

[0151] Meanwhile, by utilizing structured conflict labels and reinforcement learning with diverse rewards, this approach establishes clear evaluation criteria for the model's conflict handling process and results, optimizes the training process, and improves the controllability of training efficiency and effectiveness.

[0152] also, <think>and <conflict>Explicit labeling makes the model's decision-making logic and conflict resolution process transparent, solving the "black box" problem and ensuring the product's transparency and explainability, thereby enhancing user trust. Finally, by intelligently handling conflicts, this method enables the model to better integrate and utilize newly retrieved information, promoting the effective fusion and dynamic updating of multi-source knowledge, ensuring the timeliness of content, and improving the overall user experience.

[0153] Figure 4 This is a schematic diagram of the structure of a training device 400 for a model provided according to an embodiment of this disclosure. Figure 4 As shown, the device includes: Construction module 401 is used to construct a knowledge conflict training set, which contains structured data corresponding to the sample input and labeled with knowledge conflict processing procedures. The supervised fine-tuning module 402 is used to perform supervised fine-tuning of the first model using the knowledge conflict training set to obtain an intermediate model. The scoring module 403 is used to score the quality of the knowledge conflict handling process generated by the intermediate model in response to the input question based on a preset reward mechanism. The reinforcement learning module 404 is used to train the intermediate model using reinforcement learning based on the quality score to obtain the target model.

[0154] In one possible implementation, building module 401 is used for: Using a second model, based on internal and / or external knowledge, at least one round of processing is performed on the sample input to obtain iteratively processed data; The iteratively processed data is labeled with structure to obtain the structured data.

[0155] In one possible implementation, building module 401 is used for: Obtain the internal knowledge response generated by the second model in response to the sample input; Based on the sample input or the internal knowledge response, external knowledge retrieval is performed to obtain external knowledge; Detect whether there is a knowledge conflict between the internal knowledge response and the external knowledge, or among multiple external knowledge sources; In the event of knowledge conflict, external knowledge retrieval should be performed again to update the external knowledge. The sample input is re-answered based on the updated external knowledge to obtain iterative processing data.

[0156] In one possible implementation, building module 401 is used for: Internal knowledge tags are used to annotate the internal knowledge responses in the iteratively processed data; The external knowledge retrieval process in the iteratively processed data is labeled using retrieval information tags; The knowledge conflict analysis process in the iteratively processed data is labeled using conflict content tags; Answer labels are used to annotate the final answer in the iteratively processed data. In one possible implementation, the device further includes a filtering module for: The structured data is automatically filtered to retain structured data that simultaneously meets the following preset conditions: The iterative data processing includes at least one knowledge conflict identification and resolution process; The number of tool calls included in the iteratively processed data is not less than a preset number; and The final answer annotated in the structured data is consistent with the preset standard answer input for the sample.

[0157] In one possible implementation, the scoring module 403 is used for: The intermediate model is used to generate multiple responses for the sample input; Based on a preset reward mechanism, the knowledge conflict handling process in the multiple answer generation process is comprehensively scored to obtain the quality score; The preset reward mechanism is used to evaluate the quality of the knowledge conflict handling process from multiple dimensions; the multiple dimensions include at least two of the following: search effect dimension, conflict handling dimension, context sufficiency dimension, and format specification dimension.

[0158] In one possible implementation, the scoring module 403 is used for: For the abstract indicators in the conflict handling dimension or the context sufficiency dimension, an external model is invoked for automated scoring.

[0159] In one possible implementation, the scoring module 403 is used for: Build a prompt message that includes preset scoring criteria; The text content generated by the intermediate model corresponding to the abstract indicator and the prompt information are input into the external model; Obtain the scoring result returned by the external model, and use it as the score of the abstract indicator.

[0160] In one possible implementation, the knowledge conflict resolution process annotation includes: Internal knowledge tags used to identify the knowledge within a model; Search query tags used to identify the search query and search information tags used to identify the search content; Conflict tags used to identify knowledge conflict phenomena and conflict content tags with related background information; Answer labels are used to identify the final answer in the output.

[0161] In one possible implementation, reinforcement learning module 404 is used for: The parameters of the intermediate model are updated based on the quality score using a group relative strategy optimization algorithm to obtain the target model.

[0162] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0163] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0164] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0165] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0166] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0167] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0168] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as model training methods. For example, in some embodiments, the model training method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the model training method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform model training methods by any other suitable means (e.g., by means of firmware).

[0169] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0170] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0171] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0172] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0173] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0174] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0175] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0176] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.< / conflict> < / think> < / answer> < / context> < / conflict> < / information> < / search> < / internal> < / think> < / conflict> < / answer> < / search> < / conflict> < / search> < / conflict> < / conflict> < / answer> < / search> < / conflict> < / answer> < / search> < / search> < / conflict>

Claims

1. A training method for a large model, comprising: Construct a knowledge conflict training set, which contains structured data corresponding to the sample input and labeled with knowledge conflict processing procedures; The first model is then fine-tuned under supervised supervision using the knowledge conflict training set to obtain an intermediate model. Based on a preset reward mechanism, the quality score of the knowledge conflict handling process generated by the intermediate model in response to the sample input is performed. Based on the quality score, the intermediate model is trained using reinforcement learning to obtain the target model.

2. The method according to claim 1, wherein, The construction of the knowledge conflict training set includes: Using a second model, based on internal and / or external knowledge, at least one round of processing is performed on the sample input to obtain iteratively processed data; The iteratively processed data is labeled with structure to obtain the structured data.

3. The method according to claim 2, wherein, The step of utilizing a second model, based on internal and / or external knowledge, to perform at least one round of processing on the sample input to obtain iteratively processed data includes: Obtain the internal knowledge response generated by the second model in response to the sample input; Based on the sample input or the internal knowledge response, external knowledge retrieval is performed to obtain external knowledge; Detect whether there is a knowledge conflict between the internal knowledge response and the external knowledge, or among multiple external knowledge sources; In the event of knowledge conflict, external knowledge retrieval should be performed again to update the external knowledge. The sample input is re-answered based on the updated external knowledge to obtain iterative processing data.

4. The method according to claim 2, wherein, The step of performing structured annotation on the iteratively processed data to obtain the structured data includes: Internal knowledge tags are used to annotate the internal knowledge responses in the iteratively processed data; The external knowledge retrieval process in the iteratively processed data is labeled using retrieval information tags; The knowledge conflict analysis process in the iteratively processed data is labeled using conflict content tags; Answer labels are used to annotate the final answer in the iteratively processed data.

5. The method according to claim 2, further comprising: The structured data is automatically filtered to retain structured data that simultaneously meets the following preset conditions: The iterative data processing includes at least one knowledge conflict identification and resolution process; The number of tool calls included in the iteratively processed data is not less than a preset number; as well as The final answer annotated in the structured data is consistent with the preset standard answer input for the sample.

6. The method according to claim 1, wherein, The quality scoring of the knowledge conflict handling process generated by the intermediate model in response to the sample input, based on a preset reward mechanism, includes: The intermediate model is used to generate multiple responses for the sample input; Based on a preset reward mechanism, the knowledge conflict handling process in the multiple answer generation process is comprehensively scored to obtain the quality score; The preset reward mechanism is used to evaluate the quality of the knowledge conflict handling process from multiple dimensions; the multiple dimensions include at least two of the following: search effect dimension, conflict handling dimension, context sufficiency dimension, and format specification dimension.

7. The method according to claim 6, wherein, The quality score is obtained by comprehensively evaluating the knowledge conflict handling process during the generation of multiple answers according to a preset reward mechanism, including: For the abstract indicators in the conflict handling dimension or the context sufficiency dimension, an external model is invoked for automated scoring.

8. The method according to claim 7, wherein, The automated scoring by calling external models includes: Build a prompt message that includes preset scoring criteria; The text content generated by the intermediate model corresponding to the abstract indicator and the prompt information are input into the external model; Obtain the scoring result returned by the external model, and use it as the score of the abstract indicator.

9. The method according to claim 1, wherein, The knowledge conflict resolution process annotation includes: Internal knowledge tags used to identify the knowledge within a model; Search query tags used to identify the search query and search information tags used to identify the search content; Conflict tags used to identify knowledge conflict phenomena and conflict content tags with related background information; Answer labels are used to identify the final answer in the output.

10. The method according to claim 1, wherein, The step of training the intermediate model using reinforcement learning based on the quality score to obtain the target model includes: The parameters of the intermediate model are updated based on the quality score using a group relative strategy optimization algorithm to obtain the target model.

11. A training device for a large model, comprising: A construction module is used to construct a knowledge conflict training set, which contains structured data corresponding to the sample input and labeled with knowledge conflict processing procedures. The supervised fine-tuning module is used to perform supervised fine-tuning of the first model using the knowledge conflict training set to obtain an intermediate model. The scoring module is used to score the quality of the knowledge conflict handling process generated by the intermediate model in response to the input question based on a preset reward mechanism. The reinforcement learning module is used to train the intermediate model using reinforcement learning based on the quality score to obtain the target model.

12. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images