Model training, method for providing consultation service information, and electronic device

Abstract

Embodiments of the present application disclose a model training method, a method for providing consulting service information and an electronic device, comprising: obtaining a first training data set; training a first AI generation model in a first stage by using the first training data set, so that the first AI generation model generates final answer content on the basis of reasoning according to a plurality of reasoning steps in the inference chain; collecting answer content generated by the first AI generation model completing the first stage of training for second input data, screening out inferior answer content therefrom, and generating answer content again for the target second input data; and performing second stage training on the first AI generation model by using the target second input data, the inferior answer content and the regenerated answer content as a preference data pair. Through the embodiments of the present application, the probability of occurrence of "illusion" or logical inconsistency of the model generation result can be reduced.

Description

Technical Field

[0001] This application relates to the field of model training technology, and in particular to methods and electronic devices for model training and providing consulting service information. Background Technology

[0002] AI (Artificial Intelligence) models, due to their large parameter scale, are able to store and process massive amounts of information, thus achieving higher performance than traditional models in various tasks (such as text generation, image understanding, etc.). However, while these AI models often perform well in general domains, they face significant challenges in adapting to specific professional fields (such as AI customer service scenarios in product information service systems).

[0003] For example, the traditional "input-output" extensive learning method may cause AI models to produce "illusions" (the model fabricates phenomena that it believes to exist, or even seem reasonable or credible, but are inconsistent with the facts of the real world) or logical inconsistencies during the task processing process.

[0004] Furthermore, the industry has proposed a preference alignment method based on reinforcement learning, aiming to make the output of AI models more consistent with human preferences. However, the effectiveness of this preference alignment method typically relies on high-quality preference pair data. A "preference pair" is the carrier of alignment signals, existing in the form of (hints, winning responses, losing responses), explicitly informing the model which response is better. The model learns from these comparisons to fit the fuzzy, multi-dimensional values ​​of humans. In practical applications, these preference pairs are usually constructed manually or using stronger AI models (acting as referees). However, both methods suffer from inconsistent standards of "good" or "bad" among different annotators or AI referees. Additionally, if the difference in quality between two responses is not significant, the model struggles to learn human preference information. Ideally, a "winning response" should be clearly superior to a "losing response." In short, existing technologies generally suffer from low-quality preference pairs, leading to problems such as low training efficiency and unstable convergence. Summary of the Invention

[0005] This application provides methods and electronic devices for model training and providing consulting service information, which can reduce the probability of "illusions" or logical inconsistencies in the model-generated results and improve training efficiency.

[0006] This application provides the following solution:

[0007] A model training method for training a first artificial intelligence (AI) generative model, comprising: Obtain a first training data set, which includes multiple first data pairs. Each first data pair includes first input data and corresponding high-quality answer content. Each first data pair also corresponds to step-by-step answer content at multiple inference steps in a preset inference chain. The first AI generation model is trained in the first stage using the first training data set, so that the first AI generation model generates the final answer content based on reasoning according to multiple reasoning steps in the inference chain. The first AI generation model, which has completed the first stage of training, collects the answers generated by the second input data, filters out inferior answers, checks the defects in the inferior answers, and generates prompt information based on the target second input data corresponding to the inferior answers and the detected defects. The first AI generation model that has completed the first stage of training is invoked according to the prompt information to regenerate the answer content for the target second input data; The target second input data, the poor-quality answer content, and the regenerated answer content are combined into preference data pairs, so that the first AI generation model can be trained in the second stage using a second training data set composed of multiple preference data pairs.

[0008] The acquisition of the first training data set includes: Obtain the plurality of first data pairs; The second AI generation model is used to reverse-generate step-by-step answers for multiple inference steps in a pre-set inference chain for the first data pair, and the first training data set is generated based on the first data pair and the corresponding step-by-step answers for multiple inference steps.

[0009] The first input data is obtained from historical service records generated within the consultation service system of the commodity information service system, including the consultation questions entered by the user and related context data. The pre-set inference chain includes multiple inference steps, such as: Intent recognition, knowledge assessment, strategy formulation, and expression adjustment; The reasoning steps corresponding to intent recognition are used to analyze the user's input inquiry and identify the user's core needs and / or potential intents. The reasoning steps corresponding to the knowledge assessment are used to associate the knowledge base, retrieve and filter key information related to the current request and / or intent; The reasoning steps corresponding to the strategy formulation are used to plan the core response strategy based on the analysis results of intent recognition and knowledge assessment. The reasoning steps corresponding to the expression adjustment are used to determine the language style and emotional tone of the response.

[0010] The high-quality answers in the first data pair are obtained through expert annotation or annotation by a third AI generation model.

[0011] The inspection of the defects in the substandard answers includes: The step-by-step answer content generated at multiple inference steps in the inference chain by the first AI generation model that has completed the first stage of training during the process of generating the poor answer content for the target second input data; Defect detection is performed by analyzing the poor-quality answers and the step-by-step answers.

[0012] The prompt information includes a description of how to improve the detected defects, so that the first AI generation model, which has completed the first stage of training, can regenerate the response content for the target second input data by referring to the description of the improvement methods.

[0013] Specifically, when generating prompt information based on the target second input data corresponding to the substandard answer content and the detected defects, the substandard answer content is added to the prompt information so that the answer content regenerated by the first AI generation model that has completed the first stage of training for the target second input data is different from the substandard answer content.

[0014] The second stage of training includes training the first AI generation model using a preference alignment method based on reinforcement learning.

[0015] A model training method for training a first AI-generated model, comprising: Acquire multiple data pairs, each including input data and corresponding high-quality answer content; The second AI generation model is used to reverse-generate step-by-step answer content for multiple inference steps in the preset inference chain for the data pair, and a training data set is generated based on the data pair and the corresponding step-by-step answer content for multiple inference steps. The first AI generation model is trained using the training dataset. During the training process, the first AI generation model is guided to generate step-by-step answer content according to multiple reasoning steps in the inference chain, and then generate the final answer content. The training process is supervised by the step-by-step answer content generated in reverse by the second AI generation model and the high-quality answer content, so as to complete the training of the first AI generation model.

[0016] A method for providing consulting service information includes: Receive user input data related to consultation services from the product information service system; The first AI generation model is used to generate corresponding answer content for the input data, or to generate reference answer content for human service providers; wherein, the first AI generation model is specifically used to generate the final answer content based on reasoning according to multiple reasoning steps included in the reasoning chain corresponding to the consultation service; the first AI generation model is obtained by training according to the method described in any of the foregoing items; The output of the answer generated by the first AI generation model is displayed.

[0017] The reasoning steps in the reasoning chain corresponding to the consulting service include: Intent recognition, knowledge assessment, strategy formulation, and expression adjustment; The reasoning steps corresponding to intent recognition are used to analyze the user's input inquiry and identify the user's core needs and / or potential intents. The reasoning steps corresponding to the knowledge assessment are used to associate the knowledge base, retrieve and filter key information related to the current request and / or intent; The reasoning steps corresponding to the strategy formulation are used to plan the core response strategy based on the analysis results of intent recognition and knowledge assessment. The reasoning steps corresponding to the expression adjustment are used to determine the language style and emotional tone of the response.

[0018] A model training method, comprising: The AI ​​generation model under training collects the answers generated by the input data, filters out the poor-quality answers, checks the defects in the poor-quality answers, and generates prompt information based on the target input data corresponding to the poor-quality answers and the detected defects. The AI ​​generation model is invoked according to the prompt information to regenerate the answer content for the target input data; The target input data, the poor-quality answer content, and the regenerated answer content are combined into preference data pairs, so that the AI ​​generation model can be trained based on reinforcement learning using a training data set composed of multiple preference data pairs.

[0019] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the preceding methods.

[0020] An electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of any of the preceding methods.

[0021] A computer program product includes a computer program / computer executable instructions that, when executed by a processor in an electronic device, implement the steps of any of the preceding methods.

[0022] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application proposes a two-stage training framework based on reasoning chains and defect detection-driven preference optimization. Specifically, in the first training stage, a logical reasoning pattern (e.g., the logical reasoning pattern of a human expert) can be implanted into the model. The first AI generation model is then guided to reason step-by-step according to multiple steps in the reasoning chain, generating the final answer based on the complete reasoning chain, replacing the traditional coarse-grained "input-output" learning approach. In the second training stage, to address the inefficiency caused by the lack of clear distinction between good and bad answers in preference learning, the answers generated by the first AI generation model after the first stage of training are collected. Inferior answers are then filtered out, and the inferior answers output by the model are diagnosed. Based on the diagnosed defects, the AI ​​model after the first stage of training is guided to regenerate answers. Subsequently, the aforementioned inferior answers and the regenerated answers are combined to form preference data pairs, and the first AI generation model is trained in the second stage based on these preference data pairs. In this way, the first AI generation model can generate the final answer by following multiple steps in the reasoning chain, reducing the probability of "illusions" or logical inconsistencies in the model's generated results. Furthermore, by constructing preference data pairs with more pronounced differences in quality, training efficiency can be improved, further enhancing the quality of the final answers generated by the first AI generation model.

[0023] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1This is a schematic diagram of the system architecture provided in the embodiments of this application; Figure 2 This is a flowchart of the first method provided in the embodiments of this application; Figure 3 This is a flowchart of the second method provided in the embodiments of this application; Figure 4 This is a flowchart of the third method provided in the embodiments of this application; Figure 5 This is a flowchart of the fourth method provided in the embodiments of this application; Figure 6 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0027] In this embodiment, to obtain higher-quality preference data pairs and more efficiently train the AI ​​model, a two-stage training framework driven by inference chains and defect checklists is proposed for preference optimization. Specifically, in the first training stage, logical reasoning patterns (e.g., those of human experts) can be implanted into the model. The AI ​​model then performs step-by-step reasoning according to the inference chain, generating the final answer based on the complete chain, replacing the traditional coarse-grained "input-output" learning approach. In the second training stage, to address the inefficiency caused by unclear comparisons between good and bad preferences, the answers generated by the AI ​​model after the first stage of training are collected. Inferior answers are then filtered out, and a checklist is used as the core tool to diagnose these inferior responses. Based on the diagnosed defects, the AI ​​model after the first stage of training is guided to regenerate answers. Finally, the inferior answers and the regenerated answers are combined to form preference data pairs, and the AI ​​model is trained in the second stage based on these preference data pairs.

[0028] Specifically, such as Figure 1As shown, in the first stage, to embed logical reasoning patterns into the model, one specific approach is to first acquire multiple first data pairs, including first input data and corresponding high-quality answer data. This first input data can be obtained from historical data generated within the application domain of the AI-generated model. For example, in a consultation service system within a commodity information service system (also known as an "online service platform"), the specific input data can be obtained from historical service records generated by consultation service providers (including human service providers or AI service providers) providing consultation services to consumers or merchants. For instance, the specific first input data could be a consumer's consultation question regarding an order and related contextual data (including order-related product information, buyer profiles, merchant profiles, etc.), and so on. The corresponding high-quality answer data can be obtained by pre-annotating by human experts, or it can be generated by other, more powerful AI models (for ease of distinction, the model to be trained in this embodiment is referred to as the first AI generation model, and the more powerful AI model mentioned here can be referred to as the third AI generation model) according to certain rules or business indicators to generate high-quality answer content for the first input data.

[0029] After obtaining multiple first data pairs, a more powerful AI model (e.g., a second AI generation model, also known as a "teacher model") can be used to reverse-engineer step-by-step answers at multiple inference steps in a pre-defined inference chain from the first data pairs. The specific inference chain can be a series of predefined, continuous steps. Different domains can have their own different inference chains. In this embodiment, a pre-defined inference chain, including the inference steps included, can be provided for a specific domain. When the second AI generation model reverse-engineers the step-by-step answers, information about the inference steps included in the inference chain of the specific domain can be provided to the second AI generation model through prompts, enabling the second AI generation model to generate step-by-step answers at multiple inference steps in the specific inference chain for each first data pair.

[0030] Subsequently, the specific first data pair and the corresponding step-by-step answers at multiple reasoning steps can be combined to form the first training data. This first training data is used to complete the first stage of training for the first AI generation model. During the first stage of training, information about the multiple reasoning steps included in the reasoning chain can also be provided in the prompts of the first AI generation model. This allows the first AI generation model to generate the final answer content for the first input data based on reasoning according to the multiple reasoning steps in the reasoning chain. The high-quality answer content corresponding to the first input, saved in the first training data, and the step-by-step answer content at multiple reasoning steps in the reasoning chain generated by the second AI generation model are used to supervise the training errors of the first AI generation model in the first stage, thereby embedding the logical reasoning pattern corresponding to the professional domain into the first AI generation model.

[0031] After completing the first phase of training, the first AI generation model can be deployed online to provide AI services in the corresponding domain. For example, if the first phase of training was completed using training data related to customer service in an e-commerce platform, the first AI generation model that has completed the first phase of training can be deployed to the e-commerce platform's customer service system to provide services such as specific user inquiries. During the service process, responses generated by the first AI generation model for specific second data will be produced. In this embodiment, these responses can be collected, and some low-quality responses can be filtered out for use in the second phase of training of the first AI generation model.

[0032] In the second stage, the collected substandard responses are first checked for defects. Then, based on the target second input data corresponding to the substandard responses and the detected defects, a prompt message is generated. This prompt message may include improvement suggestions based on the detected defects. For example, if a substandard response is diagnosed as lacking initiative and having a "mechanical tone," the prompt message could include: "Please ask the user about their specific preferences in a more proactive manner and reply using more friendly and human-like language." This prompt message can be re-inputted to the first AI generation model that has completed the first stage of training, guiding it to regenerate responses for the target second input data. Since the regenerated response is generated under the aforementioned prompt message, which contains improvement suggestions for the previous substandard responses, the regenerated response usually improves upon the original defects, naturally becoming a high-quality response in stark contrast to the original substandard response. Furthermore, the target second input data, the substandard responses, and the regenerated response can be combined to form a preference data pair. Other poor-quality responses can be processed in the same way to obtain a second training dataset consisting of multiple preference data pairs. This second training dataset is then used to train the first AI generation model in the second stage. Specifically, the second stage of training can employ a preference alignment method based on reinforcement learning. Because the preference data pairs used have a clearer comparison between good and bad, it is easier for the first AI generation model to learn preference information from them.

[0033] Through the two training phases described above, the first AI generative model is able to perform step-by-step reasoning according to the inference chain, and then generate a final answer that conforms to human preferences based on the complete inference chain. Since both the first and second training datasets can be from the same domain, the trained first AI generative model can be applied to specific AI service scenarios within that domain. For example, assuming the model is trained using data from the customer service domain of an e-commerce platform, the trained first AI generative model can be applied to customer service scenarios on the e-commerce platform, and so on.

[0034] The specific implementation schemes provided in the embodiments of this application will be described in detail below.

[0035] Example 1 First, Embodiment 1 of this application provides a model training method for training a first AI generation model. This first AI generation model can be a text generation model, or an image understanding model, etc. See also... Figure 2 The method may specifically include: S201: Obtain a first training data set, which includes multiple first data pairs. Each first data pair includes first input data and corresponding high-quality answer content. Each first data pair also corresponds to step-by-step answer content at multiple inference steps in a preset inference chain.

[0036] First, it's important to clarify that before training the first AI-generated model, the specific application domain must be identified. Then, the first training data can be obtained based on relevant data within that application domain. As mentioned earlier, the first input data can be historical input data generated within an application domain where the AI-generated model can be applied. For example, in the customer service domain of an e-commerce platform, the specific input data could be inquiries and complaints from consumers regarding a particular order / product, along with related contextual data (including dialogue history, order-related product information, buyer profiles, merchant profiles, etc.). In other words, the historical dialogue records of the customer service system can provide the content of inquiries / complaints entered by consumers (or merchants, etc.) during historical consultations / complaints, as well as the related contextual data.

[0037] After obtaining multiple sets of the aforementioned first input data, high-quality responses to this first input data can also be obtained. These high-quality responses can be further described as "golden responses," meaning optimal or professional-level responses. Specifically, these high-quality responses can be obtained by having domain experts annotate the first input data, or, as mentioned earlier, by a third AI generation model to generate the high-quality responses for the specific first input data, and so on. Here, the third AI generation model can be a model with stronger generation capabilities compared to the first AI generation model before training.

[0038] In addition to acquiring high-quality answers corresponding to the first input data, to enable the first AI generation model to learn the reasoning logic, step-by-step answers at multiple inference steps within a pre-defined inference chain can also be acquired for specific first data pairs. That is, not only is it necessary to pre-label the high-quality answers, but also to label how the answers were delivered at each different inference step, in order to ultimately generate the aforementioned high-quality answers. This step-by-step answer content can also be used to supervise the training process of the first AI generation model.

[0039] In obtaining step-by-step answers for multiple inference steps within a pre-defined inference chain for a specific first data pair, there are several possible methods. For example, one method involves manual annotation. Alternatively, in a more preferred approach, after obtaining multiple first data pairs, a second AI generation model can be used to reverse-engineer step-by-step answers for multiple inference steps within the pre-defined inference chain. In other words, the second AI generation model reverse-engineers the inference process from the first input data to the high-quality answer, specifically identifying the step-by-step answers made at each inference step within the inference chain. The specific inference steps included in the inference chain can be pre-defined. These steps are input to the second AI generation model via prompts or other means, enabling the model to infer the corresponding step-by-step answers for each step within the inference chain.

[0040] The reasoning chain consists of a series of predefined, sequential reasoning steps. The specific reasoning chain can differ for different application domains, and the steps included in the chain can be determined in advance based on expert experience in that domain. For example, in the customer service domain of an e-commerce platform, a specific reasoning chain might include the following steps: Step 1: Intent Identification: Analyze user questions to identify their core needs and / or potential intents; Step 2: Knowledge Assessment: Connect to the product knowledge base, retrieve and filter key information (such as attributes, selling points, taboos) that are relevant to the current intent; Step 3: Strategy Formulation: Based on the above analysis, plan the core response strategy (e.g., whether to provide a direct answer, offer a proactive recommendation, or clarify and guide the response). Step 4: Adjusting the Expression: Finalize the language style and emotional tone of the reply (e.g., professional expression, friendly communication).

[0041] The names of each of the above steps, as well as the meaning and examples of each step, can be provided to the second AI generation model in the prompt information for its reference.

[0042] After providing the second AI generation model with information on multiple reasoning steps included in the aforementioned reasoning chain through prompts or other means, the second AI generation model can generate step-by-step responses for each of the aforementioned steps for a specific <first input data, high-quality response content> pair. For example, suppose there is a <first input data, high-quality response content> pair, where the first input data includes a return request initiated by a buyer for a certain order, along with related conversation history, order information, product information, buyer information, merchant information, etc., and the high-quality response content is, "Sorry for the poor experience. To express our apologies, we can refund you a certain amount; you can handle the product yourself," etc. The step-by-step responses generated by the second AI generation model for the aforementioned first data pair could include: "Intent recognition" step: "Buyer wants to return the goods"; The "Knowledge Assessment" step states: "Available knowledge includes return policies, refund strategies, etc." "Strategy Formulation" Steps: "Retain the buyer by asking if they would accept a partial refund"; The "expression adjustment" steps are: "Express your apology first, then ask questions."

[0043] After obtaining the step-by-step responses for the corresponding steps of the first data pair, a first training data set can be generated from the first data pair and the corresponding step-by-step responses. Other first data pairs can also be processed in the same way to obtain the first training data set.

[0044] S202: The first AI generation model is trained in the first stage using the first training data set, so that the first AI generation model generates the final answer content based on reasoning according to multiple reasoning steps in the inference chain.

[0045] After obtaining the first training dataset, the first AI generation model can be trained in its first phase. Specifically, the prompts input to the first AI generation model can include the first input data from the aforementioned first training dataset, as well as information about "performing reasoning according to multiple reasoning steps in a specific reasoning chain, providing step-by-step answers for each step, and then generating the final answer." Thus, when generating answers, the first AI generation model can include not only the final answer but also step-by-step answers for multiple reasoning steps. Then, the training process can be supervised using the step-by-step answers generated by the second AI generation model for the first input data in the aforementioned steps.

[0046] After the first stage of training, logical reasoning patterns can be embedded into the first AI generation model. This allows the first AI generation model to reason according to multiple reasoning steps in a specific reasoning chain when generating responses to specific input data, providing step-by-step responses for each step before generating the final response. For example, after completing the first stage of training for a given input data, the content generated by the first AI generation model may include: "Steps": [ { "Step": "Intent Recognition" “Description”: “The buyer is interested in purchasing the product but is concerned about the cost and seeks the possibility of trying a sample before submission.” }, { "Steps": "Knowledge Assessment" "Description": "The available knowledge includes information on sample policies, pricing strategies, and cost-effective alternatives. This knowledge can fully address the buyer's concerns about price sensitivity and sample testing." }, { "Steps": "Strategy Formulation" "Description": "Respond to the buyer by highlighting any available sample or trial options, emphasizing any cost-saving promotions or discounts, and assuring them of the product's value and quality. Provide clear information on how to access these offers." } ] }, Response: "Hello! We offer sample trials and a 10% discount for first-time homebuyers; please let us know if you are interested."

[0047] In other words, the first AI generation model no longer directly generates the final answer content, but first generates the step-by-step answer content for multiple reasoning steps in the reasoning chain, and then generates the final answer content based on the step-by-step answer content for multiple reasoning steps in the complete reasoning chain.

[0048] Of course, when showing the answer to the user, only the final answer can be shown, and the step-by-step answers for the intermediate reasoning steps need not be shown to the user.

[0049] S203: Collect the response content generated by the first AI generation model that has completed the first stage of training for the second input data, filter out the inferior response content, check the defects in the inferior response content, and generate prompt information based on the target second input data corresponding to the inferior response content and the detected defects.

[0050] After completing the first phase of training, the first AI-generated model can be deployed online to provide services in real-world application scenarios. For example, it can be deployed in the customer service scenarios of e-commerce platforms to answer inquiries and complaints from buyers or sellers, and so on.

[0051] In this embodiment, during the online service provided by the first AI generation model after completing the first stage of training, the responses generated by the first AI generation model in response to the second input data (specifically, the final responses) can be collected, and substandard responses can be filtered out. The second input data can specifically be questions raised by users during the online service process, such as inquiries / complaints, along with associated contextual information. After collecting multiple pieces of second input data and their corresponding responses, substandard responses, i.e., responses of relatively poor quality, can be filtered out through rule-based judgment or judgment by other more powerful AI generation models. For example, if the user does not accept a response provided by the first AI generation model, or if the user questions whether the response was given by a real person, such responses are typically considered substandard.

[0052] After filtering out substandard responses, further examination can be conducted to identify their specific flaws. This involves determining the exact defects in the substandard responses. Several methods can be used for this assessment. For example, one approach is to systematically diagnose these substandard responses using a predefined multi-dimensional checklist. In customer service, for instance, a multi-dimensional checklist might include the following dimensions: Factual accuracy: Whether the information is consistent with domain knowledge; Effective proactive advancement: Whether to proactively advance dialogue or uncover needs; Personification: Is the language natural, fluent, and in line with human habits? Emotional investment: Does it demonstrate empathy and a positive attitude?

[0053] Specifically, multi-dimensional diagnosis can be achieved through rule-based judgment, other AI models, or a combination of both. For example, diagnosing aspects such as factual accuracy and effective proactive advancement can be done through rule-based judgment, while diagnosing dimensions such as anthropomorphism and emotional engagement can be done through AI models.

[0054] Furthermore, in the specific diagnosis, the judgment can be made directly based on the specific second input data and the corresponding poor-quality response content itself; that is, the judgment can be made based on the final response result. Alternatively, since the first AI generation model has already completed the aforementioned first stage of training, enabling it to generate step-by-step response content for multiple reasoning steps before obtaining the final response content, a comprehensive diagnosis can be made by combining the step-by-step responses for multiple reasoning steps corresponding to the specific poor-quality response content, in order to more accurately determine the specific defects in the poor-quality response content. In other words, through the aforementioned first stage of training, not only can the first AI generation model learn the logical reasoning patterns of human experts, but also, in this second training stage, it can more accurately locate at which step the specific defect may have occurred.

[0055] After diagnosing the specific flaws in a poorly written response, a prompt can be generated based on the corresponding second input data and the aforementioned flaw information. This prompt can include descriptive information such as how to improve the specific flaw. For example, if a poorly written response is diagnosed as lacking initiative and having a "mechanical tone," the prompt regarding these flaws might read: "Please ask the user for their specific preferences in a more proactive manner and reply using more friendly and human-like language," and so on.

[0056] S204: Based on the prompt information, the first AI generation model that has completed the first stage of training is invoked to regenerate the answer content for the target second input data.

[0057] After generating the prompt information, the first AI generation model, which has completed the first stage of training, can be invoked based on this prompt information to regenerate the response content for the corresponding target second input data. Since the prompt information includes information related to improving specific defects, the regenerated response content will be significantly better than the aforementioned inferior response content, at least in terms of those defects. In a specific implementation, the aforementioned inferior response information can also be included in the prompt information, allowing the first AI generation model to better and more clearly distinguish the regenerated response content from the inferior response content, thus making the comparison between the inferior and regenerated response content more distinct.

[0058] S205: The target second input data, the inferior answer content, and the regenerated answer content are combined into preference data pairs, so as to use the second training data set composed of multiple preference data pairs to train the first AI generation model in the second stage.

[0059] After regenerating the response, the aforementioned target second input data, the poor-quality response, and the regenerated response can be combined into preference data pairs. The poor-quality response can be directly labeled as "poor," and the regenerated response is naturally labeled as "better." Then, a second training data set composed of multiple aforementioned preference data pairs can be used to train the first AI generation model in the second stage. Specifically, the second stage of training may include training a preference alignment method based on reinforcement learning. Reinforcement learning-based preference alignment is essentially a "trial and error-reward" learning process, where "conforming to human preferences" can be defined as the reward to be maximized. More detailed training procedures are not the focus of this application and will not be elaborated here.

[0060] The first AI model trained in the above manner can be applied in the corresponding domain. For example, as mentioned earlier, if training is performed using data from the customer service domain of an e-commerce platform, the trained first AI generation model can be applied in the customer service domain. This includes answering user questions in place of human customer service representatives when they are offline, or providing reference answers to user questions when human customer service representatives are online, and so on. Of course, if it needs to be applied to other domains, relevant data from other domains can be used to train the first AI generation model according to the scheme provided in the embodiments of this application.

[0061] In summary, this application proposes a two-stage training framework based on reasoning chains and defect detection-driven preference optimization. Specifically, in the first training stage, a logical reasoning pattern (e.g., the logical reasoning pattern of a human expert) can be implanted into the model. The first AI generation model is then guided to reason step-by-step according to multiple steps in the reasoning chain, generating the final answer based on the complete reasoning chain, replacing the traditional coarse-grained "input-output" learning approach. In the second training stage, to address the inefficiency caused by the lack of clear distinction between good and bad answers in preference learning, the answers generated by the first AI generation model after the first stage of training are collected. Inferior answers are then filtered out, and the model's output of these inferior answers is diagnosed. Based on the diagnosed defects, the AI ​​model after the first stage of training is guided to regenerate answers. Subsequently, the aforementioned inferior answers and the regenerated answers are combined to form preference data pairs, and the first AI generation model is trained in the second stage based on these preference data pairs. In this way, the first AI generation model can generate the final answer by following multiple steps in the reasoning chain, reducing the probability of "illusions" or logical inconsistencies in the model's generated results. Furthermore, by constructing preference data pairs with more pronounced differences in quality, training efficiency can be improved, further enhancing the quality of the final answers generated by the first AI generation model.

[0062] Example 2 In the aforementioned embodiment one, the specific model training process can be divided into two stages. In practical applications, the two training stages can also exist independently. Therefore, this embodiment two provides a model training method for the aforementioned first stage training process. This method is used to train a first AI-generated model. See [link to documentation]. Figure 3 The method may include: S301: Acquire multiple data pairs, wherein the data pairs include input data and corresponding high-quality answer content; S302: Use the second AI generation model to generate step-by-step answer content for multiple inference steps in the preset inference chain for the data pair, and generate a training data set based on the data pair and the corresponding step-by-step answer content for multiple inference steps. S303: The first AI generation model is trained using the training data set. During the training process, the first AI generation model is guided to generate the final answer content based on the step-by-step answer content generated according to multiple inference steps in the inference chain. The training process is supervised by the step-by-step answer content generated in reverse by the second AI generation model and the high-quality answer content, so as to complete the training of the first AI generation model.

[0063] The solution provided in this second embodiment can implant logical reasoning patterns (e.g., logical reasoning patterns of human experts) into the model. By having the first AI generation model perform step-by-step reasoning according to multiple steps in the reasoning chain, and then generating the final answer based on the complete reasoning chain, it replaces the traditional coarse-grained learning of "input-output", thereby reducing the probability of "illusions" or logical inconsistencies in the model's generated results.

[0064] Example 3 This third embodiment provides a method for providing consultation service information, specifically addressing the application of the first AI-generated model in the consultation service system of a commodity information service system. (See [link to previous document]). Figure 4 The method may specifically include: S401: Receive input data related to consultation services from users in the product information service system; S402: Utilize the first AI generation model to generate corresponding answer content for the input data, or generate reference answer content for the human service provider; wherein, the first AI generation model is specifically used to generate the final answer content based on reasoning according to multiple reasoning steps included in the reasoning chain corresponding to the consultation service; the first AI generation model is obtained by training according to the method described in the aforementioned Embodiment 1 or Embodiment 2; S403: Output and display the answer content generated by the first AI generation model.

[0065] Example 4 This fourth embodiment provides a model training method for the second stage of the training process mentioned in embodiment one. See [link to embodiment]. Figure 5 The method may include: S501: Collect the responses generated by the AI ​​generation model to be trained based on the input data, filter out the inferior responses, check the defects in the inferior responses, and generate prompt information based on the target input data corresponding to the inferior responses and the detected defects.

[0066] In this third embodiment, it is assumed that the AI ​​generation model already possesses the ability to respond to specific domains before training, but the quality of the responses needs improvement and requires further training. Under this premise, the AI ​​generation model can first generate response content based on the input data, identify substandard responses, perform defect checks, and then construct prompt information based on the detected defects.

[0067] S502: The AI ​​generation model is invoked according to the prompt information to regenerate the answer content for the target input data.

[0068] S503: The target input data, the poor-quality answer content, and the regenerated answer content are combined into preference data pairs, so as to use the training data set composed of multiple preference data pairs to perform preference alignment training on the AI ​​generation model based on reinforcement learning.

[0069] The solution provided in this embodiment four can construct preference data pairs with more obvious differences in quality, thus improving the training efficiency of the model and further enhancing the quality of the final answer content generated by the AI ​​generation model.

[0070] For the parts not described in detail in the above embodiments two to four, please refer to the description in embodiment one and other parts of this specification, which will not be repeated here.

[0071] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0072] Corresponding to Embodiment 1, this application also provides a model training apparatus for training a first artificial intelligence (AI) generated model. The apparatus may include: The first training set acquisition unit is used to acquire the first training data set, which includes multiple first data pairs. The first data pair includes first input data and corresponding high-quality answer content. The first data pair also corresponds to the step-by-step answer content of multiple inference steps in the preset inference chain. The first training unit is used to train the first AI generation model in the first stage using the first training data set, so that the first AI generation model generates the final answer content based on reasoning according to multiple reasoning steps in the inference chain. The defect detection unit is used to collect the response content generated by the first AI generation model after completing the first stage of training in response to the second input data, filter out the inferior response content, check the defects in the inferior response content, and generate prompt information based on the target second input data corresponding to the inferior response content and the detected defects. The regeneration unit is used to invoke the first AI generation model that has completed the first stage of training according to the prompt information, so as to regenerate the answer content for the target second input data; The second training unit is used to combine the target second input data, the poor-quality answer content, and the regenerated answer content into preference data pairs, so as to use a second training data set composed of multiple preference data pairs to train the first AI generation model in the second stage.

[0073] Specifically, the first training set acquisition unit can be used for: Obtain the plurality of first data pairs; The second AI generation model is used to reverse-generate step-by-step answers for multiple inference steps in a pre-set inference chain for the first data pair, and the first training data set is generated based on the first data pair and the corresponding step-by-step answers for multiple inference steps.

[0074] Specifically, the first input data may be obtained from historical service records generated within the consultation service system of the commodity information service system, including the consultation questions entered by the user and related context data. The pre-set inference chain includes multiple inference steps, such as: Intent recognition, knowledge assessment, strategy formulation, and expression adjustment; The reasoning steps corresponding to intent recognition are used to analyze the user's input inquiry and identify the user's core needs and / or potential intents. The reasoning steps corresponding to the knowledge assessment are used to associate the knowledge base, retrieve and filter key information related to the current request and / or intent; The reasoning steps corresponding to the strategy formulation are used to plan the core response strategy based on the analysis results of intent recognition and knowledge assessment. The reasoning steps corresponding to the expression adjustment are used to determine the language style and emotional tone of the response.

[0075] Specifically, the high-quality answers in the first data pair can be obtained through expert annotation or annotation by a third-party AI generation model.

[0076] Specifically, the defect inspection unit can be used for: The step-by-step answer content generated at multiple inference steps in the inference chain by the first AI generation model that has completed the first stage of training during the process of generating the poor answer content for the target second input data; Defect detection is performed by analyzing the poor-quality answers and the step-by-step answers.

[0077] The prompt information includes a description of how to improve the detected defects, so that the first AI generation model, which has completed the first stage of training, can regenerate the response content for the target second input data by referring to the description of the improvement methods.

[0078] Specifically, when generating prompt information based on the target second input data corresponding to the poor-quality answer content and the detected defects, the poor-quality answer content can also be added to the prompt information, so that the answer content regenerated by the first AI generation model that has completed the first stage of training for the target second input data is different from the poor-quality answer content.

[0079] The second stage of training includes training the first AI generation model using a preference alignment method based on reinforcement learning.

[0080] Corresponding to Embodiment 2, this application also provides a model training apparatus for training a first AI-generated model, comprising: The data pair acquisition unit is used to acquire multiple data pairs, wherein the data pairs include input data and corresponding high-quality answer content; The step-by-step answer content acquisition unit is used to generate step-by-step answer content for multiple inference steps in a preset inference chain by using a second AI generation model for the data pair, and to generate a training data set based on the data pair and the corresponding step-by-step answer content for multiple inference steps. The model training unit is used to train the first AI generation model using the training data set. During the training process, the first AI generation model is guided to generate the final answer content based on the step-by-step answer content generated according to multiple inference steps in the inference chain. The training process is supervised by the step-by-step answer content generated in reverse by the second AI generation model and the high-quality answer content, so as to complete the training of the first AI generation model.

[0081] Corresponding to Embodiment 3, this application also provides an apparatus for providing consulting service information, which may include: The input data receiving unit is used to receive input data related to consultation services entered by users in the commodity information service system; The answer content generation unit is used to generate corresponding answer content based on the input data using a first AI generation model, or to generate reference answer content for human service providers; wherein, the first AI generation model is specifically used to generate the final answer content based on reasoning according to multiple reasoning steps included in the reasoning chain corresponding to the consultation service; the first AI generation model is obtained by training according to the method described in the aforementioned Embodiment 1 or Embodiment 2. The answer content output unit is used to output and display the answer content generated by the first AI generation model.

[0082] The reasoning steps in the reasoning chain corresponding to the consulting service include: Intent recognition, knowledge assessment, strategy formulation, and expression adjustment; The reasoning steps corresponding to intent recognition are used to analyze the user's input inquiry and identify the user's core needs and / or potential intents. The reasoning steps corresponding to the knowledge assessment are used to associate the knowledge base, retrieve and filter key information related to the current request and / or intent; The reasoning steps corresponding to the strategy formulation are used to plan the core response strategy based on the analysis results of intent recognition and knowledge assessment. The reasoning steps corresponding to the expression adjustment are used to determine the language style and emotional tone of the response.

[0083] Corresponding to Embodiment 4, this application also provides a model training apparatus, which may include: The defect detection unit is used to collect the response content generated by the AI ​​generation model to be trained in response to the input data, filter out the poor response content, check the defects in the poor response content, and generate prompt information based on the target input data corresponding to the poor response content and the detected defects. The regeneration unit is used to invoke the AI ​​generation model according to the prompt information to regenerate the answer content for the target input data; The training unit is used to combine the target input data, the poor-quality answer content, and the regenerated answer content into preference data pairs, so as to use a training data set composed of multiple preference data pairs to perform preference alignment training on the AI ​​generation model based on reinforcement learning.

[0084] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0085] And an electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0086] A computer program product includes a computer program / computer executable instructions that, when executed by a processor in an electronic device, implement the steps of the method described in the foregoing method embodiments.

[0087] in, Figure 6 An exemplary architecture of an electronic device is shown, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620 can communicate with each other via a communication bus 630.

[0088] The processor 610 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to achieve the technical solution provided in this application.

[0089] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store the operating system 621 for controlling the operation of the electronic device 600, and the basic input / output system (BIOS) for controlling the low-level operations of the electronic device 600. Additionally, it can store a web browser 623, a data storage management system 624, and a consultation service processing system 625, etc. The aforementioned consultation service processing system 625 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 620 and executed by the processor 610.

[0090] Input / output interface 613 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0091] Network interface 614 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0092] Bus 630 includes a pathway for transmitting information between various components of the device, such as processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620.

[0093] It should be noted that although the above-described device only shows the processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, memory 620, bus 630, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0094] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0095] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0096] The foregoing has provided a detailed description of the model training, consulting service information provision methods, and electronic devices offered in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A model training method, characterized in that, The method is used to train a first artificial intelligence (AI) generative model, including: Obtain a first training data set, which includes multiple first data pairs. Each first data pair includes first input data and corresponding high-quality answer content. Each first data pair also corresponds to step-by-step answer content at multiple inference steps in a preset inference chain. The first AI generation model is trained in the first stage using the first training data set, so that the first AI generation model generates the final answer content based on reasoning according to multiple reasoning steps in the inference chain. The first AI generation model, which has completed the first stage of training, collects the answers generated by the second input data, filters out inferior answers, checks the defects in the inferior answers, and generates prompt information based on the target second input data corresponding to the inferior answers and the detected defects. The first AI generation model that has completed the first stage of training is invoked according to the prompt information to regenerate the answer content for the target second input data; The target second input data, the poor-quality answer content, and the regenerated answer content are combined into preference data pairs, so that the first AI generation model can be trained in the second stage using a second training data set composed of multiple preference data pairs.

2. The method according to claim 1, characterized in that, The acquisition of the first training data set includes: Obtain the plurality of first data pairs; The second AI generation model is used to reverse-generate step-by-step answers for multiple inference steps in a pre-set inference chain for the first data pair, and the first training data set is generated based on the first data pair and the corresponding step-by-step answers for multiple inference steps.

3. The method according to claim 1, characterized in that, The first input data is obtained from historical service records generated within the consultation service system of the commodity information service system, including the consultation questions entered by the user and related context data; The pre-set inference chain includes multiple inference steps, such as: Intent recognition, knowledge assessment, strategy formulation, and expression adjustment; The reasoning steps corresponding to intent recognition are used to analyze the user's input inquiry and identify the user's core needs and / or potential intents. The reasoning steps corresponding to the knowledge assessment are used to associate the knowledge base, retrieve and filter key information related to the current request and / or intent; The reasoning steps corresponding to the strategy formulation are used to plan the core response strategy based on the analysis results of intent recognition and knowledge assessment. The reasoning steps corresponding to the expression adjustment are used to determine the language style and emotional tone of the response.

4. The method according to claim 1, characterized in that, The high-quality answers in the first data pair were obtained through expert annotation or annotation by a third-party AI generation model.

5. The method according to claim 1, characterized in that, The inspection of the defects in the poor-quality answers includes: The step-by-step answer content generated at multiple inference steps in the inference chain by the first AI generation model that has completed the first stage of training during the process of generating the poor answer content for the target second input data; Defect detection is performed by analyzing the poor-quality answers and the step-by-step answers.

6. The method according to claim 1, characterized in that, The prompt information includes a description of how to improve the detected defects, so that the first AI generation model, which has completed the first stage of training, can regenerate the response content for the target second input data by referring to the description of the improvement methods.

7. The method according to claim 1, characterized in that, When generating prompt information based on the target second input data corresponding to the poor-quality answer content and the detected defects, the poor-quality answer content is added to the prompt information so that the answer content regenerated by the first AI generation model that has completed the first stage of training for the target second input data is different from the poor-quality answer content.

8. The method according to claim 1, characterized in that, The second stage of training includes training the first AI generation model using a preference alignment method based on reinforcement learning.

9. A model training method, characterized in that, The method is used to train a first AI-generated model, including: Acquire multiple data pairs, each including input data and corresponding high-quality answer content; The second AI generation model is used to reverse-generate step-by-step answer content for multiple inference steps in the preset inference chain for the data pair, and a training data set is generated based on the data pair and the corresponding step-by-step answer content for multiple inference steps. The first AI generation model is trained using the training dataset. During the training process, the first AI generation model is guided to generate step-by-step answer content according to multiple reasoning steps in the inference chain, and then generate the final answer content. The training process is supervised by the step-by-step answer content generated in reverse by the second AI generation model and the high-quality answer content, so as to complete the training of the first AI generation model.

10. A method for providing consulting service information, characterized in that, include: Receive user input data related to consultation services from the product information service system; The first AI generation model is used to generate corresponding answer content for the input data, or to generate reference answer content for human service providers; wherein, the first AI generation model is specifically used to generate the final answer content based on reasoning according to multiple reasoning steps included in the reasoning chain corresponding to the consultation service; the first AI generation model is obtained by training according to the method described in any one of claims 1 to 9. The output of the answer generated by the first AI generation model is displayed.

11. The method according to claim 10, characterized in that, The reasoning steps in the reasoning chain corresponding to the consulting service include: Intent recognition, knowledge assessment, strategy formulation, and expression adjustment; The reasoning steps corresponding to intent recognition are used to analyze the user's input inquiry and identify the user's core needs and / or potential intents. The reasoning steps corresponding to the knowledge assessment are used to associate the knowledge base, retrieve and filter key information related to the current request and / or intent; The reasoning steps corresponding to the strategy formulation are used to plan the core response strategy based on the analysis results of intent recognition and knowledge assessment. The reasoning steps corresponding to the expression adjustment are used to determine the language style and emotional tone of the response.

12. A model training method, characterized in that, include: The AI ​​generation model under training collects the answers generated by the input data, filters out the poor-quality answers, checks the defects in the poor-quality answers, and generates prompt information based on the target input data corresponding to the poor-quality answers and the detected defects. The AI ​​generation model is invoked according to the prompt information to regenerate the answer content for the target input data; The target input data, the poor-quality answer content, and the regenerated answer content are combined into preference data pairs, so that the AI ​​generation model can be trained based on reinforcement learning using a training data set composed of multiple preference data pairs.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 12.

14. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program / computer-executable instructions, characterized in that, When the computer program / computer-executable instructions are executed by a processor in an electronic device, they implement the steps of the method according to any one of claims 1 to 12.

Images