Model training method, model application method, device, equipment, computer readable storage medium and computer program product

Abstract

The application provides a model training method, a model application method, a device, equipment, a computer readable storage medium and a computer program product, comprising: constructing a second sample data set based on a first sample data set; each first data sample in the second sample data set at least includes: a first query request included in the first sample data set and first reply data corresponding to the first query request, second reply data related to the first reply data, first reflection suggestions based on the second reply data, third reply data for the first query request based on the first reflection suggestions, and reflection result labels; using sample data with valid reflection indicated by the reflection result labels in the second sample data set, model training is performed on the first reflection model to obtain a second reflection model; and using the second sample data set, model training is performed on the first sorting model to obtain a second sorting model; the self-reflection time of the large model is shortened, and the self-reflection efficiency of the large model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, a model application method, an apparatus, a device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the development of artificial intelligence technology, significant progress has been made in the development of various types of Large Language Models (LLMs), which have demonstrated powerful capabilities in various applications such as natural language processing, machine translation, and code generation. Currently, as application scenarios become more complex, the scale of large language models is also gradually increasing. Ensuring the accuracy of recognition by large language models and correcting errors in the recognition results has become a pressing issue. To address this problem, a self-reflection technique for large language models has been proposed, enabling them to evaluate and improve their output, thereby enhancing its quality and accuracy. However, current self-reflection techniques for large language models are time-consuming to implement, resulting in low efficiency. Summary of the Invention

[0003] This application provides a model application method, apparatus, device, computer-readable storage medium, and computer program product, which shortens the self-reflection time of large models and improves the self-reflection efficiency of large models.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a model training method, including:

[0006] Based on the first sample dataset, a second sample dataset is constructed; wherein each first data sample in the second sample dataset includes at least: a first query request and a first response data corresponding to the first query request included in the first sample dataset, a second response data related to the first response data, a first reflection suggestion based on the second response data, a third response data for the first query request based on the first reflection suggestion, and a reflection result label;

[0007] The second reflection model is trained by using the reflection result labels in the second sample dataset to indicate the sample data of effective reflection, thereby obtaining the second reflection model; wherein, the first reflection model is used to perform reflection analysis on the response data corresponding to the query request;

[0008] The first ranking model is trained using the second sample dataset to obtain the second ranking model; wherein the first ranking model is used to perform quality assessment and ranking of at least one reflection suggestion.

[0009] In the above scheme, constructing the second sample dataset based on the first sample dataset includes:

[0010] Obtain the first sample dataset; wherein, the first sample dataset is a sample dataset used to perform local modeling processing on the first large model to obtain the second large model;

[0011] Obtain each first response data from the first sample dataset;

[0012] Modify each of the first response data to obtain at least one second response data;

[0013] The second sample dataset is generated based on the first sample dataset and at least one second response data corresponding to each of the first response data.

[0014] In the above scheme, generating the second sample dataset based on the first sample dataset and at least one second response data corresponding to each first response data includes:

[0015] The first reflection model is used to reflect on each first query request, the first response data corresponding to each first query request, and at least one second response data to obtain at least one first reflection suggestion corresponding to each second response data.

[0016] The second major model reflects and adopts each second response data and at least one first reflection suggestion corresponding to each second response data, and generates third response data corresponding to each first reflection suggestion for each second response data.

[0017] Based on each of the first response data and each of the corresponding third response data, the corresponding first data sample is determined, and then the second sample dataset is obtained.

[0018] In the above scheme, determining the corresponding first data sample based on each first response data and each corresponding third response data includes:

[0019] The consistency between each first response data and each corresponding third response data is determined by the first reflection model to obtain the judgment result;

[0020] Based on the judgment result, the reflection result label corresponding to the first response data is determined; wherein, the reflection result label is a valid reflection label or an invalid reflection label;

[0021] Each of the third response data is determined, and the first query request, the first response data, the second response data, and the reflection result label corresponding to each of the third response data are the first sample data.

[0022] This application provides a model application method, including:

[0023] If it is determined that the current response data generated by the second model for the current query request needs to be reflected upon, k reference reflection suggestions are generated for the current response data through the second reflection model; where k is an integer greater than 1, and the second reflection model is trained based on the model training method described in any one of claims 1 to 4;

[0024] The k reference reflection suggestions are sorted using a second sorting model to obtain the target reflection suggestions; wherein the second sorting model is trained based on the model training method described in any one of claims 1 to 4;

[0025] Based on the target reflection suggestions, the second major model generates the target response data corresponding to the current query request.

[0026] The method in the above scheme further includes:

[0027] Obtain the first historical response data and the corresponding first historical reflection suggestion corresponding to the current query request before the current time.

[0028] The current query request, the first historical response data, the current response data, and the first historical reflection suggestion are reflected upon and judged to obtain a first result; wherein, the first result is used to indicate whether it is necessary to reflect on the current response data generated by the second model in response to the current query request.

[0029] In the above scheme, generating the target response data corresponding to the current query request based on the target reflection suggestion using the second major model includes:

[0030] Based on the target reflection suggestions, the second major model generates the first reference response data corresponding to the current query request;

[0031] The current query request, the second historical response data and the second historical reflection suggestions corresponding to the first reference response data before the first reference response data are generated, and the first reference response data are reflected and judged to obtain a second result;

[0032] If the second result indicates that no reflection is needed on the first reference response data, then the target response data is determined to be the first reference response data.

[0033] The method in the above scheme further includes:

[0034] If the second result indicates that the first reference response data needs to be reconsidered, store the first reference response data;

[0035] After updating the current response data to the first reference response data, the step "generate k reference reflection suggestions for the current response data through the second reflection model" is executed until the third result of the reflection judgment on the second reference response data generated by the second model for the current query request indicates that the second reference response data does not need to be reflected, then the target response data is determined to be the second reference response data.

[0036] This application provides a model training device, comprising:

[0037] The acquisition module is used to determine a second sample dataset; wherein each first data sample in the second sample dataset includes at least: a first query request, first response data corresponding to the first query request, second response data related to the first response data, a first reflection suggestion based on the second response data, a third response data for the first query request based on the first reflection suggestion, and a reflection result label;

[0038] The training module is used to train the first reflection model by using the reflection result labels in the second sample dataset to indicate the sample data of valid reflections, so as to obtain the second reflection model.

[0039] The training module is also used to train the first ranking model using the second sample dataset to obtain a second ranking model; wherein the first ranking model is used to perform quality assessment and ranking of at least one reflection suggestion.

[0040] This application provides a model application device, including:

[0041] The generation module is used to generate k reference reflection suggestions for the current response data generated by the second model in response to the current query request if it is determined that reflection is needed; wherein, k is an integer greater than 1, and the second reflection model is trained based on the model training method described above.

[0042] The sorting module is used to sort the k reflection suggestions using a second sorting model to obtain the target reflection suggestion; wherein the second sorting model is trained based on the model training method described above.

[0043] The generation module is also used to generate target response data corresponding to the current query request based on the target reflection suggestion through the second major model.

[0044] This application provides a model training device, including:

[0045] The first memory is used to store executable instructions;

[0046] A first processor, when executing executable instructions stored in the first memory, implements the method described in any of the preceding descriptions.

[0047] This application provides a model application device, including:

[0048] The second memory is used to store executable instructions;

[0049] The second processor, when executing executable instructions stored in the second memory, implements the method described in any of the preceding descriptions.

[0050] This application provides a computer-readable storage medium storing executable instructions for implementing the method described in any of the above-described embodiments when executed by a processor.

[0051] This application provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the method described in any of the above-described embodiments.

[0052] The embodiments of this application have the following beneficial effects:

[0053] By constructing a second sample dataset based on a first sample dataset, and using the reflection result labels in the second sample dataset to indicate valid reflection sample data, the first reflection model is trained to obtain a second reflection model. Similarly, the first ranking model is trained using the second sample dataset to obtain a second ranking model. Thus, by determining a second sample dataset containing at least the following sample data: a first query request, first response data corresponding to the first query request, second response data related to the first response data, first reflection suggestions based on the second response data, third response data based on the first reflection suggestions for the first query request, and reflection result labels, a second reflection model and a second ranking model are obtained. Since the sample data in the second training sample dataset represents different possible responses to the query request and their corresponding reflection suggestions, the reflection suggestions obtained by the second reflection model are more accurate, improving reflection efficiency. Furthermore, using the second ranking model to rank the reflection suggestions by reliability allows for rapid identification of valid reflection suggestions, reducing the reflection processing time for selecting reliable reflection suggestions when processing large models. Attached Figure Description

[0054] Figure 1This is an optional flowchart illustrating the model training method provided in the embodiments of this application;

[0055] Figure 2 This is a schematic diagram illustrating the implementation process of an application embodiment of the model training method provided in this application.

[0056] Figure 3 This is a schematic diagram of an optional application process of the model application method provided in the embodiments of this application;

[0057] Figure 4 This is a schematic diagram illustrating the implementation process of an application embodiment of the model application method provided in this application.

[0058] Figure 5 This is an optional structural schematic diagram of the model training device provided in the embodiments of this application;

[0059] Figure 6 This is an optional structural schematic diagram of the model application device provided in the embodiments of this application;

[0060] Figure 7 This is an optional structural diagram of the model training device provided in the embodiments of this application;

[0061] Figure 8 This is an optional structural schematic diagram of the model application device provided in the embodiments of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0064] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0065] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0067] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0068] Large Language Models (LLMs) are deep learning models that can be trained on large amounts of text data to generate natural language text or understand the meaning of language text. LLMs can handle various natural language tasks, such as text classification, question answering, and dialogue, and are an important pathway to artificial intelligence. Currently, LLMs employ a Transformer architecture and pre-training objectives (such as Language Modeling) similar to small models, differing only in increased model size, training data, and computational resources.

[0069] Multi-Model LLM refers to a large language model that can be trained by combining modal information such as audio, images, and text, enabling it to process multi-modal data. Currently, commonly used multi-model LLMs can take audio, images, and text as input to generate text responses, and further, can output audio, images, text, and other content processed by the model.

[0070] Supervised fine-tuning (SFT) refers to the fine-tuning of a model using labeled data to adapt the model to a specific task or domain.

[0071] This application provides a model training method and model application method, apparatus, device, computer-readable storage medium, and computer program product. It trains a model by determining a second sample dataset, including at least a first query request, first response data corresponding to the first query request, second response data related to the first response data, a first reflection suggestion based on the second response data, a third response data for the first query request based on the first reflection suggestion, and reflection result labels. This yields a second reflection model and a second ranking model. The second reflection model and the second ranking model are then used to reflect on and rank the current response data generated by the second model for the current query request, thereby obtaining the target response data corresponding to the current query request. This achieves model training of the first reflection model and the first ranking model based on the first sample dataset, resulting in a second reflection model and a second ranking model that can be used for reflection applications. Applying the second reflection model and the second ranking model improves reflection efficiency, simplifies the reflection process, and reduces the time spent on the reflection stage.

[0072] In some embodiments, the model training device and the model application device can be devices with computing capabilities, such as computer devices, server devices, virtual computer devices, etc. In some application scenarios, the model training device and the model application device can be a single device or two independent devices. When the model training device and the model application device are two independent devices, the model training device can be a server-side device, and the model application device can be a client-side device. When the model training device and the model application device are a single device, it can be an online device, such as a server device. The specific choice depends on the actual situation, and this application embodiment does not limit the choice.

[0073] See Figure 1 , Figure 1 The following is an optional flowchart illustrating a model training method provided in an embodiment of this application. The method is applied to a model training device and includes steps S101 to S104, as follows:

[0074] S101. Based on the first sample dataset, construct the second sample dataset.

[0075] Each first data sample in the second sample dataset includes at least: a first query request and a first response data corresponding to the first query request included in the first sample dataset, a second response data related to the first response data, a first reflection suggestion based on the second response data, a third response data for the first query request based on the first reflection suggestion, and a reflection result label.

[0076] In this embodiment, the first sample dataset is typically application sample data from the application scenario where the model training device is located. Each sample dataset corresponds to at least a first query request and a first response data corresponding to the first query request. The second response data related to the first response data is obtained by modifying the first response. The second response data related to the first response data may include one or more data sets, the specific number of which can be determined by the actual situation. The third response data is response data generated based on the first reflection suggestion for the first query request. The reflection result label is obtained by performing consistency analysis on the first query request, the first response data corresponding to the first query request, the second response data related to the first response data, the first reflection suggestion based on the second response data, and the third response data for the first query request based on the first reflection suggestion. If, after performing consistency analysis on the first query request, the first response data corresponding to the first query request, the second response data related to the first response data, the first reflection suggestion based on the second response data, and the third response data for the first query request based on the first reflection suggestion, and it is determined that the first response data for the first query request, the second response data related to the first response data, the first reflection suggestion based on the second response data, and the third response data for the first query request based on the first reflection suggestion are consistent, then the reflection result label for this set of data can be recorded as a valid reflection label. This means that the third response data obtained under the first reflection suggestion of the second response data is close to the expected response, indicating that the reflection suggestion is accurate. Otherwise, the reflection result label for this set of data is recorded as an invalid reflection label.

[0077] The sample data in the first sample dataset is processed to obtain the second sample dataset.

[0078] S102. Using the labels of the reflection results in the second sample dataset to indicate the sample data of effective reflection, train the first reflection model to obtain the second reflection model.

[0079] The first reflection model is used to reflect on and analyze the response data corresponding to the query request, and generate reflection suggestions.

[0080] In this embodiment, sample data with valid reflection labels are obtained from the second sample dataset. Then, a model training method is used to train the first reflection model based on the sample data with valid reflection labels, resulting in a trained second reflection model. This approach, by using sample data with good reflection performance for model training, ensures that the trained second reflection model has high quality and good reflection efficiency.

[0081] S103. The first ranking model is trained using the second sample dataset to obtain the second ranking model.

[0082] The first ranking model is used to assess and rank at least one reflective suggestion.

[0083] In this embodiment, the first ranking model is a model for ranking reflection suggestions. Since the second sample dataset includes sample data with reflection results labeled as valid or invalid reflections, the first ranking model can be trained using the second sample dataset. For example, during this training process, sample data with reflection results labeled as valid reflections can be ranked first, and sample data with reflection results labeled as invalid reflections can be ranked last. In this way, a trained second ranking model can be obtained, making the ranking results of the second ranking model more reliable and effective.

[0084] Based on the foregoing embodiments, in other embodiments of this application, S101, based on the first sample dataset, constructs the second sample dataset, which can be achieved by the following steps:

[0085] Obtain the first sample dataset; where the first sample dataset is the sample dataset used to perform local modeling processing on the first large model to obtain the second large model;

[0086] Retrieve each first response data point from the first sample dataset;

[0087] Modify each first response data to obtain at least one second response data;

[0088] A second sample dataset is generated based on the first sample dataset and at least one second response data corresponding to each first response data.

[0089] In this embodiment, the first sample dataset can be stored in the storage unit of the model training device or in a cloud storage area accessible by the model training device. The sample data in the first sample dataset is sample data with characteristics of local operation or task of the model training device. The first large model is a model used to generate corresponding response content after receiving user query data, such as a large language model or a multimodal large language model. The second large model is a large model obtained by locally modeling the first large model using the first sample dataset; that is, the second large model is a personalized large model that conforms to the model training device. Response data corresponding to each query request is obtained from the first sample dataset, resulting in each first response data. Then, each obtained first response data is modified according to the response modification rules to obtain at least one second response data corresponding to each first response data. The specific number of second response data can be determined by the response modification rules.

[0090] After obtaining at least one second response data corresponding to each first response data, perform data statistical processing on the first sample dataset and at least one second response data corresponding to each first response data in the first sample dataset to obtain the second sample dataset.

[0091] Thus, after step S101 is completed as described above, steps S102 to S103 are executed to train the model and obtain the second reflection model and the second ranking model.

[0092] Based on the foregoing embodiments, in other embodiments of this application, the step of generating a second sample dataset based on a first sample dataset and at least one second response data corresponding to each first response data can be implemented by the following steps:

[0093] The first reflection model is used to reflect on each first query request, the first response data corresponding to each first query request, and at least one second response data to obtain at least one first reflection suggestion corresponding to each second response data.

[0094] The second major model reflects and adopts each second response data and at least one first reflection suggestion corresponding to each second response data, and generates a third response data corresponding to each first reflection suggestion for each second response data.

[0095] Based on each first response data and each corresponding third response data, the corresponding first data sample is determined, and then the second sample dataset is obtained.

[0096] In this embodiment of the application, each first query request, the first response data corresponding to each first query request, and at least one second response data are input into the first reflection model, so that the first reflection model can perform reflection processing on each second response data based on each first query request and each first response data, and obtain at least one first reflection suggestion for each second response data.

[0097] After obtaining at least one first reflection suggestion for each second response data corresponding to each first query request, each second response data and its corresponding at least one first reflection suggestion are input into the second large model. The second large model has a reflection and adoption function. Based on each second response data, the second large model adopts at least one first reflection suggestion corresponding to each second response data, and then generates a corresponding response data based on each first reflection suggestion, resulting in third response data. Since each second response data corresponds to at least one first reflection suggestion, each second response data will generate the same number of third response data as the number of first reflection suggestions. For example, if each second response data corresponds to 3 first reflection suggestions, each second response data will generate 3 third response data. If each first response data corresponds to 4 second response data, then each first response data corresponds to 4*3=12 third response data. In this way, the second large model generates response data based on reflection suggestions, making the obtained response data more likely to meet the user's actual requirements. Finally, the trained model is trained in the desired direction, ensuring the model's reliability.

[0098] After obtaining the third response data for each first reflection suggestion, each first response data and its corresponding third response data are grouped and analyzed. Then, based on each first response data and its corresponding third response data, a corresponding first data sample is generated. In this way, the second sample dataset can be obtained.

[0099] Based on the foregoing embodiments, in other embodiments of this application, the step of determining the corresponding first data sample based on each first response data and each corresponding third response data can be implemented by the following steps:

[0100] The consistency between each first response data point and its corresponding third response data point is determined using the first reflection model to obtain the judgment result.

[0101] Based on the judgment results, determine the reflection result label for the corresponding first response data; whereby the reflection result label is either a valid reflection label or an invalid reflection label;

[0102] Each third response data point is identified as the first sample data, consisting of the first query request, the first response data, the second response data, and the corresponding reflection result label.

[0103] In this embodiment, each third response data and its corresponding third response data are input into a first reflection model. The first reflection model performs a response data consistency judgment analysis on each first response data and its corresponding third response data to obtain a judgment result. The judgment result can indicate that the input data is consistent or inconsistent. Then, the model training device determines the reflection result label of the input first response data based on the judgment result. For example, if the judgment result indicates that the data is inconsistent, the reflection result label is an invalid reflection label; if the judgment result indicates that the data is consistent, the reflection result label is a valid reflection label. For example, if the first response data is data A, and the corresponding three third response data are B, C, and D, a second model must perform consistency analysis on A and B, A and C, and A and D respectively to obtain the reflection result labels for A and B, A and C, and A and D.

[0104] Each third response data point, along with its corresponding first query request, first response data, second response data, and reflection result label, is combined to obtain the first sample data. For example, suppose the third response data B, the first query request corresponding to third response data B, the first response data A corresponding to third response data B, and the second response data corresponding to third response data B are grouped together to obtain the first sample data.

[0105] Based on the foregoing embodiments, this application provides an implementation example of a model training method, such as... Figure 2 The diagram shown is a schematic representation of an implementation process provided in this application, which includes at least the following implementation steps:

[0106] Step a11: Obtain the user input query request (query) and the corresponding response (res) from the existing SFT dataset.

[0107] The SFT dataset is the local dataset for obtaining the initial large model corresponding to LLM-G, which is the first large model mentioned above. In other words, after obtaining the pre-trained large model, the SFT dataset is used to localize the pre-trained large model, achieving supervised fine-tuning so that the resulting first large model conforms to the current application scenario. The SFT dataset stores some query requests input by users in the current application scenario, and the response content to these query requests in that application scenario.

[0108] Step a12: Based on the principle of breaking rewriting, rewrite res to obtain the rewritten result.

[0109] The principles of destructive rewriting can include, for example, not changing the core meaning of the original response, rewriting grammar and vocabulary, simplifying and clarifying sentences, using colloquial language, and modifying the meaning to be opposite to the core meaning of the original response. To improve rewriting efficiency, a destructive rewriting model can be designed based on these principles to rewrite the response and obtain the corresponding rewritten result.

[0110] For example, the destruction rewriting model is used to rewrite and destroy res, resulting in two rewriting results, which can be specifically denoted as [res-stage1-1, res-stage1-2].

[0111] Step a13: Using LLM-R, compare the original response res and the rewritten results [res-stage1-1, res-stage1-2] based on the query to generate at least two reflection suggestions.

[0112] Here, llm-R is a pre-trained reflective large language model used for reflecting on responses. For example, (query, res, res-stage1-1) is input into llm-R, and llm-R performs reflective processing on (query, res, res-stage1-1) to generate multiple reflective suggestions, which can be denoted as list[ref-1-res-stage1-1, ref-2-res-stage1-1].

[0113] Step a14: Input (res-stage1-1, list[ref-1-res-stage1-1, ref-2-res-stage1-1]) into llm-G and get the output result ([res-stage2-1-res-stage1-1, res-stage2-2-res-stage1-1]).

[0114] LLM-G performs reflection and adoption processing based on the input (res-stage1-1, list[ref-1-res-stage1-1, ref-2-res-stage1-1]), generates the result corresponding to each reflection suggestion, and obtains the response result corresponding to the reflection suggestion ref-1-res-stage1-1 [res-stage2-1-res-stage1-1, and the response result corresponding to the reflection suggestion res-stage2-2-res-stage1-1 res-stage2-2-res-stage1-1].

[0115] Step a15: Input the response data into llm-R to obtain the reflection judgment label.

[0116] The response data pair consists of each query's corresponding `res` and each response result corresponding to the rewritten result. For example, `res` and `res-stage2-1-res-stage1-1` form one response data pair, and `res` and `res-stage2-2-res-stage1-1` form another. Thus, by using LLM-R to determine whether the response result generated after the adoption reflection is consistent with the original `res`, the effectiveness of the reflection process can be judged.

[0117] Assuming that the reflection judgment result obtained by LLM-R is consistent with res-stage2-1-res-stage1-1, it indicates that the reflection process can effectively restore the original result and is an effective reflection correction process. Therefore, the reflection result label can be determined as an effective reflection label. Correspondingly, a set of sample data can be recorded as (query, res, res-stage1-1, ref-1-res-stage1-1, res-stage2-1-res-stage1-1, effective reflection).

[0118] Suppose that the reflection judgment result obtained by LLM-R is inconsistent with res-stage2-2-res-stage1-1, indicating that the reflection process cannot restore the original result and is an invalid reflection process. Therefore, the reflection result label can be determined as an invalid reflection label, and the corresponding sample data can be recorded as (query, res, res-stage1-1, ref-2-res-stage1-1, res-stage2-2-res-stage1-1).

[0119] Step a16: Train the models for llm-R and llm-R-sort respectively to obtain the updated llm-R and the updated llm-R-sort.

[0120] Specifically, when training the model for llm-R, the sample data with the reflection result labels obtained in step a15 as valid reflection labels are used to train llm-R, so that when the trained llm-R query and res-stage1-1 are executed, the output response result is ref-1-res-stage1-1.

[0121] The `llm-R-sort` model is used to rank reflective suggestions. During training, `llm-R-sort` can generate either valid or invalid reflective responses. Correspondingly, for the trained `llm-R` model, given the query `res-stage1-1`, the expected recommended responses are `ref-1-res-stage1-1`, and the expected non-recommended responses are `ref-2-res-stage1-1`. In other words, when the trained `llm-R` model prioritizes recommendations from left to right, `ref-1-res-stage1-1` is ranked higher than `ref-2-res-stage1-1`.

[0122] In this way, by constructing training sample data for the reflection model and the reflection result ranking model using existing local sample datasets, the difficulty of collecting training sample data is reduced, and the collected training sample data has a certain orientation, which can effectively ensure the reliability and accuracy of the trained model and improve the efficiency of model training.

[0123] Understandably, by constructing a second sample dataset based on the first sample dataset, and then using the reflection result labels in the second sample dataset to indicate the effective reflection sample data, the first reflection model is trained to obtain the second reflection model. Similarly, the first ranking model is trained using the second sample dataset to obtain the second ranking model. Thus, by determining a second sample dataset containing at least the first query request, the first response data corresponding to the first query request, the second response data related to the first response data, the first reflection suggestion based on the second response data, the third response data based on the first reflection suggestion for the first query request, and reflection result labels, a second reflection model and a second ranking model are obtained. These models are then used to reflect on and rank the current response data generated by the second main model for the current query request, obtaining the target response data corresponding to the current query request. This achieves the goal of constructing a second sample dataset based on the first sample dataset to train the first reflection model and the first ranking model, resulting in a second reflection model and a second ranking model that can be used for reflection applications. Applying these models improves reflection efficiency, simplifies the reflection process, and reduces the time spent on the reflection phase.

[0124] Based on the foregoing embodiments, see Figure 2 , Figure 2An optional flowchart illustrating the model application method provided in this application embodiment, wherein the method is applied to a model application device, including steps S201 to S204, as follows:

[0125] S201. If it is determined that the current response data generated by the second major model for the current query request needs to be reflected upon, k reference reflection suggestions are generated for the current response data through the second reflection model.

[0126] Where k is an integer greater than 1, the second reflection model is trained based on the aforementioned model training method.

[0127] In this embodiment, the model application device obtains the result of whether the current response data generated by the second model for the current query request needs to be reflected. If the result indicates that the current response data does not need to be reflected, the subsequent operation ends, or the current response data corresponding to the current query request is output to achieve a response to the current query request. If the result indicates that the current response data needs to be reflected, k reference reflection suggestions are generated for the current response data through the second reflection model. The value of k is usually determined by the second reflection model, but in some application scenarios, it can also be an empirical value set based on a large number of experiments, or an empirical value set based on user needs. The specific value is determined by the actual situation and is not specifically limited here.

[0128] For example, when the second model is deployed on the model application device, after receiving the current query request input by the user, the model application device generates the current response data corresponding to the current query request through the second model, and determines whether the current response data needs to be reflected. If the current response data needs to be reflected, when the model application device and the model training device are different devices, the model application device can use the second reflection model obtained from the model training device to reflect on the current response data, obtaining k reference reflection suggestions for the current response data. The second reflection model can be actively sent from the model training device to the model application device, or it can be obtained by the model application device through a model acquisition request and stored in the storage area of ​​the model application device. If the model application device and the model training device are the same device, the model application device can obtain it from its storage area.

[0129] S202. The k reference reflection suggestions are sorted using the second sorting model to obtain the target reflection suggestions.

[0130] The second ranking model is trained using the aforementioned model training method.

[0131] In this embodiment, the second ranking model can be actively sent from the model training device to the model application device, or it can be obtained by the model application device after sending a model acquisition request and stored in the storage area of ​​the model application device. If the model application device and the model training device are the same device, the model application device can obtain it from its storage area. K reference reflection suggestions are input into the second ranking model, which performs a quality assessment on the k reference reflection suggestions and ranks them based on the assessment results to obtain a ranking result. Then, the target reflection suggestion is determined based on the ranking result.

[0132] For example, when the ranking result output by the second ranking model is obtained by ranking from high to low evaluation quality, the first reference reflection suggestion in the ranking result can be selected as the target reflection suggestion, that is, the reference reflection suggestion with the best evaluation quality is selected from k reference reflection suggestions as the target reflection suggestion.

[0133] S203. Based on the target reflection suggestions, the second major model generates the target response data corresponding to the current query request.

[0134] In this embodiment of the application, the target reflection suggestions are input into the second major model. The second major model adopts the target reflection suggestions and generates target response data corresponding to the current query request based on the target reflection suggestions.

[0135] Based on the foregoing embodiments, in other embodiments of this application, before step S201, the model application device may also be used to perform the following steps:

[0136] Retrieve the first historical response data and the corresponding first historical reflection suggestion for the current query request up to the current time.

[0137] The system reflects on and judges the current query request, the first historical response data, the current response data, and the first historical reflection suggestions to obtain the first result; the first result is used to indicate whether it is necessary to reflect on the current response data generated by the second model in response to the current query request.

[0138] In this embodiment, the first historical response data stores the response data generated by the second largest model for the current query request up to the current time. The corresponding first historical reflection suggestion is the reflection suggestion generated by the second reflection model for each response data in the first historical response data. It should be noted that the first historical response data may have no response data, indicating that the current response data obtained at the current time is the first response generated by the second largest model for the current query request.

[0139] Using a judgment rule, based on the current query request, the first historical response data, and the first historical reflection suggestions, the current response data is reflected and judged to determine whether the current response data needs to be reflected and processed, and the first result is obtained. The judgment rules used here may include one or more of the following: Whether the number of reflections on the current query request exceeds a preset number, for example, 5 times, meaning no more than 5 rounds of reflection; if the difference between the target reflection suggestion and the most recent reflection suggestion in the first historical reflection suggestion is small, it indicates that the reflection content is duplicated, and reflection should be stopped; if the suggestion quality assessment result of the target reflection suggestion is below a certain threshold, reflection should be stopped; if the difference between the response result obtained after the second model adopts the target reflection suggestion and the response result before adopting the target reflection suggestion is small, reflection should be stopped; if the second reflection model evaluates and scores the current number of responses generated by the second model, and then analyzes based on the scores of the historical reflection suggestions in the first historical reflection suggestions and the current score, for example, analyzing backwards from the current score, if there is a continuous decrease in score or a significant increase in score, reflection can be stopped; if extreme rules are continuously hit in the first historical response data and the current response data, such as the continuous appearance of sensitive keywords in the first historical response data and the current response data.

[0140] Thus, when the first result determines that the current response data generated by the second major model for the current query request needs to be reflected, the process of steps S201 to 203 is executed to realize the reflection processing of the current response data.

[0141] Based on the foregoing embodiments, in other embodiments of this application, the step of generating target response data corresponding to the current query request based on target reflection suggestions using the second major model can be implemented by the following steps:

[0142] The second major model generates the first reference response data corresponding to the current query request based on goal reflection suggestions;

[0143] The second result is obtained by reflecting on and judging the current query request, the second historical response data corresponding to the first reference response data before it was generated, the second historical reflection suggestions, and the first reference response data.

[0144] If the second result indicates that no reflection is needed on the first reference response data, then the target response data is determined as the first reference response data.

[0145] In this embodiment of the application, the process of generating target response data corresponding to the current query request based on the target reflection suggestion of the second major model can be an iterative process. That is, the second major model adopts the target reflection suggestion, generates the first reference response data corresponding to the current query request, and performs judgment processing on the first reference response data based on the aforementioned judgment rules to obtain a second result. When the second result indicates that the first reference response data does not need to be reflected, the first reference response data can be used as the target response data.

[0146] Based on the foregoing embodiments, in other embodiments of this application, the model application device can also be used to complete the following steps:

[0147] If the second result indicates that the first reference response data needs to be reconsidered, store the first reference response data;

[0148] After updating the current response data to the first reference response data, the step "generate k reference reflection suggestions for the current response data through the second reflection model" is executed until the third result of the reflection judgment on the second reference response data generated by the second model for the current query request indicates that the second reference response data does not need to be reflected. Then, the target response data is determined as the second reference response data.

[0149] In this embodiment of the application, when the second result indicates that the first reference response data needs to be reflected, the first reference response data is stored and processed to obtain the first historical response data. Then, after the current response data is updated to the first reference response data, the step S201 "generates k reference reflection suggestions for the current response data through the second reflection model" is executed until the judgment result of the second model in the last iteration on the second reference response data generated for the current query request is that no reflection is required, and the target response data is determined to be the second reference response data.

[0150] Based on the foregoing embodiments, this application provides a model application method, namely, a reasoning implementation process, such as... Figure 4 As shown, it includes at least the following implementation steps:

[0151] Step b11: Receive the query input from the user.

[0152] Step b12: llm-G generates the response RES corresponding to the query.

[0153] Here, llm-G corresponds to the second major model mentioned above. One implementation process for llm-G to generate a response RES corresponding to a query can be as follows: llm-G understands the user's instructions and the idea behind the subsequent response generation based on the query; that is, llm-G understands the user's intent based on the query, and then generates the response RES based on the query and the user's intent.

[0154] Step b13: The reflection controller determines whether to perform reflection on RES based on the Query, historical response content, and historical reflection content. If the reflection controller determines that reflection is not required, proceed to step b14; otherwise, proceed to step b15.

[0155] Step b14: Output the response RES.

[0156] Step b15: Based on the response RES, llm-R generates k reflection suggestions.

[0157] Here, llm-R corresponds to the aforementioned second reflection model.

[0158] Step b16, llm-R-sort sorts the k reflection suggestions to obtain the optimal reflection suggestion, and then inputs the optimal reflection suggestion into llm-G. Steps b12 to b16 are repeated until the reflection controller determines that no reflection is needed.

[0159] Here, llm-R-sort corresponds to the second ranking model mentioned above. The corresponding llm-R-sort sorts the k reflection suggestions to obtain the optimal reflection suggestions through the following steps: Semantic-level filtering is performed on the k reflection suggestions, such as deduplicating or merging two reflection records with the same content to obtain a filtering result. Then, a reflection quality assessment is performed on the filtering result to obtain a reflection quality assessment value. Based on the reflection quality assessment value, the reflection suggestions in the filtering result are ranked. This process can be implemented using a reflection quality ranking model, thereby selecting the optimal reflection results, topreflects.

[0160] This approach simplifies the model reflection process and reduces the time complexity of the reflection phase.

[0161] Understandably, when it is determined that the current response data generated by the second major model for the current query request needs to be reflected, the second reflection model generates k reference reflection suggestions for the current response data, the second ranking model sorts the k reference reflection suggestions to obtain the target reflection suggestions, and then the second major model generates the target response data corresponding to the current query request based on the target reflection suggestions. In this way, by determining a second sample dataset that includes at least a first query request, a first response data corresponding to the first query request, a second response data related to the first response data, a first reflection suggestion based on the second response data, a third response data based on the first reflection suggestion for the first query request, and reflection result labels, a second reflection model and a second ranking model are obtained. The second reflection model and the second ranking model are then used to reflect on and rank the current response data generated by the second model for the current query request to obtain the target response data corresponding to the current query request. This achieves the construction of a second sample dataset based on the first sample dataset to train the first reflection model and the first ranking model, resulting in a second reflection model and a second ranking model that can be used for reflection applications. Applying the second reflection model and the second ranking model improves reflection efficiency, simplifies the reflection process, and reduces the time spent on the reflection stage during the reflection process.

[0162] It should be noted that the aforementioned methods can be applied to the field of question-and-answer interaction, such as intelligent voice interaction scenarios for vehicles, intelligent question-and-answer application scenarios for education guidance, and question-and-answer application scenarios for government services. This is only an example and is not a specific limitation.

[0163] This application provides a model training device, such as... Figure 5 As shown, the model training device 1 may include:

[0164] Module 11 is used to construct a second sample dataset based on the first sample dataset; wherein each first data sample in the second sample dataset includes at least: a first query request, first response data corresponding to the first query request, second response data related to the first response data, first reflection suggestion based on the second response data, third response data based on the first reflection suggestion for the first query request, and reflection result label;

[0165] Training module 12 is used to train the first reflection model by using the reflection result labels in the second sample dataset to indicate the sample data of effective reflection, and to obtain the second reflection model; wherein, the first reflection model is used to perform reflection analysis on the response data corresponding to the query request and generate reflection suggestions;

[0166] Training module 12 is also used to train the first ranking model using the second sample dataset to obtain the second ranking model; wherein the first ranking model is used to perform quality assessment and ranking of at least one reflection suggestion.

[0167] In other embodiments of this application, the building module can be used to implement the following steps:

[0168] Obtain the first sample dataset; where the first sample dataset is the sample dataset used to perform local modeling processing on the first large model to obtain the second large model;

[0169] Retrieve each first response data point from the first sample dataset;

[0170] Modify each first response data to obtain at least one second response data;

[0171] A second sample dataset is generated based on the first sample dataset and at least one second response data corresponding to each first response data.

[0172] In other embodiments of this application, when the construction module generates the second sample dataset based on the first sample dataset and at least one second response data corresponding to each first response data, it can be achieved through the following steps:

[0173] The first reflection model is used to reflect on each first query request, the first response data corresponding to each first query request, and at least one second response data to obtain at least one first reflection suggestion corresponding to each second response data.

[0174] The second major model reflects and adopts each second response data and at least one first reflection suggestion corresponding to each second response data, and generates a third response data corresponding to each first reflection suggestion for each second response data.

[0175] Based on each first response data and each corresponding third response data, the corresponding first data sample is determined, and then the second sample dataset is obtained.

[0176] In other embodiments of this application, when the construction module performs the step of determining the corresponding first data sample based on each first response data and each corresponding third response data, it can be achieved through the following steps:

[0177] The consistency between each first response data point and its corresponding third response data point is determined using the first reflection model to obtain the judgment result.

[0178] Based on the judgment results, determine the reflection result tags for the corresponding first response data; among them, the reflection result tags are either valid reflection tags or invalid reflection tags;

[0179] Each third response data point is identified as the first sample data, consisting of the first query request, the first response data, the second response data, and the corresponding reflection result label.

[0180] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0181] This application provides a model application device, such as... Figure 6 As shown, the model application device 2 may include:

[0182] The generation module 21 is used to generate k reference reflection suggestions for the current response data generated by the second reflection model for the current query request if it is determined that reflection is needed; wherein, k is an integer greater than 1, and the second reflection model is trained based on the model training method of any one of claims 1 to 4.

[0183] The sorting module 22 is used to sort the k reference reflection suggestions through a second sorting model to obtain the target reflection suggestion; wherein the second sorting model is trained based on the model training method of any one of claims 1 to 4;

[0184] The generation module 21 is also used to generate target response data corresponding to the current query request based on the target reflection suggestions through the second major model.

[0185] In other embodiments of this application, the model application apparatus further includes:

[0186] The acquisition module is used to obtain the first historical response data and the corresponding first historical reflection suggestion corresponding to the current query request up to the current time.

[0187] The judgment module is used to reflect on and judge the current query request, the first historical response data, the current response data, and the first historical reflection suggestions to obtain the first result; wherein, the first result is used to indicate whether it is necessary to reflect on the current response data generated by the second model in response to the current query request.

[0188] In other embodiments of this application, the generation module can be implemented through the following steps:

[0189] The second major model generates the first reference response data corresponding to the current query request based on goal reflection suggestions;

[0190] The second result is obtained by reflecting on and judging the current query request, the second historical response data corresponding to the first reference response data before it was generated, the second historical reflection suggestions, and the first reference response data.

[0191] If the second result indicates that no reflection is needed on the first reference response data, then the target response data is determined as the first reference response data.

[0192] In other embodiments of this application, the generation module is further configured to perform the following steps:

[0193] If the second result indicates that the first reference response data needs to be reconsidered, store the first reference response data;

[0194] After updating the current response data to the first reference response data, the step "generate k reference reflection suggestions for the current response data through the second reflection model" is executed until the third result of the reflection judgment on the second reference response data generated by the second model for the current query request indicates that the second reference response data does not need to be reflected. Then, the target response data is determined as the second reference response data.

[0195] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0196] This application provides a model training device, such as... Figure 7 As shown, the model training device 3 may include a first memory 32 and a first processor 33. The first memory 32 and the first processor 33 are connected via a first communication bus 34. The first memory 32 stores executable instructions; the first processor 33 executes the executable instructions stored in the first memory 32 to implement the model training method provided in this embodiment.

[0197] This application provides a model application device, such as... Figure 8 As shown, the model application device 4 may include a second memory 42 and a second processor 43. The second memory 42 and the second processor 43 are connected via a second communication bus 44. The second memory 42 is used to store executable instructions; the second processor 43 is used to implement the model application method provided in this application embodiment when executing the executable instructions stored in the second memory 42.

[0198] This application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored and when executed by a processor, they will cause the processor to execute any of the above-described model application methods provided in this application.

[0199] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0200] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0201] As an example, executable instructions may, but do not necessarily, correspond to files in the file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0202] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0203] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A model training method, characterized in that, include: Based on the first sample dataset, a second sample dataset is constructed; wherein each first data sample in the second sample dataset includes at least: a first query request and a first response data corresponding to the first query request included in the first sample dataset, a second response data related to the first response data, a first reflection suggestion based on the second response data, a third response data for the first query request based on the first reflection suggestion, and a reflection result label; The second reflection model is trained using the reflection result labels in the second sample dataset to indicate the sample data of effective reflections. The first reflection model is then used to perform reflection analysis on the response data corresponding to the query request and generate reflection suggestions. The first ranking model is trained using the second sample dataset to obtain the second ranking model; wherein the first ranking model is used to perform quality assessment and ranking of at least one reflection suggestion.

2. The method according to claim 1, characterized in that, The construction of the second sample dataset based on the first sample dataset includes: Obtain the first sample dataset; wherein, the first sample dataset is a sample dataset used to perform local modeling processing on the first large model to obtain the second large model; Obtain each first response data from the first sample dataset; Modify each of the first response data to obtain at least one second response data; The second sample dataset is generated based on the first sample dataset and at least one second response data corresponding to each of the first response data.

3. The method according to claim 2, characterized in that, The step of generating the second sample dataset based on the first sample dataset and at least one second response data corresponding to each first response data includes: The first reflection model is used to reflect on each first query request, the first response data corresponding to each first query request, and at least one second response data to obtain at least one first reflection suggestion corresponding to each second response data. The second major model reflects and adopts each second response data and at least one first reflection suggestion corresponding to each second response data, and generates third response data corresponding to each first reflection suggestion for each second response data. Based on each of the first response data and each of the corresponding third response data, the corresponding first data sample is determined, and then the second sample dataset is obtained.

4. The method according to claim 3, characterized in that, The step of determining the corresponding first data sample based on each first response data and each corresponding third response data includes: The consistency between each first response data and each corresponding third response data is determined by the first reflection model to obtain the judgment result; Based on the judgment result, the reflection result label corresponding to the first response data is determined; wherein, the reflection result label is a valid reflection label or an invalid reflection label; Each of the third response data is determined, and the first query request, the first response data, the second response data, and the reflection result label corresponding to each of the third response data are the first sample data.

5. A method for applying a model, characterized in that, include: If it is determined that the current response data generated by the second model for the current query request needs to be reflected upon, k reference reflection suggestions are generated for the current response data through the second reflection model; where k is an integer greater than 1, and the second reflection model is trained based on the model training method described in any one of claims 1 to 4; The k reference reflection suggestions are sorted using a second sorting model to obtain the target reflection suggestions; wherein the second sorting model is trained based on the model training method described in any one of claims 1 to 4; Based on the target reflection suggestions, the second major model generates the target response data corresponding to the current query request.

6. The method according to claim 5, characterized in that, The method further includes: Obtain the first historical response data and the corresponding first historical reflection suggestion corresponding to the current query request before the current time. The current query request, the first historical response data, the current response data, and the first historical reflection suggestion are reflected upon and judged to obtain a first result; wherein, the first result is used to indicate whether it is necessary to reflect on the current response data generated by the second model in response to the current query request.

7. The method according to claim 5 or 6, characterized in that, The step of generating target response data corresponding to the current query request based on the target reflection suggestion using the second major model includes: Based on the target reflection suggestions, the second major model generates the first reference response data corresponding to the current query request; The current query request, the second historical response data and the second historical reflection suggestions corresponding to the first reference response data before the first reference response data are generated, and the first reference response data are reflected and judged to obtain a second result; If the second result indicates that no reflection is needed on the first reference response data, then the target response data is determined to be the first reference response data.

8. The method according to claim 7, characterized in that, The method further includes: If the second result indicates that the first reference response data needs to be reconsidered, store the first reference response data; After updating the current response data to the first reference response data, the step "generate k reference reflection suggestions for the current response data through the second reflection model" is executed until the third result of the reflection judgment on the second reference response data generated by the second model for the current query request indicates that the second reference response data does not need to be reflected, and then the target response data is determined to be the second reference response data.

9. A model training device, characterized in that, include: The acquisition module is used to determine a second sample dataset; wherein each first data sample in the second sample dataset includes at least: a first query request, first response data corresponding to the first query request, second response data related to the first response data, a first reflection suggestion based on the second response data, a third response data for the first query request based on the first reflection suggestion, and a reflection result label; The training module is used to train the first reflection model by using the reflection result labels in the second sample dataset to indicate the sample data of effective reflection, thereby obtaining the second reflection model; wherein, the first reflection model is used to perform reflection analysis on the response data corresponding to the query request; The training module is also used to train the first ranking model using the second sample dataset to obtain a second ranking model; wherein the first ranking model is used to perform quality assessment and ranking of at least one reflection suggestion.

10. A model application device, characterized in that, include: A generation module is used to generate k reference reflection suggestions for the current response data generated by the second model in response to the current query request if it is determined that reflection is needed; wherein, k is an integer greater than 1, and the second reflection model is trained based on the model training method described in any one of claims 1 to 4. A sorting module is used to sort the k reflection suggestions using a second sorting model to obtain target reflection suggestions; wherein the second sorting model is trained based on the model training method according to any one of claims 1 to 4; The generation module is also used to generate target response data corresponding to the current query request based on the target reflection suggestion through the second major model.

11. A model training device, characterized in that, include: The first memory is used to store executable instructions; A first processor, when executing executable instructions stored in the first memory, implements the method according to any one of claims 1 to 4.

12. A model application device, characterized in that, include: The second memory is used to store executable instructions; The second processor, when executing executable instructions stored in the second memory, implements the method according to any one of claims 5 to 8.

13. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the method of any one of claims 1 to 4, or claims 5 to 8, when executed by a processor.

14. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method described in any one of claims 1 to 4, or claims 5 to 8.

Images