Data generation method, electronic equipment and storage medium

Abstract

The invention discloses a data generation method, electronic equipment and a storage medium, and relates to the technical field of large model technology and natural language processing. The method comprises the following steps: constructing a plurality of prompt word texts based on a plurality of inquiry data; inputting the plurality of prompt word texts into a first generation model, and generating first reply data corresponding to the plurality of prompt word texts by utilizing the first generation model; any cue word text and the corresponding first reply data are input into a plurality of big language judgment models, the first reply data are judged through the big language judgment models, a plurality of first judgment results corresponding to the first reply data are obtained, and the first judgment results are used for representing whether the first reply data are correct or not; and constructing training data based on the plurality of prompt word texts, the first reply data corresponding to any one prompt word text and the plurality of first judgment results. The technical problem of low training data generation efficiency is solved.

Description

Technical Field

[0001] The present application relates to the fields of large model technology and natural language processing technology, and more specifically, to a data generation method, an electronic device and a storage medium. Background Art

[0002] Currently, when tuning a large language model, a large amount of text is usually used to pre-train the large language model so that the large language model has stronger language capabilities. Then, supervised fine-tuning (SFT) is used to inject instruction capabilities into the large language model. Finally, reinforcement learning is used to align the large language model so that the large language model can learn human preferences.

[0003] However, the above method requires a lot of manpower for labeling when constructing training data, which consumes a lot of resources and accumulates data slowly, resulting in the problem of low efficiency in generating training data.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a data generation method, an electronic device, and a storage medium to at least solve the technical problem of low efficiency in generating training data.

[0006] According to one aspect of an embodiment of the present application, a data generation method is provided. The method may include: constructing multiple prompt word texts based on multiple inquiry data, wherein different prompt word texts at least contain different inquiry data; inputting multiple prompt word texts into a first generation model, and using the first generation model to generate first reply data corresponding to multiple prompt word texts; inputting any prompt word text and the corresponding first reply data into multiple large language judgment models, and using multiple large language judgment models to judge the first reply data respectively, and obtaining multiple first judgment results corresponding to the first reply data, wherein the first judgment result is used to characterize whether the first reply data is correct; constructing training data based on multiple prompt word texts, the first reply data corresponding to any prompt word text, and the multiple first judgment results, wherein the training data is used to train the first generation model.

[0007] According to another aspect of the embodiment of the present application, a data generation method is provided. The method may include: in response to an input instruction acting on an operation interface, displaying multiple inquiry data on the operation interface; in response to a processing instruction acting on the operation interface, displaying training data on the operation interface, wherein the training data is constructed based on multiple prompt word texts, first reply data corresponding to the multiple prompt word texts, and multiple first judgment results corresponding to the first reply data, the multiple prompt word texts are constructed based on multiple inquiry data, the first reply data is generated by inputting the multiple prompt word texts into a first generation model and using the first generation model, the multiple first judgment results are obtained by inputting any prompt word text and the first reply data into multiple large language judgment models, and using multiple large language judgment models to judge the first reply data respectively, the first judgment result is used to characterize whether the first reply data is correct, and the training data is used to train the first generation model.

[0008] According to another aspect of the embodiment of the present application, a data generation method is provided. The method may include: obtaining multiple inquiry data by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes multiple inquiry data; constructing multiple prompt word texts based on the multiple inquiry data, wherein different prompt word texts at least contain different inquiry data; inputting the multiple prompt word texts into a first generation model, and using the first generation model to generate first reply data corresponding to the multiple prompt word texts; inputting any prompt word text and the corresponding first reply data into multiple large language judgment models, and using multiple large language judgment models to judge the first reply data respectively, and obtaining multiple first judgment results corresponding to the first reply data, wherein the first judgment result is used to characterize whether the first reply data is correct; constructing training data based on multiple prompt word texts, the first reply data corresponding to any prompt word text, and the multiple first judgment results, wherein the training data is used to train the first generation model; outputting the training data by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the training data.

[0009] According to another aspect of the embodiments of the present application, a computing device is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the method in each embodiment of the present application is executed when the program is running.

[0010] According to another aspect of the embodiments of the present application, an electronic device is also provided, including: a memory storing an executable program; a processor connected to the memory via a bus and used to run the program, wherein the method of each embodiment of the present application is executed when the program is running.

[0011] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in each embodiment of the present application.

[0012] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0013] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0014] According to another aspect of the embodiments of the present application, a computer program is also provided, and when the computer program is executed by a processor, the methods in the various embodiments of the present application are implemented.

[0015] In an embodiment of the present application, based on multiple inquiry data, multiple prompt word texts are constructed, wherein different prompt word texts contain at least different inquiry data; multiple prompt word texts are input into a first generation model, and the first generation model is used to generate first reply data corresponding to multiple prompt word texts; any one prompt word text and the corresponding first reply data are input into multiple large language judgment models, and the first reply data are judged by multiple large language judgment models respectively, and multiple first judgment results corresponding to the first reply data are obtained, wherein the first judgment result is used to characterize whether the first reply data is correct; based on multiple prompt word texts, the first reply data corresponding to any one prompt word text, and multiple first judgment results, training data is constructed, wherein the training data is used to train the first generation model. That is to say, after the embodiment of the present application generates the first reply data corresponding to multiple prompt word texts using the first generation model, the first reply data is judged and cleaned by multiple large language judgment models, and then the training data is constructed based on the multiple first judgment results. The constructed training data does not require the participation of manual annotation, and the data accumulation is fast, thereby achieving the technical effect of improving the efficiency of generating training data and solving the technical problem of low efficiency in generating training data.

[0016] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 is a schematic diagram of an application scenario of a data generation method according to an embodiment of the present application;

[0019] Figure 2 is a flow chart of a data generation method according to an embodiment of the present application;

[0020] Figure 3 is a flow chart of another data generation method according to an embodiment of the present application;

[0021] Figure 4 is a flow chart of another data generation method according to an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of constructing training data according to an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of voting of a large language referee model according to an embodiment of the present application;

[0024] Figure 7 is a schematic diagram of a data generating device according to an embodiment of the present application;

[0025] Figure 8 is a schematic diagram of another data generating device according to an embodiment of the present application;

[0026] Fig. 9 is a schematic diagram of a data generating device according to an embodiment of the present application;

[0027] Fig.10 is a structural block diagram of a computing device according to an embodiment of the present application;

[0028] Fig.11 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] The technical solution provided in this application is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a foundation model / foundation model. The large model is pre-trained through large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (Large Language Model, LLM), multi-modal pre-training model (multi-modal pre-training model), etc.

[0032] It should be noted that, when the large model is actually used, the pre-trained model can be fine-tuned by a small number of samples, so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (Natural Language Processing, referred to as NLP), computer vision, speech processing and other fields, and can be specifically applied to computer vision tasks such as visual question answering (Visual Question Answering, referred to as VQA), image description (Image Capt ion, referred to as IC), image generation, etc., and can also be widely used in text-based sentiment classification, text summary generation, machine translation and other natural language processing tasks. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present application, the data processing by generating models and large language referee models in the e-commerce scenario is explained as an example.

[0033] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0034] Retrieval Augmented Generation (RAG), a model architecture that combines information retrieval and text generation techniques, can be used to generate more accurate and detailed text output. When processing an input request (e.g., asking for data), RAG first retrieves supplementary information related to the input from an external database or document, and then inputs the supplementary information into the text generation model together with the original input to generate output based on the retrieved information;

[0035] Reinforcement Learning from Human Feedback (RLHF), a reinforcement learning training method that uses user feedback (positive or negative) to guide and optimize the behavior of large language models;

[0036] Direct Preference Optimization (DPO), a method for training large language models that directly optimizes the large language model's predictions of user preferences or behaviors;

[0037] Supervised Fine-Tuning (SFT) is a method of training on the basis of a pre-trained model. By using a labeled supervised dataset, a large language model can be made better at performing specific tasks.

[0038] Prompt text (prompt), input the prompt word of the large language model;

[0039] Response data (response), the output of the large language model.

[0040] According to an embodiment of the present application, a data generation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] Considering the huge number of model parameters of the large model and the limited computing resources of the mobile terminal, the above method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 is a schematic diagram of an application scenario of a data generation method according to an embodiment of the present application. Figure 1In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. An operation interface for obtaining multiple query data can be deployed on the graphical user interface of the client device. The client device 20 can interact with the user through the graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0042] In an embodiment of the present application, the system composed of the client device and the server can perform the following steps: a corresponding operation can be performed in the operation interface on the client device 20 to obtain multiple inquiry data. The client device can obtain the multiple inquiry data and send it to the server through the network. After receiving the multiple inquiry data, the server can perform the following steps: step S102, based on the multiple inquiry data, construct multiple prompt word texts, wherein different prompt word texts contain at least different inquiry data; step S104, input the multiple prompt word texts into the first generation model, and use the first generation model to generate the first reply data corresponding to the multiple prompt word texts; step S106, input any prompt word text and the corresponding first reply data into multiple large language judgment models, and use multiple large language judgment models to judge the first reply data respectively, and obtain multiple first judgment results corresponding to the first reply data, wherein the first judgment result is used to characterize whether the first reply data is correct; step S108, based on the multiple prompt word texts, the first reply data corresponding to any prompt word text and the multiple first judgment results, construct training data, wherein the training data is used to train the first generation model.

[0043] It should be noted that, with the rapid development of high-performance computing units, in other application scenarios, the above method provided in the embodiment of the present application can also be applied to the model all-in-one machine. In an optional embodiment, a plurality of models are built into the model all-in-one machine, and the user can choose to adjust a model as needed to obtain the user's own model, so that the high-performance computing unit built into the model all-in-one machine can directly call the adjusted model to execute the above method provided in the embodiment of the present application. In another optional embodiment, a trained model is built into the large model all-in-one machine, so that the high-performance computing unit built into the model all-in-one machine can directly call the model to execute the above method provided in the embodiment of the present application.

[0044] Furthermore, when users need to train their own models, they can also upload their own data sets through the client, which are sent by the client to the server, so that the server can adjust the pre-trained model with the data set to obtain the user's own model and then deploy it to the production environment. In order to facilitate users' needs for model adjustment, the server can provide complete adjustment tools, development frameworks and processes, and can support multiple adjustment strategies, so that the adjusted model can better adapt to applications in different fields and achieve a high degree of customization.

[0045] Under the above operating environment, this application provides Figure 2 The data generation method shown. Figure 2 is a flow chart of a data generation method according to an embodiment of the present application, such as Figure 2 As shown, the method may include the following steps:

[0046] Step S202: construct multiple prompt word texts based on multiple query data.

[0047] In the technical solution provided in the above step S202 of the present application, multiple prompt word texts can be constructed based on multiple inquiry data. Among them, the inquiry data can be the inquiry data input by the user in natural language. For example, in the e-commerce scenario, the inquiry data can be the information about the product asked by the user (for example, a consumer) to the customer service (for example, an intelligent customer service). The inquiry data can be used to inquire about information related to the product. For example, the inquiry information can be used to inquire about the size, material, quality, attributes and performance of the product. It should be noted that the above inquiry data for the e-commerce scenario is only for example, and does not specifically limit the application scenario and the type of inquiry data in the application scenario.

[0048] In this embodiment, the inquiry data is usually associated with a specific product or store. The prompt word text prompt can be an input text constructed according to the inquiry data and the relevant context of the inquiry data (e.g., product information, store information), which can be used to guide the large model to generate a specific type of answer. Different prompt word texts contain at least different inquiry data.

[0049] Optionally, multiple inquiry data of the user's questions are obtained based on multiple conversations between the user and the customer service, and the product information and store information related to the corresponding inquiry data can be obtained. The obtained multiple product information and store information are further dynamically filled into the pre-built prompt word text template to obtain multiple prompt word texts.

[0050] For example, a pre-built prompt word text template may be "A user asked a question about [product name] in [store name], and the information about [product name] is [attribute / function], and the user's question is [inquiry data], please answer". The user's inquiry data obtained may be "What is the material of product Y?", and the product information and store information related to the inquiry data may be obtained. The product information may include at least the product name, product material, etc., for example, the product name is product Y, and the product material is metal, and the store information may include at least the store name, etc., for example, the store name is store name X. The obtained product information and store information are further dynamically filled into the pre-built prompt word text template, and the resulting prompt word text is "A user asked a question about product Y in store X, and the information about product Y is metal. The user's question is what is the material of product Y, please answer".

[0051] Step S204: input the plurality of prompt word texts into the first generation model, and use the first generation model to generate first reply data corresponding to the plurality of prompt word texts.

[0052] In the technical solution provided in the above step S204 of the present application, after constructing multiple prompt word texts based on multiple query data, the obtained multiple prompt word texts can be input into the first generation model, and then the first generation model is used to generate the first response data corresponding to the multiple prompt word texts. Among them, the first generation model can be a generation model with command following and role-playing capabilities. The first generation model is constrained by time delay and deployment cost, etc., and the parameter amount is usually 7-30B, B is billion (Billion), and the first generation model can be represented by generation model A. The first response data response can be the corresponding answer data generated by the first generation model according to the prompt word text.

[0053] In this embodiment, after a large number of prompts are constructed based on the user's query data, and the product information and store information corresponding to the query data, the constructed prompts can be input into the generation model A, so that the first response generated by the generation model A according to the prompts can be obtained.

[0054] Step S206, input any prompt word text and the corresponding first reply data into multiple large language judgment models, use multiple large language judgment models to judge the first reply data respectively, and obtain multiple first judgment results corresponding to the first reply data, wherein the first judgment result is used to indicate whether the first reply data is correct.

[0055] In the technical solution provided in the above step S206 of the present application, after a plurality of prompt word texts are input into the first generation model and the first generation model is used to generate the first reply data corresponding to the plurality of prompt word texts, any prompt word text and the corresponding first reply data can be input into a plurality of large language adjudication models, and then the first reply data can be adjudicated by the plurality of large language adjudication models respectively to obtain a plurality of first adjudication results corresponding to the first reply data. Among them, the large language adjudication model can be a large language model with strong instruction following ability and logical reasoning ability, the parameter amount of the large language adjudication model can be more than 70B, and the large language adjudication model can be referred to as the adjudication model for short. The first adjudication result can be used to characterize whether the first reply data is correct.

[0056] For example, the number of large language judgment models is 3. Any prompt and the corresponding first response can be input into three large language judgment models, and the three large language judgment models respectively judge whether the first response is correct, thereby obtaining three first judgment results. For example, the three first judgment results all indicate that the first response is correct. The three first judgment results can also be two first judgment results indicating that the first response is wrong, and the other first judgment result indicating that the first response is correct. This is only for illustration, and no specific restrictions are imposed on the number of large language judgment models and multiple first judgment results.

[0057] Step S208, constructing training data based on multiple prompt word texts, first reply data corresponding to any prompt word text, and multiple first judgment results.

[0058] In the technical solution provided in the above step S208 of the present application, after using multiple large language adjudication models to adjudicate the first reply data respectively and obtaining multiple first adjudication results corresponding to the first reply data, training data can be constructed based on multiple prompt word texts, the first reply data corresponding to any prompt word text, and the multiple first adjudication results. Among them, the training data can be used to train the first generation model, and the training data may include the training data of the SFT stage and the training data of the alignment stage. For example, the training data of the alignment stage includes but is not limited to the training data of DPO alignment. It should be noted that different training data can be used in the SFT stage and the alignment stage, and the training data of the alignment stage may include correct data (chosen response) and incorrect data (rejected response).

[0059] In this embodiment, training data can be constructed based on the number of first judgment results that characterize the first response errors among multiple first judgment results. Optionally, when the number of first judgment results that characterize the first response errors among multiple first judgment results is less than or equal to a preset number, multiple prompts and corresponding first responses can be used as training data for the SFT stage. When the number of first judgment results that characterize the first response errors among multiple first judgment results is greater than a preset number, the parameters of the first generation model can be adjusted, and the response can be regenerated using the adjusted first generation model, and then the regenerated response can be judged by multiple large language judgment models. If the regenerated judgment results all characterize that the regenerated response is correct, the multiple prompts, the first response, and the regenerated response can be used as training data for the alignment stage.

[0060] For example, the number of large language judge models is 3. When two of the three first judge results indicate that the response is correct, or when all three first judge results indicate that the first response is correct, multiple prompts and corresponding first responses can be used as training data for the SFT stage.

[0061] For another example, when two of the three first judgment results indicate that the response is wrong, or when all three first judgment results indicate that the first response is wrong, the response and the three judgment results can be regenerated in the above manner. When the three regenerated judgment results all indicate that the regenerated response is correct, the multiple prompts, the first responses, and the regenerated responses are used as training data for the alignment stage, otherwise the corresponding data is discarded, that is, this embodiment only takes the responses that are correctly judged by multiple large language judgment models as the correct data for the alignment stage.

[0062] Through the above steps S202 to S208 of the present application, based on multiple inquiry data, multiple prompt word texts are constructed, wherein different prompt word texts contain at least different inquiry data; multiple prompt word texts are input into the first generation model, and the first generation model is used to generate the first reply data corresponding to multiple prompt word texts; any one prompt word text and the corresponding first reply data are input into multiple large language judgment models, and the first reply data are judged by multiple large language judgment models respectively, and multiple first judgment results corresponding to the first reply data are obtained, wherein the first judgment result is used to characterize whether the first reply data is correct; based on multiple prompt word texts, the first reply data corresponding to any one prompt word text and multiple first judgment results, training data is constructed, wherein the training data is used to train the first generation model. That is to say, after the embodiment of the present application generates the first reply data corresponding to multiple prompt word texts using the first generation model, the first reply data is judged and cleaned by multiple large language judgment models, and then the training data is constructed based on the multiple first judgment results. The constructed training data does not require the participation of manual annotation, and the data accumulation is fast, thereby achieving the technical effect of improving the efficiency of generating training data and solving the technical problem of low efficiency in generating training data.

[0063] The above method of this embodiment is further introduced below.

[0064] As an optional implementation, the training data includes first-stage data for supervised adjustment of the first generation model, and second-stage data for preference alignment of the first generation model; step S208, constructing training data based on multiple prompt word texts, first reply data corresponding to any prompt word text, and multiple first judgment results, including: for any prompt word text, determining the type of the prompt word text based on multiple first judgment results; constructing the first-stage data based on the first prompt word text of the first type among the multiple prompt word texts, and the first reply data corresponding to the first prompt word text; inputting the second prompt word text of the second type among the multiple prompt word texts into the second generation model, generating second reply data corresponding to the second prompt word text using the second generation model, and constructing the second-stage data based on the second prompt word text, the first reply data corresponding to the second prompt word text, and the second reply data corresponding to the second prompt word text.

[0065] In this embodiment, the first stage data may also be referred to as the training data of the SFT stage, hereinafter referred to as SFT data. The second stage data may also be referred to as the training data of the alignment stage, hereinafter referred to as alignment data. The first type may be the type of prompt word text used in multiple first adjudication results to characterize that the number of errors in the first reply data is less than or equal to a preset number. The second type may be the type of prompt word text used in multiple first adjudication results to characterize that the number of errors in the first reply data is greater than a preset number.

[0066] It should be noted that the second generation model can be an offline model of the first generation model. The second generation model is less constrained by time delay and deployment cost, and the number of parameters can reach more than 70B. The second generation model can be represented by generation model B. The performance indicators of the second generation model are greater than those of the first generation model, and the performance indicators include but are not limited to indicators such as parameter quantity, instruction following ability, and logical reasoning ability. The second reply data can be the corresponding answer data generated by the second generation model according to the second prompt word text.

[0067] Optionally, when the number of errors in the first response in the plurality of first adjudication results is less than or equal to a preset number, the type of prompt can be determined to be type 1. Based on the first prompt of the first type and the first response corresponding to the first prompt, SFT data can be constructed, for example, the first prompt and the first response can be used as SFT data.

[0068] Optionally, when the number of errors in the first judgment results used to characterize the first response is greater than a preset number, the type of prompt can be determined to be the second type. Alignment data can be constructed based on the second prompt of the second type and the first response corresponding to the second prompt. For example, the second prompt of the second type can be input into the generation model B to obtain the second response generated by the generation model B according to the second prompt, and then the second prompt, the first response corresponding to the second prompt, and the second response corresponding to the second prompt are used as alignment data.

[0069] As an optional implementation, for any prompt word text, the type of the prompt word text is determined based on multiple first judgment results, including: determining the number of target judgment results among the multiple first judgment results, wherein the target judgment results are used to characterize the first reply data error; when the number is less than or equal to a preset number, determining the type of the prompt word text to be the first type; when the number is greater than the preset number, setting the type of the prompt word text to the second type.

[0070] In this embodiment, the target referee result can be used to characterize the first reply data error. The preset number can be the number of target referee results set according to actual conditions, for example, the preset number can be 1, which is only an example and does not specifically limit the value of the preset number.

[0071] For example, the number of large language judgment models is 3 and the preset number is 1. After any prompt and the corresponding first response are input into three large language judgment models and three first judgment results are obtained, if the number of target judgment results in the first judgment results is 0, since the number of target judgment results is less than the preset number, the type of prompt can be determined to be the first type; if the number of target judgment results in the first judgment results is 1, since the number of target judgment results is equal to the preset number, the type of prompt can be determined to be the first type.

[0072] For another example, if the number of target judgment results in the first judgment result is 2, since the number of target judgment results is greater than the preset number, the type of prompt can be determined to be the second type; if the number of target judgment results in the first judgment result is 3, since the number of target judgment results is greater than the preset number, the type of prompt can be determined to be the second type.

[0073] As an optional implementation, second-stage data is constructed based on the second prompt word text, the first reply data corresponding to the second prompt word text, and the second reply data corresponding to the second prompt word text, including: inputting the second prompt word text and the corresponding second reply data into multiple large language judgment models, using the multiple large language judgment models to judge the second reply data respectively, and obtaining multiple second judgment results corresponding to the second reply data, wherein the second judgment results are used to characterize whether the second reply data is correct; when multiple second judgment results all characterize that the second reply data is correct, determining that the second prompt word text, the corresponding first reply data, and the corresponding second reply data are the second-stage data.

[0074] In this embodiment, the second judgment result can be used to characterize whether the second reply data is correct. If multiple second judgment results all characterize that the second reply data is correct, the second prompt word text, the corresponding first reply data, and the corresponding second reply data can be used as the second stage data. If there is still a second judgment result among multiple second judgment results that characterizes that the second reply data is wrong, the second prompt word text, the corresponding first reply data, and the corresponding second reply data can be discarded, that is, this part of the data is not used as the second stage data.

[0075] Optionally, when multiple second judgment results all indicate that the second response is correct, the second response can be used as correct response data (chosen response), and the first response corresponding to the second prompt can be used as incorrect response data (rejected response), and then the second prompt, chosen response, and rejected response can be used together as alignment data.

[0076] For example, the number of large language judgment models is 3. After the second prompt of the second type is input into the generation model B and the second response generated by the generation model B according to the second prompt is obtained, the second prompt and the second response can be input into three large language judgment models, and the three large language judgment models respectively judge whether the second response is correct, thereby obtaining three second judgment results. When the three first judgment results all indicate that the second response is correct, the second prompt, the first response corresponding to the second prompt, and the second response corresponding to the second prompt can be used as alignment data.

[0077] As an optional implementation, step S202, based on multiple query data, constructs multiple prompt word texts, including: determining the item data of the target item corresponding to any query data, and the object data of the target object to which the target item belongs; based on multiple query data, item data and object data, generates multiple prompt word texts.

[0078] Optionally, in an e-commerce scenario, the target item may be a commodity on an e-commerce platform, the item data may be used to represent the commodity information, the target object may be the store to which the commodity belongs, and the object data may be used to represent the store information.

[0079] Optionally, after determining the item data of the target item corresponding to any query data, and the object data of the target object to which the target item belongs, the determined multiple query data, item data and object data can be filled into a pre-built prompt word text template to obtain multiple prompt word texts. For example, the obtained prompt word text can be "The user asked a question about product Y in store X. The information of product Y is metal. The user's question is what material is product Y made of. Please answer."

[0080] As an optional implementation, the method also includes: using the first stage data included in the training data to perform supervised adjustment on the first generation model to obtain an adjusted generation model; using the second stage data included in the training data to perform preference alignment on the adjusted generation model to obtain a target generation model.

[0081] In this embodiment, the target generative model can be a generative model after supervised adjustment and preference alignment of the first generative model. The SFT data obtained above can be used to perform supervised adjustment (i.e., supervised fine-tuning) on ​​the first generative model to obtain an adjusted generative model, and further alignment can be used to perform preference alignment on the adjusted generative model to obtain the target generative model.

[0082] In related technologies, when tuning a large language model, the processing flow usually adopted can be pre-training, supervised fine-tuning SFT, and preference alignment. That is, a large amount of text is used to pre-train the large language model so that the large language model has stronger language capabilities, and then the instruction capabilities are injected into the large language model through supervised fine-tuning. Finally, the large language model is aligned and tuned through reinforcement learning so that the large language model can learn human preferences.

[0083] It should be noted that the large language model that has been aligned and optimized can maintain a strong ability to follow instructions, and is superior to the large language model that has not been aligned and optimized in terms of accuracy and risk rate of the output answer. Therefore, in the related art, by optimizing the alignment stage, the large language model does not need to be aligned through reinforcement learning, and the large language model can be directly optimized by gradient descent, omitting the training of the reward model and reducing the need for training data. However, the above method requires a lot of manpower to label the training data in the SFT and alignment stages, which consumes a lot of resources and accumulates data slowly, resulting in a technical problem of low efficiency in generating training data.

[0084] However, in an embodiment of the present application, in order to address the problem of low efficiency in generating training data in the above-mentioned related technologies, after using a first generation model to generate first reply data corresponding to multiple prompt word texts, this embodiment uses multiple large language judgment models to perform judgment cleaning on the first reply data, and then constructs training data based on multiple first judgment results. Since the constructed training data includes SFT and alignment stages, no manual annotation is required, and data accumulation is fast, thereby achieving the technical effect of improving the efficiency of generating training data and solving the technical problem of low efficiency in generating training data.

[0085] As an optional implementation, the second-stage data included in the training data is used to perform preference alignment on the adjusted generation model to obtain a target generation model, including one of the following: constructing a reward model based on the second-stage data, and using the reward model to perform preference alignment on the adjusted generation model to obtain the target generation model; using the second-stage data and a direct preference optimization algorithm to perform preference alignment on the adjusted generation model to obtain the target generation model.

[0086] In this embodiment, the reward model may be a model for evaluating the reply data generated by the large language model. The reward model may assign a score or reward to each reply data generated by the large language model by learning human preferences for text. The score or reward may be in numerical form and may be used to indicate the quality, relevance or satisfaction of the reply data.

[0087] Optionally, after determining the second prompt, the chosen response, and the rejected response, a reward model can be trained based on the second prompt, the chosen response, and the rejected response, and the reward model can be used to perform preference alignment on the adjusted generation model to obtain a target generation model. For example, the generation model can be aligned using a human feedback reinforcement learning RLHF algorithm, and the reward model gives a reward to the response data output by the adjusted generation model. The adjusted generation model updates parameters based on the feedback of the reward model through a reinforcement learning mechanism, and the goal is to maximize the reward, that is, to learn to generate more high-quality response data like the chosen response.

[0088] Optionally, after determining the second prompt, the chosen response, and the rejected response, the adjusted generative model can be aligned and trained using the second prompt, the chosen response, the rejected response, and the direct preference optimization algorithm DPO to obtain a target generative model. That is, based on the multiple adjudication results output by the multiple large language adjudication models, the parameters of the generative model are adjusted until the generative model has a significantly higher preference for the chosen response than the rejected response when generating a response.

[0089] In an embodiment of the present application, based on multiple inquiry data, multiple prompt word texts are constructed, wherein different prompt word texts contain at least different inquiry data; multiple prompt word texts are input into a first generation model, and the first generation model is used to generate first reply data corresponding to multiple prompt word texts; any one prompt word text and the corresponding first reply data are input into multiple large language judgment models, and the first reply data are judged by multiple large language judgment models respectively, and multiple first judgment results corresponding to the first reply data are obtained, wherein the first judgment result is used to characterize whether the first reply data is correct; based on multiple prompt word texts, the first reply data corresponding to any one prompt word text, and multiple first judgment results, training data is constructed, wherein the training data is used to train the first generation model. That is to say, after the embodiment of the present application generates the first reply data corresponding to multiple prompt word texts using the first generation model, the first reply data is judged and cleaned by multiple large language judgment models, and then the training data is constructed based on the multiple first judgment results. The constructed training data does not require the participation of manual annotation, and the data accumulation is fast, thereby achieving the technical effect of improving the efficiency of generating training data and solving the technical problem of low efficiency in generating training data.

[0090] The present application also provides a data generation method. Figure 3 is a flow chart of another data generation method according to an embodiment of the present application. Figure 3 As shown, the method may include the following steps:

[0091] Step S302, in response to an input instruction on the operation interface, a plurality of inquiry data are displayed on the operation interface.

[0092] In the technical solution provided in the above step S302 of the present application, in response to the input command acting on the operation interface, multiple query data can be displayed on the operation interface. The operation interface can be an operation interface displayed on the above client device, for example, it can be an interface for interaction between a user and a computer or software system, it can be a graphical user interface, a command line interface, etc. This is only an example, and the form of the operation interface is not specifically limited.

[0093] In this embodiment, the input instruction can be an instruction for controlling the operation interface to display multiple query data. For example, it can be an instruction entered by a user on the operation interface. The user can generate the input instruction by entering text through the keyboard, clicking buttons or options on the operation interface with the mouse, or dragging multiple query data to a designated area.

[0094] In an optional embodiment, the user can open the operation interface deployed on the client device and input multiple query data on the operation interface to generate input instructions. According to the generated input instructions, the corresponding multiple query data can be loaded and displayed on the operation interface.

[0095] Step S304, in response to the processing instruction acting on the operation interface, the training data is displayed on the operation interface, wherein the training data is constructed based on multiple prompt word texts, first reply data corresponding to the multiple prompt word texts, and multiple first judgment results corresponding to the first reply data, the multiple prompt word texts are constructed based on multiple inquiry data, the first reply data is input into the first generation model, and the first generation model is used to generate the multiple prompt word texts, the multiple first judgment results are input into any prompt word text and the first reply data into multiple large language judgment models, and the first reply data are judged by the multiple large language judgment models respectively, the first judgment result is used to characterize whether the first reply data is correct, and the training data is used to train the first generation model.

[0096] In the technical solution provided in the above step S304 of the present application, the processing instruction may be an instruction for processing multiple query data displayed on the operation interface. The user may generate the processing instruction by clicking on the multiple query data displayed on the operation interface. This is only an example and does not impose specific restrictions on the processing instruction.

[0097] In this embodiment, multiple inquiry data of the user's questions are obtained based on multiple conversations between the user and the customer service, and the product information and store information related to any inquiry data can be obtained. The obtained multiple product information and store information are further dynamically filled into the pre-built prompt word text template to obtain multiple prompt word texts.

[0098] Optionally, after constructing a large number of prompts based on the user's query data, and the product information and store information corresponding to the query data, the constructed prompts can be input into the generation model A, so that the first response generated by the generation model A according to the prompts can be obtained.

[0099] For example, the number of large language judgment models is 3. Any prompt and the corresponding first response can be input into three large language judgment models, and the three large language judgment models respectively judge whether the first response is correct, thereby obtaining three first judgment results. For example, the three first judgment results all indicate that the first response is correct. The three first judgment results can also be two first judgment results indicating that the first response is wrong, and the other first judgment result indicating that the first response is correct. This is only for illustration, and no specific restrictions are imposed on the number of large language judgment models and multiple first judgment results.

[0100] Optionally, when multiple first judgment results all indicate that the first response is correct, multiple prompts and corresponding first responses can be used as training data for the SFT stage. When there is a first judgment result among multiple first judgment results that indicates that the first response is wrong, the parameters of the first generation model can be adjusted, and a new response can be generated using the adjusted first generation model, and then the new response can be judged by multiple large language judgment models to obtain a new judgment result, so that multiple prompts, first responses and new responses can be used as training data for the alignment stage.

[0101] Through the above steps S302 to S304 of the present application, in response to the input instruction acting on the operation interface, multiple inquiry data are displayed on the operation interface; in response to the processing instruction acting on the operation interface, training data is displayed on the operation interface, wherein the training data is constructed based on multiple prompt word texts, first reply data corresponding to the multiple prompt word texts, and multiple first judgment results corresponding to the first reply data, the multiple prompt word texts are constructed based on multiple inquiry data, the first reply data is input into the first generation model, and the first generation model is used to generate the multiple first judgment results, any prompt word text and the first reply data are input into multiple large language judgment models, and the first reply data are judged by the multiple large language judgment models, the first judgment result is used to characterize whether the first reply data is correct, and the training data is used to train the first generation model. That is to say, after using the first generation model to generate first reply data corresponding to multiple prompt word texts, the embodiment of the present application uses multiple large language judgment models to perform judgment cleaning on the first reply data, and then constructs training data based on multiple first judgment results. The constructed training data does not require the participation of manual annotation, and the data accumulates quickly, thereby achieving the technical effect of improving the efficiency of generating training data and solving the technical problem of low efficiency in generating training data.

[0102] The present application also provides a data generation method. Figure 4 is a flow chart of another data generation method according to an embodiment of the present application, such as Figure 4 As shown, the method may include the following steps:

[0103] Step S402: Acquire multiple query data by calling the first interface.

[0104] In the technical solution provided in the above step S402 of the present application, multiple query data can be obtained by calling the first interface, wherein the first interface can include a first parameter, and the parameter value of the first parameter can include multiple query data.

[0105] Step S404: construct multiple prompt word texts based on the multiple query data.

[0106] In the technical solution provided in the above step S404 of the present application, after acquiring multiple query data by calling the first interface, multiple prompt word texts can be constructed based on the acquired multiple query data, wherein different prompt word texts prompt at least contain different query data.

[0107] In this embodiment, multiple inquiry data of the user's questions are obtained based on multiple conversations between the user and the customer service, and the product information and store information related to any inquiry data can be obtained. The obtained multiple product information and store information are further dynamically filled into the pre-built prompt word text template to obtain multiple prompt word texts.

[0108] For example, a pre-built prompt word text template may be "A user asked a question about [product name] in [store name], and the information about [product name] is [attribute / function], and the user's question is [inquiry data], please answer". The user's inquiry data obtained may be "What is the material of product Y?", and the product information and store information related to the inquiry data may be obtained. The product information may include at least the product name, product material, etc., for example, the product name is product Y, and the product material is metal, and the store information may include at least the store name, etc., for example, the store name is store name X. The obtained product information and store information are further dynamically filled into the pre-built prompt word text template, and the resulting prompt word text is "A user asked a question about product Y in store X, and the information about product Y is metal. The user's question is what is the material of product Y, please answer".

[0109] Step S406: input the plurality of prompt word texts into the first generation model, and use the first generation model to generate first reply data corresponding to the plurality of prompt word texts.

[0110] In the technical solution provided in the above step S406 of the present application, after constructing multiple prompt word texts based on multiple inquiry data, the multiple prompt word texts can be input into the first generation model, and then the first generation model is used to generate first reply data corresponding to the multiple prompt word texts.

[0111] In this embodiment, after a large number of prompts are constructed based on the user's query data, and the product information and store information corresponding to the query data, the constructed prompts can be input into the generation model A, so that the first response generated by the generation model A according to the prompts can be obtained.

[0112] Step S408, input any prompt word text and the corresponding first reply data into multiple large language judgment models, use the multiple large language judgment models to judge the first reply data respectively, and obtain multiple first judgment results corresponding to the first reply data.

[0113] In the technical solution provided in the above step S408 of the present application, after a plurality of prompt word texts are input into the first generation model and the first response data corresponding to the plurality of prompt word texts are generated by the first generation model, any prompt word text and the corresponding first response data can be input into a plurality of large language adjudication models, and the plurality of large language adjudication models are used to adjudicate the first response data respectively, to obtain a plurality of first adjudication results corresponding to the first response data. The first adjudication result can be used to characterize whether the first response data is correct.

[0114] For example, the number of large language judgment models is 3. Any prompt and the corresponding first response can be input into three large language judgment models, and the three large language judgment models respectively judge whether the first response is correct, thereby obtaining three first judgment results. For example, the three first judgment results all indicate that the first response is correct. The three first judgment results can also be two first judgment results indicating that the first response is wrong, and the other first judgment result indicating that the first response is correct. This is only for illustration, and no specific restrictions are imposed on the number of large language judgment models and multiple first judgment results.

[0115] Step S410, constructing training data based on multiple prompt word texts, first reply data corresponding to any prompt word text, and multiple first judgment results. In the technical solution provided in the above step S410 of the present application, after using multiple large language judgment models to judge the first reply data respectively and obtaining multiple first judgment results corresponding to the first reply data, training data can be constructed based on multiple prompt word texts, first reply data corresponding to any prompt word text, and multiple first judgment results. Among them, the training data can be used to train the first generation model.

[0116] In this embodiment, when multiple first judgment results all indicate that the first response is correct, multiple prompts and corresponding first responses can be used as training data for the SFT stage. When there is a first judgment result among multiple first judgment results that indicates that the first response is wrong, the parameters of the first generation model can be adjusted, and a new response can be generated using the adjusted first generation model, and then the new response can be judged by multiple large language judgment models to obtain a new judgment result, so that multiple prompts, first responses and new responses can be used as training data for the alignment stage.

[0117] Step S412: output the training data by calling the second interface.

[0118] In the technical solution provided in the above step S412 of the present application, after constructing the training data based on the multiple prompt word texts, the first reply data corresponding to any prompt word text, and the multiple first judgment results, the training data can be output by calling the second interface. The second interface can include a second parameter, and the parameter value of the second parameter can include the training data.

[0119] Through the above steps S402 to S412 of the present application, a plurality of inquiry data are obtained by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes a plurality of inquiry data; based on the plurality of inquiry data, a plurality of prompt word texts are constructed, wherein different prompt word texts at least contain different inquiry data; the plurality of prompt word texts are input into the first generation model, and the first generation model is used to generate first reply data corresponding to the plurality of prompt word texts; any one of the prompt word texts and the corresponding first reply data are input into a plurality of large language adjudication models, and the plurality of large language adjudication models are used to adjudicate the first reply data respectively, to obtain a plurality of first adjudication results corresponding to the first reply data, wherein the first adjudication result is used to characterize whether the first reply data is correct; based on the plurality of prompt word texts, the first reply data corresponding to any one of the prompt word texts, and the plurality of first adjudication results, training data are constructed, wherein the training data is used to train the first generation model; the training data is output by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the training data. That is to say, after using the first generation model to generate first reply data corresponding to multiple prompt word texts, the embodiment of the present application uses multiple large language judgment models to perform judgment cleaning on the first reply data, and then constructs training data based on multiple first judgment results. The constructed training data does not require the participation of manual annotation, and the data accumulates quickly, thereby achieving the technical effect of improving the efficiency of generating training data and solving the technical problem of low efficiency in generating training data.

[0120] The technical solution of the embodiment of the present disclosure is further introduced below with examples in combination with preferred implementation modes.

[0121] At present, traditional intelligent customer service mainly relies on merchants to configure knowledge according to preset questions during conversations. However, in this method, merchants need to invest a lot of time and manpower costs, and in actual applications, there are problems such as limited coverage of user (e.g., consumer) needs and irrelevant answers. Based on the current development of artificial intelligence (AI), it is necessary to use the capabilities of AI large models to answer consumers' questions in real time, while reducing merchants' operating costs and improving user satisfaction and purchase conversion rates. In this context, by building a RAG framework, reference knowledge in the database is recalled in real time according to consumers' questions, and LLM is used to generate answers for consumers. Since the answers generated by LLM will be directly displayed to consumers, it is more important to accurately generate answers based on the recalled information. A large amount of high-quality data is required to train high-quality models in order to efficiently obtain high-quality customer service data.

[0122] In the related art, when tuning a large language model, the commonly used processing flow can be pre-training, supervised fine-tuning SFT, and preference alignment, that is, using a large amount of text to pre-train the large language model so that the large language model has stronger language capabilities, and then injecting the instruction capabilities into the large language model through supervised fine-tuning, and finally aligning and tuning the large language model through reinforcement learning so that the large language model can learn human preferences. Since the large language model that has been aligned and tuned can maintain a strong instruction following ability, and is superior to the large language model that has not been aligned and tuned in terms of the accuracy and risk rate of the output answer. Therefore, by optimizing the alignment stage, the large language model does not need to be aligned through reinforcement learning, and the large language model can be directly optimized by gradient descent, omitting the training of the reward model reward model and reducing the need for training data.

[0123] In addition, when using a large language model to synthesize data, the large language model can be used to synthesize data to perform SFT on the large language model to improve the effect of the large language model. The synthesized data needs to generate prompts and corresponding responses. During the iteration of the large language model, LLM is used as a referee to select the better of the two responses generated under the same prompt as the chosen response and the other as the rejected response. The data set is constructed and the large language model is trained with DPO to iterate the large language model.

[0124] However, the above method for tuning a large language model requires a large amount of manpower to annotate the training data in the SFT and alignment stages, which consumes a lot of resources and accumulates data slowly, thus resulting in a technical problem of low efficiency in generating training data. In addition, when synthesizing data using LLM, even if LLM is used as a referee to screen the data, there will still be certain errors, which will affect the quality of data generation. Moreover, when synthesizing data, SFT and alignment data are generated separately, resulting in a long process and the inability to reuse data, which affects the efficiency of generating training data, thus resulting in a problem of low efficiency in generating training data.

[0125] In order to solve the above problems, this application proposes a method for automatically constructing a large language model SFT and aligning training data by voting. While reducing the cost of training data construction, the quality of data generation can be guaranteed. The training data can be used to optimize the large language model of intelligent customer service, significantly improve the accuracy of the large language model's answers, reduce customer complaints and increase conversions, and this method can also be expanded to generate data for large models in other fields besides e-commerce.

[0126] The following is an example of three major language adjudication models: Figure 5 is a schematic diagram of constructing training data according to an embodiment of the present application, such as Figure 5 As shown, based on the questions asked by consumers using the large language model, as well as the product information and store information corresponding to the questions, a large number of prompts (i.e., the total set of prompts) can be constructed. The constructed prompt can be "The user asked a question about product Y in store X. The information about product Y is metal. The user's question is what material product Y is made of. Please answer." The above prompt is input into the generative model A. The number of parameters of the generative model A is usually 7-30B, and the generative model A has the ability to follow instructions and play roles, and can generate responses according to the prompt.

[0127] The prompt and response are input into three large language referee models (i.e., referee models). The parameter volume of the three large language referee models can reach more than 70B, and they have strong instruction following ability and logical reasoning ability. The three large language referee models judge the correctness of the response respectively, and then the three large language referee models vote to decide the next operation.

[0128] Figure 6 is a schematic diagram of a large language referee model voting according to an embodiment of the present application, such as Figure 6As shown, the three large language referee models are the first referee model, the second referee model and the third referee model. When the three large language referee models all judge that the response is correct, the corresponding prompt and response are directly stored as SFT data, or as correct data (chosen response) in alignment. When two or more of the three large language referee models judge that the response is wrong, the corresponding response is used as error data, that is, as error data (rejected response) in alignment.

[0129] Furthermore, when two or more of the three large language referee models judge that the response is wrong, the prompt can be input into the generation model B which is stronger than the generation model A to generate a new response. The generated new response is input into the three large language referee models, and voting is performed again. When the three large language referee models all judge correctly, the response is used as the chosen response, and the original wrong response is used as the rejected response, and stored together with the prompt as alignment data (for example, DPO alignment data).

[0130] After obtaining the prompt, chosen response, and rejected response, the reward model can be trained based on the prompt, chosen response, and rejected response, and then the reward model can be used for RLHF alignment, or alignment algorithms such as the direct preference optimization algorithm DPO can be used to align the generated model A to obtain the final model.

[0131] This embodiment uses the capability of a large language model to automatically generate training data, and through voting and cleaning with multiple large language referee models, SFT and alignment data can be obtained at the same time. This embodiment constructs a process from data generation to data cleaning, without the participation of manual annotation, and has a fast data accumulation. It can generate SFT and alignment corpora at the same time, and reuses the erroneous corpora generated in the previous step of SFT corpora in the process of generating aligned corpora, so the data generation efficiency is high. Voting and judgment are performed by three different referee models, which greatly eliminates the discrimination preference of the large language model, and clearly defines the chosen response and rejected response in the SFT data and alignment data, ensuring the accuracy of the referee model screening. In addition, this embodiment is not limited to data generation in the e-commerce field, but can also be extended to other fields for reuse.

[0132] In an embodiment of the present application, based on multiple inquiry data, multiple prompt word texts are constructed, wherein different prompt word texts contain at least different inquiry data; multiple prompt word texts are input into a first generation model, and the first generation model is used to generate first reply data corresponding to multiple prompt word texts; any one prompt word text and the corresponding first reply data are input into multiple large language judgment models, and the first reply data are judged by multiple large language judgment models respectively, and multiple first judgment results corresponding to the first reply data are obtained, wherein the first judgment result is used to characterize whether the first reply data is correct; based on multiple prompt word texts, the first reply data corresponding to any one prompt word text, and multiple first judgment results, training data is constructed, wherein the training data is used to train the first generation model. That is to say, after the embodiment of the present application generates the first reply data corresponding to multiple prompt word texts using the first generation model, the first reply data is judged and cleaned by multiple large language judgment models, and then the training data is constructed based on the multiple first judgment results. The constructed training data does not require the participation of manual annotation, and the data accumulation is fast, thereby achieving the technical effect of improving the efficiency of generating training data and solving the technical problem of low efficiency in generating training data.

[0133] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0134] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0135] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including several instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0136] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 2 A data generating device according to the data generating method shown.

[0137] Figure 7 is a schematic diagram of a data generating device according to an embodiment of the present application, such as Figure 7 As shown, the data generating device 700 may include: a first constructing unit 702 , a first generating unit 704 , a first judging unit 706 and a second constructing unit 708 .

[0138] The first constructing unit 702 is used to construct multiple prompt word texts based on multiple query data, wherein different prompt word texts at least contain different query data.

[0139] The first generating unit 704 is used to input the plurality of prompt word texts into the first generating model, and generate first reply data corresponding to the plurality of prompt word texts by using the first generating model.

[0140] The first judgment unit 706 is used to input any prompt word text and the corresponding first reply data into multiple large language judgment models, and use multiple large language judgment models to judge the first reply data respectively to obtain multiple first judgment results corresponding to the first reply data, wherein the first judgment result is used to indicate whether the first reply data is correct.

[0141] The second construction unit 708 is used to construct training data based on multiple prompt word texts, first reply data corresponding to any prompt word text, and multiple first judgment results, wherein the training data is used to train the first generation model.

[0142] It should be noted that the first construction unit 702, the first generation unit 704, the first adjudication unit 706 and the second construction unit 708 correspond to steps S202 to S208 in the above embodiment, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server 10 provided in the above embodiment as part of the device.

[0143] In the data generation device, a plurality of prompt word texts are constructed based on a plurality of inquiry data by a first construction unit 702, wherein different prompt word texts at least contain different inquiry data. A plurality of prompt word texts are input into a first generation model by a first generation unit 704, and first response data corresponding to a plurality of prompt word texts are generated by using the first generation model. An arbitrary prompt word text and the corresponding first response data are input into a plurality of large language judgment models by a first judgment unit 706, and the first response data are judged by using a plurality of large language judgment models respectively to obtain a plurality of first judgment results corresponding to the first response data, wherein the first judgment results are used to characterize whether the first response data is correct. A training data is constructed based on a plurality of prompt word texts, the first response data corresponding to an arbitrary prompt word text, and a plurality of first judgment results by a second construction unit 708, wherein the training data is used to train the first generation model. That is to say, after using the first generation model to generate first reply data corresponding to multiple prompt word texts, the embodiment of the present application uses multiple large language judgment models to perform judgment cleaning on the first reply data, and then constructs training data based on multiple first judgment results. The constructed training data does not require the participation of manual annotation, and the data accumulates quickly, thereby achieving the technical effect of improving the efficiency of generating training data and solving the technical problem of low efficiency in generating training data.

[0144] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 3 A data generating device according to the data generating method shown.

[0145] Figure 8 is a schematic diagram of a data generating device according to an embodiment of the present application, such as Figure 8 As shown, the data generating device 800 may include: a first display unit 802 and a second display unit 804 .

[0146] The first display unit 802 is used to respond to input instructions acting on the operation interface and display multiple query data on the operation interface.

[0147] The second display unit 804 is used to respond to the processing instructions acting on the operation interface and display the training data on the operation interface, wherein the training data is constructed based on multiple prompt word texts, first reply data corresponding to the multiple prompt word texts and multiple first judgment results corresponding to the first reply data, the multiple prompt word texts are constructed based on multiple inquiry data, the first reply data is input into the first generation model, and the first generation model is used to generate the multiple prompt word texts, the multiple first judgment results are input into any prompt word text and the first reply data into multiple large language judgment models, and the first reply data are judged by the multiple large language judgment models respectively, the first judgment result is used to characterize whether the first reply data is correct, and the training data is used to train the first generation model.

[0148] It should be noted that the first display unit 802 and the second display unit 804 correspond to steps S302 to S304 in the above embodiment, and the two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run as part of the device in the server 10 provided in the above embodiment.

[0149] In the data generating device, a plurality of inquiry data are displayed on the operation interface in response to input instructions on the operation interface through the first display unit 802. Training data are displayed on the operation interface in response to processing instructions on the operation interface through the second display unit 804, wherein the training data is constructed based on a plurality of prompt word texts, first reply data corresponding to the plurality of prompt word texts, and a plurality of first judgment results corresponding to the first reply data, the plurality of prompt word texts are constructed based on a plurality of inquiry data, the first reply data is generated by inputting a plurality of prompt word texts into a first generation model and using the first generation model, the plurality of first judgment results are obtained by inputting any prompt word text and the first reply data into a plurality of large language judgment models, and using the plurality of large language judgment models to judge the first reply data respectively, the first judgment result is used to characterize whether the first reply data is correct, and the training data is used to train the first generation model. That is to say, after using the first generation model to generate first reply data corresponding to multiple prompt word texts, the embodiment of the present application uses multiple large language judgment models to perform judgment cleaning on the first reply data, and then constructs training data based on multiple first judgment results. The constructed training data does not require the participation of manual annotation, and the data accumulates quickly, thereby achieving the technical effect of improving the efficiency of generating training data and solving the technical problem of low efficiency in generating training data.

[0150] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 4 A data generating device according to the data generating method shown.

[0151] Fig. 9 is a schematic diagram of a data generating device according to an embodiment of the present application, such as Fig. 9 As shown, the data generating device 900 may include: a first acquiring unit 902 , a third constructing unit 904 , a second generating unit 906 , a second judging unit 908 , a fourth constructing unit 910 and an output unit 912 .

[0152] The first acquisition unit 902 is configured to acquire multiple query data by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter includes multiple query data.

[0153] The third constructing unit 904 is configured to construct a plurality of prompt word texts based on the plurality of query data, wherein different prompt word texts at least contain different query data.

[0154] The second generating unit 906 is used to input the plurality of prompt word texts into the first generating model, and generate first reply data corresponding to the plurality of prompt word texts by using the first generating model.

[0155] The second judgment unit 908 is used to input any prompt word text and the corresponding first reply data into multiple large language judgment models, and use multiple large language judgment models to judge the first reply data respectively to obtain multiple first judgment results corresponding to the first reply data, wherein the first judgment result is used to indicate whether the first reply data is correct.

[0156] The fourth construction unit 910 is used to construct training data based on multiple prompt word texts, first reply data corresponding to any prompt word text, and multiple first judgment results, wherein the training data is used to train the first generation model.

[0157] The output unit 912 is used to output the training data by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the training data.

[0158] It should be noted that the first acquisition unit 902, the third construction unit 904, the second generation unit 906, the second adjudication unit 908, the fourth construction unit 910 and the output unit 912 correspond to steps S402 to S412 in the above embodiment, and the six modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server 10 provided in the above embodiment as part of the device.

[0159] In the data generating device, a first acquisition unit 902 acquires a plurality of inquiry data by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes a plurality of inquiry data. A third construction unit 904 constructs a plurality of prompt word texts based on the plurality of inquiry data, wherein different prompt word texts at least include different inquiry data. A second generation unit 906 inputs a plurality of prompt word texts into a first generation model, and the first generation model is used to generate a plurality of first reply data corresponding to the prompt word texts. A second judgment unit 908 inputs any prompt word text and the corresponding first reply data into a plurality of large language judgment models, and the plurality of large language judgment models are used to judge the first reply data respectively, to obtain a plurality of first judgment results corresponding to the first reply data, wherein the first judgment result is used to characterize whether the first reply data is correct. A fourth construction unit 910 constructs training data based on a plurality of prompt word texts, the first reply data corresponding to any prompt word text, and the plurality of first judgment results, wherein the training data is used to train the first generation model. An output unit 912 outputs the training data by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the training data. That is to say, after using the first generation model to generate first reply data corresponding to multiple prompt word texts, the embodiment of the present application uses multiple large language judgment models to perform judgment cleaning on the first reply data, and then constructs training data based on multiple first judgment results. The constructed training data does not require the participation of manual annotation, and the data accumulates quickly, thereby achieving the technical effect of improving the efficiency of generating training data and solving the technical problem of low efficiency in generating training data.

[0160] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the above embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the above embodiments.

[0161] An embodiment of the present application may provide a computing device. Fig.10 is a structural block diagram of a computing device according to an embodiment of the present application, such as Fig.10 As shown, the computing device 1000 may include: one or more (only one is shown in the figure) processors 1002, a memory 1004, a storage controller, and a peripheral interface.

[0162] The above-mentioned computing device can be understood as an integrated intelligent terminal, including but not limited to a server, a desktop computer, a PC (Personal Computer), a model all-in-one machine, etc., and the computing device can be pre-installed with the model described in the above-mentioned embodiments of the present application.

[0163] Specifically, the computing device can pre-set multiple types of models, including but not limited to models in the fields of natural language processing, visual processing, speech processing, code processing, multimodal task processing, etc., so as to provide a variety of model choices. In different product forms, the computing device can support one or more model usage methods, including but not limited to model training, model calling, model fine-tuning, model deployment, model reasoning and application, etc. In some product forms, the computing device also supports model management, including but not limited to multi-type model management (supporting the management of multiple types of models such as discriminants and generative models), model version control (supporting the control of different model versions), model evaluation (based on model evaluation tools, evaluating the performance and effect of the model), etc. In other product forms, the computing device can also create applications based on the model, provide API calling capabilities, and can call the model to the created application through the API interface, while providing application management tools to achieve application management and control.

[0164] Furthermore, the computing device can also include data management (supporting the creation and management of model tuning data sets), a training center (providing rich training resources to help users learn and master AI technology), and basic management and control capabilities (providing enterprise-level basic management and control capabilities to ensure the security and efficient operation of the system). Through the above functions, a comprehensive, integrated AI development, training, deployment and application device is provided.

[0165] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0166] The processor may call the executable program stored in the memory through the transmission device to execute the method described in any one of the above embodiments.

[0167] An embodiment of the present application may provide an electronic device. Fig.11 is a structural block diagram of an electronic device according to an embodiment of the present application, such as Fig.11As shown, the electronic device may include: an input / output device 1102 ; a memory 1104 , and a processor 1106 , wherein the processor 1106 is connected to the input / output device 1102 and the memory 1104 via a bus 1108 .

[0168] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0169] The processor may call the executable program stored in the memory through the transmission device to execute the method described in any one of the above embodiments.

[0170] It will be understood by those skilled in the art that the structure shown in the figure is for illustration only, and the computing device may also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, and a mobile Internet device (Mobile Internet Devices, MID), PAD, etc. The figure does not limit the structure of the above computing device. For example, the computing device 1000 may also include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a configuration different from that shown in the figure.

[0171] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0172] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.

[0173] Optionally, in this embodiment, the above storage medium may be located in a computing device.

[0174] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program, and when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the method described in any one of the above embodiments.

[0175] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and the computer program implements the method provided in the embodiment when executed by a processor.

[0176] The embodiments of the present application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program, and when the computer program is executed by a processor, the method provided in the embodiments is implemented.

[0177] The embodiment of the present application further provides a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the method provided in the above embodiment is implemented.

[0178] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0180] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0182] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0183] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data generation method, characterized in that: include: Based on the multiple query data, construct multiple prompt word texts, wherein different prompt word texts at least contain different query data; Inputting the plurality of prompt word texts into a first generation model, and using the first generation model to generate first reply data corresponding to the plurality of prompt word texts; Input any prompt word text and the corresponding first reply data into multiple large language judgment models, and use the multiple large language judgment models to judge the first reply data respectively to obtain multiple first judgment results corresponding to the first reply data, wherein the first judgment results are used to indicate whether the first reply data is correct; Based on the multiple prompt word texts, the first reply data corresponding to any one of the prompt word texts and the multiple first judgment results, training data is constructed, wherein the training data is used to train the first generation model.

2. The method according to claim 1, characterized in that: The training data includes first-stage data for supervised adjustment of the first generation model and second-stage data for preference alignment of the first generation model; the training data is constructed based on the plurality of prompt word texts, the first reply data corresponding to any one of the prompt word texts, and the plurality of first judgment results, including: For any prompt word text, based on the multiple first judgment results, determine the type of the prompt word text; constructing the first stage data based on a first prompt word text belonging to the first type among the plurality of prompt word texts and first reply data corresponding to the first prompt word text; A second prompt word text belonging to the second type among the multiple prompt word texts is input into a second generation model, second reply data corresponding to the second prompt word text is generated using the second generation model, and the second stage data is constructed based on the second prompt word text, the first reply data corresponding to the second prompt word text, and the second reply data corresponding to the second prompt word text, wherein a performance index of the second generation model is greater than a performance index of the first generation model.

3. The method according to claim 2, characterized in that The step of determining the type of any prompt word text based on the plurality of first judgment results includes: Determining the number of target referee results among the plurality of first referee results, wherein the target referee results are used to characterize the first reply data error; When the number is less than or equal to the preset number, determining the type of the prompt word text to be the first type; When the number is greater than the preset number, the type of the prompt word text is changed to the second type.

4. The method according to claim 2, characterized in that: The constructing the second-stage data based on the second prompt word text, the first reply data corresponding to the second prompt word text, and the second reply data corresponding to the second prompt word text includes: Inputting the second prompt word text and the corresponding second reply data into the multiple large language adjudication models, and using the multiple large language adjudication models to adjudicate the second reply data respectively to obtain multiple second adjudication results corresponding to the second reply data, wherein the second adjudication results are used to indicate whether the second reply data is correct; When the plurality of second judgment results all indicate that the second reply data is correct, the second prompt word text, the corresponding first reply data and the corresponding second reply data are determined to be the second stage data.

5. The method according to claim 1, characterized in that The method of constructing multiple prompt word texts based on multiple query data includes: Determine item data of a target item corresponding to any query data, and object data of a target object to which the target item belongs; The plurality of prompt word texts are generated based on the plurality of query data, the item data and the object data.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Using the first stage data included in the training data to perform supervised adjustment on the first generation model to obtain an adjusted generation model; The second-stage data included in the training data is used to perform preference alignment on the adjusted generation model to obtain a target generation model.

7. The method according to claim 6, characterized in that The step of using the second-stage data included in the training data to perform preference alignment on the adjusted generation model to obtain a target generation model includes one of the following: Building a reward model based on the second-stage data, and using the reward model to perform preference alignment on the adjusted generation model to obtain the target generation model; The second-stage data and a direct preference optimization algorithm are used to perform preference alignment on the adjusted generation model to obtain the target generation model.

8. A data generation method, characterized in that: include: In response to an input instruction acting on the operation interface, displaying a plurality of inquiry data on the operation interface; In response to a processing instruction acting on the operation interface, training data is displayed on the operation interface, wherein the training data is constructed based on multiple prompt word texts, first reply data corresponding to the multiple prompt word texts, and multiple first judgment results corresponding to the first reply data, the multiple prompt word texts are constructed based on the multiple inquiry data, the first reply data is generated by inputting the multiple prompt word texts into a first generation model and using the first generation model, the multiple first judgment results are obtained by inputting any prompt word text and the first reply data into multiple large language judgment models, and using the multiple large language judgment models to judge the first reply data respectively, the first judgment result is used to characterize whether the first reply data is correct, and the training data is used to train the first generation model.

9. A data generation method, characterized in that: include: Acquire multiple query data by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter includes the multiple query data; Based on the plurality of inquiry data, construct a plurality of prompt word texts, wherein different prompt word texts at least contain different inquiry data; Inputting the plurality of prompt word texts into a first generation model, and using the first generation model to generate first reply data corresponding to the plurality of prompt word texts; Input any prompt word text and the corresponding first reply data into multiple large language judgment models, and use the multiple large language judgment models to judge the first reply data respectively to obtain multiple first judgment results corresponding to the first reply data, wherein the first judgment results are used to indicate whether the first reply data is correct; constructing training data based on the multiple prompt word texts, the first reply data corresponding to any one of the prompt word texts, and the multiple first judgment results, wherein the training data is used to train the first generation model; The training data is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter includes the training data.

10. A computing device, characterized in that include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 9 when running.

11. An electronic device, characterized in that: include: A memory storing an executable program; A processor, connected to the memory via a bus, and configured to run the program, wherein the program executes the method described in any one of claims 1 to 9 when running.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images