Data processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps that description information of an auxiliary object is obtained, and the auxiliary object is used for assisting a first target model to process a question and answer task; through a second target model, on the basis of the description information, generating target information which needs to be generated when the first target model calls the auxiliary object to perform question and answer task processing; and through the second target model, a problem which needs to be input into the first target model and is used for triggering generation of the target information is predicted, the target information and the problem are used for generating a training sample of the first target model, and the generation efficiency of the target information and the problem can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] A Large Language Model (LLM) refers to a deep learning model trained using a large amount of text data, which can handle question-answering tasks.

[0003] In order to improve the effect of the large language model in handling question-answering tasks in a certain field, the question-answering data related to the auxiliary object in a certain field will be used to fine-tune the large language model. For example, when the auxiliary object is an article in the medical field, the question-answering data related to the auxiliary object can be the information in the article and the questions entered by the user. For example, when the auxiliary object is a function tool, the question-answering data related to the auxiliary object can be the answer output by the function tool and the question entered by the user.

[0004] However, currently, the question and answer data related to the auxiliary objects when fine-tuning the large language model are all manually generated by users, which reduces the efficiency of question and answer data generation. Summary of the invention

[0005] The embodiments of the present application provide a data processing method, device, electronic device and computer-readable storage medium, which can improve the efficiency of generating question and answer data related to auxiliary objects.

[0006] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0007] Obtain description information of an auxiliary object, where the auxiliary object is used to assist the first target model in processing the question-answering task;

[0008] Generate, by means of the second target model, target information required to be generated when the first target model calls the auxiliary object to process the question-answering task based on the description information;

[0009] Through the second target model, it is predicted that questions need to be input into the first target model to trigger the generation of the target information, and the target information and the questions are used to generate training samples for the first target model.

[0010] In a second aspect, an embodiment of the present application further provides a data processing device, the device comprising:

[0011] An information acquisition module, used to acquire description information of auxiliary objects, where the auxiliary objects are used to assist the first target model in processing question-answering tasks;

[0012] A first generating module is used to generate target information required to be generated when the first target model calls the auxiliary object to process the question-answering task based on the description information through the second target model;

[0013] The second generation module is used to predict, through the second target model, the questions that need to be input into the first target model to trigger the generation of the target information. The target information and the questions are used to generate training samples for the first target model.

[0014] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory storing a plurality of instructions; the above-mentioned processor loads instructions from the above-mentioned memory to execute any one of the data processing methods provided in the embodiments of the present application.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for loading by a processor to execute any data processing method provided in the embodiment of the present application.

[0016] In an embodiment of the present application, descriptive information of an auxiliary object is obtained, and the auxiliary object is used to assist the first target model in processing a question and answer task; through the second target model, based on the descriptive information, target information required to be generated when the first target model calls the auxiliary object to process the question and answer task is generated; through the second target model, it is predicted that questions that need to be input into the first target model to trigger the generation of target information are used, and the target information and questions are used to generate training samples for the first target model, thereby realizing automatic generation of question and answer data related to the auxiliary object without manual generation, thereby improving the generation efficiency of question and answer data related to the auxiliary object. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 This is a flow chart of an embodiment of a data processing method provided in an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of the model training process provided in the embodiment of the present application;

[0020] Figure 3 is a schematic diagram of a confusion matrix provided in an embodiment of the present application;

[0021] Figure 4is a schematic diagram of a model evaluation method provided in an embodiment of the present application;

[0022] Figure 5 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0023] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0025] Before explaining the embodiments of the present application in detail, some terms involved in the embodiments of the present application are first explained.

[0026] Among them, in the description of the embodiment of the present application, the terms "first", "second", etc. can be used in this article to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. "Multiple" refers to two or more.

[0027] The embodiments of the present application provide a data processing method, device, electronic device and computer-readable storage medium. Specifically, the data processing method of the embodiments of the present application can be executed by an electronic device, wherein the electronic device can be a terminal or a server.

[0028] The terminal can be a terminal device such as a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer (PC), a personal digital assistant (PDA), etc. The terminal can also include a client, which can be a game application client, a browser client carrying a game program, or an instant messaging client, etc.

[0029] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0030] The following is a detailed description of the embodiments in conjunction with the accompanying drawings. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.

[0031] In this embodiment, a terminal is taken as an example for description. This embodiment provides a data processing method, such as Figure 1 As shown, the specific process of the data processing method can be as follows:

[0032] 101. Obtain description information of an auxiliary object, where the auxiliary object is used to assist the first target model in processing the question-answering task.

[0033] The first target model may be an untrained neural network model or a trained neural network model. When the second target model is a trained neural network model, the second target model may refer to a large language model (LLM), which can process natural language tasks, including question-answering tasks.

[0034] The auxiliary object is used to assist the first target model in processing question-answering tasks. It can be understood that after the first target model is trained using question-answering data related to the auxiliary object, the ability of the trained target model to process question-answering tasks in the target domain can be improved. The target domain refers to the domain where the auxiliary object is located. The target domain can include a general domain or a target professional domain.

[0035] The target professional field refers to a specific knowledge category, which can be set according to actual conditions. For example, the target professional field can be the medical field or the automotive field, which is not limited in this embodiment.

[0036] It can be understood that after the first target model is trained using the question and answer data related to the auxiliary object, the trained target model can be enabled to have the ability to understand questions in the target domain and answer questions in the target domain. Therefore, when the trained target model is used to process question and answer tasks in the target domain, the trained target model can be used without the help of the auxiliary object. Alternatively, after the target model is trained using the question and answer data related to the auxiliary object, the trained target model can be enabled to have the ability to understand questions in the target domain and call the auxiliary object. Therefore, when the trained target model is used to process question and answer tasks in the target domain, the trained target model can call the auxiliary object to obtain answers.

[0037] The type of auxiliary object can be set according to actual conditions. For example, when the first target model needs to be improved in processing question-answering tasks in the medical field, the auxiliary object can be an article in the medical field. For example, when the first target model needs to process question-answering tasks in a certain professional field by calling a function tool, the auxiliary object can be a function tool. The embodiments of the present application are not limited here.

[0038] The description information of the auxiliary object may refer to text that explains the auxiliary object. For example, when the auxiliary object is an article, the description information of the auxiliary object is the content of the article; when the auxiliary object is a function tool, the description information of the auxiliary object is the function description of the function tool and the parameter description of the parameters required to call the function tool; for another example, when the auxiliary object is a code library, the description information of the auxiliary object is the code in the code library.

[0039] 102. Generate target information required when the first target model calls the auxiliary object to process the question-answering task based on the description information through the second target model.

[0040] Among them, the second target model may refer to a trained neural network model, for example, the second target model may refer to a large language model (LLM), which can process natural language tasks, including question-answering tasks. The specific type of the large language model can be based on actual conditions, for example, the second target model may be Azure-OpenAI-GPT-3.5 or Microsoft Azure-GPT-4.0, which is not limited in the embodiments of the present application. When the first target model is also a large language model, the second target model and the first target model may be the same model or different models.

[0041] The target information required to be generated when the first target model calls the auxiliary object to process the question and answer task can refer to the answer generated when the first target model calls the auxiliary object to process the question and answer task, or it can refer to the instruction of the first target model to call the auxiliary object, and the instruction is used to call the auxiliary object to obtain the answer.

[0042] For example, when the auxiliary object is an article, the target information may be the answer to the question; when the auxiliary object is a function tool, the target information may be the instruction to call the function tool, or the answer obtained after calling the function tool.

[0043] In this embodiment, the target information required to be generated when the first target model calls the auxiliary object to process the question-answering task is automatically generated based on the description information, without the need for the user to manually generate the target information, thereby improving the efficiency of generating the target information.

[0044] In some embodiments, when the auxiliary object is an article or a code library, the target information is an answer, and the target information required to be generated when the first target model calls the auxiliary object to perform question-answering task processing is generated based on the description information through the second target model, including:

[0045] Through the second target model, key content is selected from the description information;

[0046] Combine the key points to get the answer.

[0047] In some embodiments, when the auxiliary object is a function tool, the target information may be an instruction for calling the function tool.

[0048] In some embodiments, when the auxiliary object is a function tool, the target information required to be generated when the first target model calls the auxiliary object to process the question-answering task is generated based on the description information through the second target model, including:

[0049] Generate target parameters required to be input by the first target model for calling the function tool according to the function description and the parameter description through the second target model;

[0050] Assemble the target parameter and the tool name of the function tool to obtain an instruction for calling the function tool.

[0051] Among them, the function tool refers to a tool obtained by encapsulating the code that implements certain functions, which may include at least one. The type of code that implements certain functions can be set according to actual conditions, for example, the code that implements certain functions can be a function, an interface, a model, or a workflow, etc., which is not limited in this embodiment. Certain functions that are implemented can be set according to actual conditions, for example, the functions that are implemented can be a weather query function or a product query function, which is not limited in this embodiment.

[0052] When the auxiliary object is a function tool, the description information of the auxiliary object is the function description of the function tool and the parameter description of the parameters required to call the function tool. The function description of the function tool refers to the information describing the function of the function tool. For example, if the function of the function tool is a weather query function, the function description of the function tool can be that the function tool is used to query the weather of a certain place.

[0053] When calling a function tool, you need to enter some parameters. The description of these parameters is the parameter description. When the function tool implements different functions, the parameters required to call the function tool are different, and the parameter description is also different. For example, the function of the function tool is a weather query function, and the parameters required to be entered are location and time. The parameter description can be the parameters that need to be entered to represent the location and time. For another example, the function of the function tool is a salary query function, and the parameter required to be entered is a name. The parameter description is the parameters that need to be entered to represent the name.

[0054] The target parameter required to be input when calling a function tool refers to the specific value of the parameter required to be input when calling the function tool. For example, the function of the function tool is the weather query function, the parameters required to be input when calling the function tool are the parameters representing the location and time, and the target parameters required to be input when calling the function tool are Wuhan and tomorrow.

[0055] After obtaining the target parameter, the terminal may assemble the target parameter and the tool name of the function tool to obtain an instruction for calling the function tool.

[0056] For example, the function of the function tool is the salary query function, the target parameter is "Zhang San", the tool name is salary_search, and the instruction for calling the function tool can be "salary_search(name='Zhang San')".

[0057] It can be understood that when there are multiple parameters required to be input for calling a function tool, and each of the required input parameters corresponds to multiple target parameters, the terminal can first generate each target parameter through a second target model, and then combine the various target parameters to obtain a combined parameter, and then assemble the combined parameter and the tool name of the function tool to obtain an instruction to call the function tool.

[0058] For example, the parameters required to call a function tool include parameters representing time and parameters representing location. The target parameters generated for the parameters representing time include tomorrow and the day after tomorrow. The target parameters generated for the parameters representing location include Wuhan and Shenzhen. Tomorrow and Wuhan are combined to obtain a combined parameter. Tomorrow and Shenzhen are combined to obtain a combined parameter. The day after tomorrow and Shenzhen are combined to obtain a combined parameter. The day after tomorrow and Wuhan are combined to obtain a combined parameter. A total of 4 combined parameters are obtained. The 4 combined parameters are then combined with the tool name respectively to obtain 4 instructions for calling the function tool.

[0059] Alternatively, when the parameters required to be input for calling the function tool include multiple parameters, and each required input parameter corresponds to multiple target parameters, the terminal can directly generate a combined parameter through the second target model, and the combined parameter includes at least two target parameters. For example, the parameters required to be input for calling the function tool include a parameter representing time and a parameter representing location. The terminal can directly generate the target parameter "Wuhan" and the target parameter "tomorrow" through the second target model to obtain the combined parameter.

[0060] Optionally, when one of the target parameters affects the generation of other target parameters, a method of directly generating the combined parameter through the first target model may be adopted.

[0061] Optionally, when generating the target parameters required for inputting a function tool through the second target model according to the function description and parameter description, the parameter generation prompt template can also be input into the second target model, so as to generate the target parameters required for inputting a function tool through the second target model according to the function description, parameter description and parameter generation prompt template.

[0062] Optionally, if samples or generation rules are required to generate the target parameters, the samples or generation rules can be stored in the parameter description. In this case, the parameter description not only includes a description of the parameters required to be input, but also includes samples or generation rules, so that the second target model can refer to the samples or generation rules and generate the target parameters according to the function description and parameter description.

[0063] In some embodiments, the terminal may generate instructions according to the function information of the function tool, wherein the function information includes the tool name, function description, parameter name, parameter description, parameter type, and parameter necessity of the function tool.

[0064] 103. Through the second target model, predict the problems that need to be input into the first target model to trigger the generation of target information.

[0065] Among them, the question is equivalent to the content input by the user. When processing the question-and-answer task through the first target model, the user usually needs to manually input the question, and the first target model then generates an answer based on the question. At this time, the answer can be the answer directly generated by the first target model, or the answer can also be the answer obtained by the first target model calling a function tool.

[0066] For example, the question input by the user is "What's wrong with frequent nosebleeds?", and the answer directly output by the first target model based on "What's wrong with frequent nosebleeds?" is "It may be due to dry nasal cavity or injury to the inner wall of the nasal cavity."

[0067] In this embodiment, first generate the answer "It may be due to dry nasal cavity or injury to the inner wall of the nasal cavity" based on the auxiliary object, and then generate the question "What's wrong with frequent nosebleeds?" based on this answer. "What's wrong with frequent nosebleeds?" is the question that needs to be input into the first target model to trigger the generation of "It may be due to dry nasal cavity or injury to the inner wall of the nasal cavity."

[0068] Another example is that the question input by the user is "What is Zhang San's salary?", and the instruction output by the first target model based on "What is Zhang San's salary?" is "salary_search(name='Zhang San')", and the answer obtained by calling the function tool based on this instruction is "Zhang San's salary is XXXX."

[0069] In this embodiment, first generate the instruction "salary_search(name='Zhang San')" based on the function tool, and then generate "What is Zhang San's salary?" based on the instruction. "What is Zhang San's salary?" is the question that needs to be input into the first target model to trigger the generation of "salary_search(name='Zhang San')".

[0070] In this embodiment, first automatically generate the target information, and then simulate the question input by the user according to the target information, without the user manually inputting the question, which improves the generation efficiency of the question, and further improves the generation efficiency of the question-and-answer data. The question-and-answer data includes the target information and the question.

[0071] In some embodiments, the question can be a short sentence or a long sentence. For example, when the auxiliary object is a function tool and the function tool is a weather query function, the question can be "Wuhan, tomorrow's weather", "Wuhan, tomorrow's weather" is a short sentence. Another example is that the question can be "I want to ask about the weather in Wuhan tomorrow", "I want to ask about the weather in Wuhan tomorrow" is a long sentence.

[0072] In some embodiments, when the auxiliary object is a function tool and the target information is an instruction, before predicting, through the second target model, the question that needs to be input into the first target model to trigger the generation of the target information, it further includes:

[0073] Perform quality inspection on the instruction and obtain the quality inspection result of the instruction;

[0074] Determine the instructions corresponding to the quality test results that meet the preset quality conditions,

[0075] Through the second target model, the problems that need to be input into the first target model to trigger the generation of target information are predicted, including:

[0076] Through the second target model, it is predicted that problems need to be input into the first target model to trigger the generation of target instructions, and the target instructions refer to instructions corresponding to quality inspection results that meet preset quality conditions.

[0077] The process of performing quality inspection on the instruction and obtaining the quality inspection result of the instruction may be:

[0078] The quality inspection of the instruction is performed according to at least one of the composition structure of the instruction, the tool name in the instruction, the parameter in the instruction and the type of the parameter in the instruction to obtain a quality inspection result of the instruction.

[0079] For example, when quality inspection is performed on the instruction according to the composition structure of the instruction, it can be analyzed whether the instruction includes the tool name and parameters. If the instruction includes the tool name and parameters, the quality inspection result of the instruction is represented as the first quality inspection result. The first quality inspection result represents the quality inspection result that meets the preset quality conditions.

[0080] For another example, when quality inspection is performed on an instruction based on the parameters in the instruction, it can be analyzed whether the parameters in the instruction are parameters that must appear and whether they are parameters that need to be input. If the parameters in the instruction are parameters that need to be input, it indicates that the quality inspection result of the instruction is a first quality inspection result. If the parameters in the instruction are parameters that do not need to be input, it indicates that the quality inspection result of the instruction is a second quality inspection result. The second quality inspection result is a quality inspection result that does not meet the preset quality conditions.

[0081] It can be understood that when quality inspection is performed on instructions based on multiple types of instruction composition structure, tool name in the instruction, parameters in the instruction, and parameter types in the instruction, the quality inspection result of the instruction can be determined based on various sub-quality inspection results.

[0082] For example, the instruction is quality checked based on its composition structure, and the obtained sub-quality check result is the first quality check result. The instruction is quality checked based on the parameters in the instruction, and the obtained sub-quality check result is the second quality check result. The quality check result of the instruction is determined based on the first quality check result and the second quality check result.

[0083] It can be understood that when the sub-quality detection result includes the second quality detection result, the quality detection result of the instruction can be determined as the second quality detection result.

[0084] In some embodiments, when the auxiliary object is a function tool, the question includes target parameters that need to be input.

[0085] In some embodiments, when a question includes target parameters that need to be input, the question includes a complete question and an incomplete question. The complete question includes all target parameters that need to be input when calling a function tool, and the incomplete question includes some target parameters that need to be input when calling a function tool.

[0086] For example, the parameters required to call a function tool include parameters representing location and parameters representing time. A complete question may be "What will the weather be like in Wuhan tomorrow?" In this case, the complete question includes the target parameter "Wuhan" representing the location and the target parameter "tomorrow" representing the time. An incomplete question may be "What will the weather be like tomorrow?" In this case, the incomplete question only includes the target parameter "tomorrow" representing the time, and lacks the target parameter "Wuhan" for the location.

[0087] In an embodiment of the present application, questions include complete questions and incomplete questions. Complete questions include all target parameters required to be input for calling a function tool, and incomplete questions include some target parameters required to be input for calling a function tool, thereby diversifying the generated questions. After the first target model is subsequently trained according to the question and target information, the ability of the trained target model to handle question-answering tasks in the target field can be further improved.

[0088] In some embodiments, the present embodiment further includes:

[0089] Generate supplementary questions based on the incomplete question through the second target model, where the supplementary questions are used to prompt the input of target parameters missing in the incomplete question;

[0090] Supplementary information is generated based on the supplementary questions through the second target model, and the supplementary information includes the missing target parameters. The supplementary questions and the supplementary information are used to generate training samples for the first target model.

[0091] Among them, a complete question and an incomplete question can be generated based on the instruction. Alternatively, a complete question can be generated based on the instruction, and then an incomplete question can be generated according to the target parameter.

[0092] After obtaining the incomplete question, a supplementary question is generated based on the incomplete question, and supplementary information is generated based on the supplementary question.

[0093] For example, the incomplete question is "I want to check the salary for January". The incomplete question lacks the target parameter representing the name. The generated supplementary question can be "Are you querying for January of this year? In addition, can you provide the name of the query?" Based on the supplementary question, the generated supplementary information can be, for example, "It is January 2023, and the name is Zhang San."

[0094] After obtaining the supplementary questions and the supplementary information, the terminal may generate training samples of the first target model according to the supplementary questions, the supplementary information, the instructions and the questions.

[0095] Specifically, the terminal can generate training samples based on incomplete questions and supplementary questions, in which the incomplete questions are used as data and the supplementary questions are used as labels, and the training samples are used to train the questioning ability of the first target model. Training samples can be generated based on incomplete questions, supplementary questions, supplementary information and instructions. In the training samples, the incomplete questions, supplementary questions and supplementary information are used as data and the instructions are used as labels, and the training samples are used to train the ability of the first target model to call function tools.

[0096] It can be understood that the types of training samples generated based on incomplete questions and supplementary questions and the types of training samples generated based on incomplete questions, supplementary questions, supplementary information, instructions and function tool sets belong to multi-round dialogue types.

[0097] In an embodiment of the present application, target information and questions are used to generate training samples for a first target model. A supplementary question is generated based on an incomplete question through a second target model. The supplementary question is used to prompt for input of target parameters missing from the incomplete question. Supplementary information is generated based on the supplementary question through the second target model. The supplementary information includes the missing target parameters. The supplementary question and supplementary information are used to generate training samples for the first target model. The training samples include training samples, so that after the first target model is trained according to the training samples, the trained target model has the ability to ask questions, so that the trained target model can more clearly understand the user's intentions and further enhance the ability of the trained target model to call function tools.

[0098] In some embodiments, the present embodiment further includes:

[0099] According to the instructions, call the function tool to get the parameter value corresponding to the problem;

[0100] Based on the parameter values, the answers corresponding to the questions are generated, and the parameter values ​​and answers are used to generate training samples.

[0101] For example, the question is "What is Zhang San's salary in January 2023", and the instruction is "salary_search(name='Zhang San')". According to the instruction, a function tool is called, and the parameter value corresponding to the question is obtained as "{ 'emp_no': '123456', 'name': 'Zhang San', 'time': '2023 / 01','salary': 'xxxx'}". The answer corresponding to the question is "Hello, it is found that Zhang San's salary in January 2023 is xxxx".

[0102] At this time, training samples can be generated according to the instruction and the question. In this training sample, the question is used as data and the instruction is used as a label. This training sample is used to train the ability of the first target model to call function tools. Training samples can be generated according to the parameter value and the answer. In this training sample, the parameter value is used as data and the answer is used as a label. This training sample is used to train the ability of the first target model to summarize the parameter value returned by the function tool into an answer. Training samples can be generated according to the instruction, the question, the function tool set, the parameter value, and the answer. In this training sample, the instruction, the question, and the parameter value are used as data and the answer is used as a label. This training sample is used to train the ability of the first target model to summarize the parameter value returned by the function tool into an answer.

[0103] It can be understood that the types of training samples generated according to the parameter value and the answer and the training samples generated according to the instruction, the question, the function tool set, the parameter value, and the answer belong to the multi-round dialogue type.

[0104] It can be understood that when the question includes an incomplete question, not only can training samples be generated according to the incomplete question and the supplementary question, according to the incomplete question, the supplementary question, the supplementary information, and the instruction, but also training samples can be generated according to the incomplete question, the supplementary question, the supplementary information, the instruction, the function tool set, the parameter value, and the answer. In this training sample, the incomplete question, the supplementary question, the supplementary information, the instruction, and the parameter value are used as data and the answer is used as a label. This training sample is used to train the ability of the first target model to summarize the parameter value returned by the function tool into an answer. The type of training sample generated according to the incomplete question, the supplementary question, the supplementary information, the instruction, the parameter value, and the answer belongs to the multi-round dialogue type.

[0105] In the embodiment of the present application, according to the instruction, a function tool is called to obtain the parameter value corresponding to the question. According to the parameter value, the answer corresponding to the question is generated. The parameter value and the answer are used to generate training samples, so that after training the first target model according to the training samples, the ability of the trained target model to summarize the parameter value into an answer can be improved, making it easier for users to understand the answer.

[0106] In some embodiments, the second target model predicts the problem that needs to be input into the first target model to trigger the generation of target information, including:

[0107] Get the set role information;

[0108] Through the second target model, based on the target information and the role information, it is predicted that the questions that need to be input into the first target model are used to trigger the generation of the target information.

[0109] The role information refers to information describing the identity of the user. Different role information indicates different users. For example, the role information may be information indicating the identity of a personnel or information indicating the identity of an ordinary employee.

[0110] For example, the function of the function tool is a salary query function. If the role information is information indicating the identity of a personnel person, then the question that needs to be input into the first target model to trigger the generation of the target information may be "I need to inquire about Zhang San's salary." If the role information is information indicating the identity of an ordinary employee, then the question that needs to be input into the first target model to trigger the generation of the target information may be "I want to know how much Zhang San earns every month."

[0111] Since different users may input different questions for the same instruction, in this embodiment, the second target model is used to predict the questions that need to be input into the first target model to trigger the generation of target information based on the target information and role information, thereby diversifying the generated questions and further improving the ability of the trained target model obtained by training the first target model based on the target information and questions to handle question-answering tasks in the target field.

[0112] In some embodiments, after predicting, through the second target model, the problem that needs to be input into the first target model to trigger the generation of target information, the method further includes:

[0113] The problem is enhanced to obtain the enhanced problem.

[0114] The enhanced question and target information are used together to generate training samples of the first target model. The type of enhancement processing can be set according to actual conditions, for example, the enhancement processing includes at least one of replacement processing, rewriting processing, simplification processing and error addition processing, which is not limited in this embodiment.

[0115] Replacement processing refers to replacing non-critical content in the question, and the replaced content does not affect the generation of instructions. Rewriting processing refers to changing the grammar and questioning method without changing the critical content. Simplification processing refers to simplifying the question into a short sentence. Error addition processing refers to adding typos to the question.

[0116] In the embodiment of the present application, after the questions are obtained, the questions are enhanced so that the questions ultimately used to generate training samples are diversified.

[0117] In some embodiments, after predicting, through the second target model, the problem that needs to be input into the first target model to trigger the generation of target information, the method further includes:

[0118] Generate training samples of the first target model according to the target information and the question;

[0119] According to the training samples, the first target model is trained to obtain a trained target model.

[0120] In this embodiment, training samples of the first target model are generated based on the target information and questions, and the first target model is trained based on the training samples to obtain a trained target model, thereby improving the ability of the trained target model to handle question-answering tasks in the target field.

[0121] Furthermore, since the target information and questions are automatically generated as question-and-answer data for the second target model, the efficiency of generating question-and-answer data is improved. Therefore, the efficiency of training the first target model can be improved based on the target information and questions.

[0122] In addition, since the target information is first generated by the second target model and then questions are generated based on the target information, not only can the quality of the questions and the quality of the target information not be heavily dependent on the capabilities of the second target model, but the questions and target information can also be made diverse, structurally complete and stable, thereby further improving the ability of the trained target model to handle question-answering tasks in the target field.

[0123] In some embodiments, the auxiliary object is a function tool, which generates a training sample of the first target model according to the target information and the problem, including:

[0124] The problem, the target information and the function tool set where the function tool is located are assembled to obtain the training samples of the first target model.

[0125] The process of assembling the problem, the target information, and the function tool set where the function tool is located to obtain the training sample of the first target model may be:

[0126] Generate a training sample of a first target model according to the partial instructions, the partial questions, and the function tool set where the function tool is located;

[0127] The order of the function tools in the function tool set is adjusted to obtain an adjusted function tool set;

[0128] Generate training samples for the first target model based on the additional instructions, the additional questions and the adjusted function tool set.

[0129] For example, a function tool set includes function tool c1, function tool c2, and function tool c3. Function tool c1 is the first function tool in the function tool set, function tool c2 is the second function tool in the function tool set, and function tool c3 is the third function tool in the function tool set. After adjusting the order of the function tools in the function tool set, function tool c1 is the second function tool in the adjusted function tool set, function tool c3 is the first function tool in the adjusted function tool set, and function tool c2 is the third function tool in the adjusted function tool set.

[0130] In this embodiment, training samples of the first target model are generated according to some instructions, some problems and the function toolset in which the function tools are located, and the order of the function tools in the function toolset is adjusted to obtain the adjusted function toolset. According to other parts of the instructions, other parts of the problems and the adjusted function toolset, training samples of the first target model are generated, so that the order of the function tools in the function toolset in different training samples is different, thereby preventing the trained target model from overfitting the order of the function tools in the function toolset and improving the generalization ability of the trained target model in the target domain.

[0131] In some embodiments, the auxiliary object is a function tool, the target information is an instruction, and the training sample of the first target model is generated according to the target information and the problem, including:

[0132] Assemble an instruction, the problem corresponding to the instruction, and the function tool set to obtain a training sample;

[0133] Assemble the two instructions, the problems corresponding to the two instructions, and the function tool set to obtain training samples.

[0134] The two instructions may be any two instructions in the generated instructions. Optionally, the order of the questions corresponding to the two instructions in the training sample is determined based on the order of the two instructions in the training sample. For example, the instruction for calling the function tool c1 is instruction d1, and the question that triggers the generation of instruction d1 is q1. The instruction for calling the function tool c2 is instruction d2, and the question that triggers the generation of instruction d2 is q2. In the training sample generated according to instruction d1, question q1, instruction d2, and question q2, the order of instruction d1 precedes the order of instruction d2, and the order of question q1 precedes the order of question q2.

[0135] It can be understood that by assembling the two instructions, the problems corresponding to the two instructions, and the function tool set, the type of training samples obtained belongs to the parallel type.

[0136] In the present application, an instruction, the problem corresponding to the instruction, and the function tool set are assembled to obtain a training sample, and two instructions, the problems corresponding to the two instructions, and the function tool sets are assembled to obtain a training sample, so that one training sample includes two instructions, so that two function tools can be called in parallel, thereby improving the availability of the target model after training.

[0137] In some embodiments, the type of training samples includes at least two of a single-tool type, a multi-tool type, a parallel type, and a multi-round dialogue type. The single-tool type refers to a type in which a function tool set includes only one function tool, the multi-tool type refers to a type in which a function tool set includes at least two function tools, the parallel type refers to a type in which a training sample includes two instructions and two questions that trigger the generation of the two instructions; the multi-round dialogue type refers to a type in which a training sample contains multi-round dialogue information, wherein the multi-round dialogue information is used to inquire based on the parameters required to input the function tool to obtain all the required input parameters.

[0138] In an embodiment of the present application, the types of training samples include at least two, which improves the diversity of training samples and further improves the ability of the trained target model obtained by training the training samples to handle question-answering tasks in the target field.

[0139] In some embodiments, the order of each type of training samples in the training set is disrupted, so that the trained target model obtained by training the first target model according to the training samples will not be overfitted, thereby improving the generalization ability of the trained target model.

[0140] In some embodiments, the first target model may also be trained based on irrelevant type training samples, where irrelevant type training samples are training samples generated based on question-and-answer data that are irrelevant to the function tool call.

[0141] In some embodiments, training the first target model according to the training sample to obtain a trained target model includes:

[0142] Inputting the problem and function tool set in the training sample into the first target model for processing to obtain a first prediction instruction;

[0143] Determine a loss function value based on the first predicted instruction and instructions in the training sample;

[0144] If the loss function value satisfies the preset loss condition, the first target model is determined as the trained target model;

[0145] If the loss function value does not meet the preset loss condition, the weight of the first target model is updated according to the loss function value, and the step of inputting the problem and function tool set in the training sample into the first target model for processing is returned.

[0146] It can be understood that the loss function value determined based on the first prediction instruction and the instruction in the training sample is called the first loss function value. When the training sample includes an incomplete question and a supplementary question, the incomplete question is input into the first target model for processing to obtain a predicted supplementary question, and the second loss function value is determined based on the predicted supplementary question and the supplementary question. When the training sample includes a parameter value and an answer, the parameter value is input into the first target model for processing to obtain a predicted answer, and the third loss function value is determined based on the answer and the predicted answer. At this time, the loss function value of the first target model can be determined based on the first loss function value, the second loss function value, and the third loss function value. If the loss function value meets the preset loss condition, the first target model is determined as the trained target model. If the loss function value does not meet the preset loss condition, the weight of the first target model is updated according to the loss function value, and the step of inputting the training sample into the first target model for processing is returned.

[0147] In some embodiments, training the first target model according to the training sample to obtain a trained target model includes:

[0148] Obtain real question-and-answer data;

[0149] Generate real training samples based on real question-answering data;

[0150] The first target model is trained according to the training samples and the real training samples to obtain a trained target model.

[0151] The target information and questions generated by the second target model in this embodiment can be referred to as simulated question-answering data. The real question-answering data includes questions manually input by the user and the target information corresponding to the manually input questions.

[0152] In an embodiment of the present application, real question and answer data is obtained, and real training samples are generated based on the real question and answer data. The first target model is trained based on the training samples and the real training samples to obtain a trained target model. The first target model is trained based on the simulated question and answer data and the real question and answer data, thereby further improving the ability of the trained target model to handle question and answer tasks in the target field.

[0153] In some embodiments, the auxiliary object is an auxiliary object in a target professional field, and based on the training sample, the first target model is trained to obtain a trained target model, including:

[0154] Obtain training samples in general domains;

[0155] Based on the training samples and the training samples in the general field, the first target model is trained to obtain a trained target model.

[0156] In an embodiment of the present application, training samples in the general field are obtained, and a first target model is trained based on the training samples and the training samples in the general field to obtain a trained target model, so that the trained target model is not only good at handling question-answering tasks in the target professional field, but also can maintain the ability to handle question-answering tasks in the general field.

[0157] In some embodiments, based on the training sample, the first target model is trained to obtain a trained target model, including:

[0158] Set the adapter of the first target model;

[0159] Input the training sample into the first target model for processing to obtain the loss function value;

[0160] If the loss function value does not meet the preset loss condition, the weight of the adapter is updated based on the loss function value, and the step of inputting the training sample into the first target model for processing to obtain the loss function value is returned;

[0161] If the loss function value meets the preset loss condition, the weight of the adapter and the weight of the network layer in the first target model are fused to obtain the trained target model.

[0162] Among them, setting the adapter of the first target model may refer to adding a new network layer in parallel with at least one network layer in the first target model, and the weight of the new network layer is obtained by multiplying two low-rank matrices. The size of the weight of the new network layer is the same as the size of the weight of at least one network layer. If the size of the weight of at least one network layer is m*n, the two matrices obtained by splitting the m*n matrix are m*r and r*n, that is, the sizes of the two matrices of the new added network layer are m*r and r*n respectively. If the weight of at least one network layer is updated directly, the amount of parameters to be updated is m*n, and if the weight of the additional network layer is updated, the amount of parameters to be updated is m*r+r*n, and m*r+r*n is less than m*n.

[0163] Therefore, in an embodiment of the present application, an adapter of the first target model is set, and the training sample is input into the first target model for prediction processing to obtain a loss function value. If the loss function value does not meet the preset loss condition, the weight of the adapter is updated based on the loss function value, and the step of inputting the training sample into the first target model for processing to obtain the loss function value is returned to execute. If the loss function value meets the preset loss condition, the last updated weight of the adapter and the weight of at least one network layer in the first target model are fused to obtain a trained target model, thereby reducing the amount of parameters that need to be updated and further improving the training efficiency of the first target model.

[0164] For example, when the first target model is trained based on the training samples and the real training samples to obtain the trained target model, the training process of the first target model can be as follows: Figure 2 shown.

[0165] In some embodiments, the present embodiment further includes:

[0166] Get the preset update times;

[0167] Get the weight of the adapter after the preset update times;

[0168] The adapter weights are fused with the weights of the network layer in the first target model to obtain a trained target model, including:

[0169] The weights updated at the preset update times are merged to obtain the weight of the adapter;

[0170] The adapter weights are fused with the weights of the network layers in the second target model to obtain the trained target model.

[0171] The preset update times may also be referred to as breakpoints, which refer to some of the update times of the adapter. Since the model training times can be set before training, the preset update times can be determined according to the training times. For example, the preset update times may be the 1 / 3, 2 / 3 and last of the training times, which is not limited in this embodiment.

[0172] For example, the number of training times is 30 times. When the loss function value meets the preset loss condition, the weight of the adapter is updated 30 times. The preset update times are the 10th, 20th and 30th times. The weight of the adapter after the 10th update, the weight after the 20th update and the weight after the 30th update are fused to obtain the weight of the adapter. Then, the weight of the adapter is fused with the weight of at least one network layer in the first target model to obtain the trained target model. At this time, the weight of the adapter fused with the weight of at least one network layer in the first target model is not the weight of the adapter after the last update, but the weight after the weight of the adapter is fused after the preset update times.

[0173] It is understandable that the terminal can record the weight of the adapter after being updated for a preset number of times, and then obtain the weight of the adapter after being updated for a preset number of times when the loss function value meets the preset loss condition.

[0174] As the training time increases, the ability of the first target model to handle question-answering tasks in general fields will decrease. In this embodiment, a preset number of updates is obtained, the weight of the adapter after the preset number of updates is obtained, the weight after the preset number of updates is fused to obtain the weight of the adapter, and the weight of the adapter and the weight of the network layer in the first target model are fused to obtain the trained target model, which can not only improve the ability of the trained target model to handle question-answering tasks in the target professional field, but also maintain the ability of the trained target model to handle question-answering tasks in general fields, so that the trained target model will not be overfitted to the training samples, and the generalization ability of the trained target model is improved.

[0175] In some embodiments, the auxiliary object is an auxiliary object in a target professional field, and before generating a training sample of the first target model according to the target information and the question, the method further includes:

[0176] Obtain model introduction information of each candidate large language model;

[0177] According to the target professional field and model introduction information, a first target model is determined from various candidate large language models.

[0178] The model introduction information includes at least one of basic information about the model, features of the model, and a log of the model.

[0179] The terminal can determine the degree of match between each candidate large language model and the target professional field based on the model introduction information and the requirements of the target professional field, and then determine the candidate large language model with the highest matching degree as the first target model. Alternatively, the terminal can predict the degree of match between the candidate large language model and the target professional field based on the model introduction information and the requirements of the target professional field, recommend each candidate large language model to the user according to the matching degree, and then, in response to the user's selection operation on the candidate large language model, determine the candidate large language model corresponding to the selection operation as the first target model.

[0180] Specifically, the matching degree between each candidate large language model and the target professional field can be determined according to the characteristics of the model and the requirements of the target professional field. For example, the target professional field is the hotel customer service field, and the problems in this field are usually in the form of text. At this time, the matching degree of the candidate large language model that is good at processing text-based problems is greater than the matching degree of the candidate large language model that is good at processing image-based problems. For another example, the target professional field is the legal field, and the problems in this field are usually long. At this time, the matching degree of the candidate large language model that is good at processing long problems is greater than the matching degree of the candidate large language model that is good at processing short problems.

[0181] In an embodiment of the present application, model introduction information of each candidate large language model is obtained, and a first target model is determined from each candidate large language model based on the target professional field and the model introduction information, so that when the first target model is trained, the training of the first target model can be completed more quickly, further improving the efficiency of obtaining the trained target model.

[0182] In some embodiments, before generating a training sample of the first target model according to the target information and the question, the method further includes:

[0183] Display each candidate large language model;

[0184] In response to a selection operation on each candidate large language model, the candidate large language model corresponding to the selection operation is determined as a first target model.

[0185] In the embodiment of the present application, each candidate large language model is displayed. In response to a selection operation on each candidate large language model, the candidate large language model corresponding to the selection operation is determined as the first target model, so that the user can select the first target model by himself, thereby improving the user experience.

[0186] In some embodiments, the auxiliary object is an auxiliary object in the target professional field, and obtaining description information of the auxiliary object includes:

[0187] Display the data generation interface;

[0188] In response to an input operation on the data generation interface, obtaining a name of a target professional field;

[0189] Through the second target model, based on the name of the target professional field, the description information of the auxiliary object of the target professional field is obtained.

[0190] In an embodiment of the present application, a data generation interface is displayed, and in response to an input operation on the data generation interface, the name of the target professional field is obtained. Based on the name of the target professional field, descriptive information of the auxiliary objects of the target professional field is obtained through a second target model, so that target information and questions are generated based on the descriptive information, so that the user can select the target professional field according to his or her needs, thereby making the trained target model good at handling question-and-answer tasks in the professional field required by the user, thereby enabling the user to customize the model.

[0191] It is understandable that when the user manually selects the first target model, the user may choose to generate target information and questions before selecting the first target model, or may choose to generate target information and questions while selecting the first target model, which is not limited in this embodiment.

[0192] In some embodiments, the user may also provide auxiliary objects in the target professional field so that the trained target model can better meet the user's needs.

[0193] In some embodiments, after obtaining the trained target model, the method further includes:

[0194] Obtaining an evaluation set including at least one evaluation sample;

[0195] Based on the evaluation samples, the ability of the trained target model to handle question-answering tasks in the target domain is evaluated.

[0196] Among them, the evaluation samples in the evaluation set can be generated manually or according to the target information and questions. The generation process can refer to the process of generating training samples according to the target information and questions, and this embodiment will not be repeated here.

[0197] When the auxiliary object is a function tool, the evaluation sample includes question-answering data related to the function tool in the function tool set and the function tool set. The question-answering data includes questions and instructions. For the convenience of description, the questions in the training samples are called first questions, the instructions in the training samples are called first instructions, the questions in the evaluation samples are called second questions, and the instructions in the evaluation samples are called second instructions. At this time, based on the evaluation samples, the ability of the trained target model to handle the question-answering task in the target field is evaluated, including:

[0198] Inputting the second problem and the function tool set into the trained target model for processing, and generating a second prediction instruction that calls the function tools in the function tool set;

[0199] Based on the function tools called by the second prediction instruction and the function tools called by the second instruction, the ability of the trained target model to call the function tools in the function toolset to process question and answer tasks is evaluated.

[0200] Among them, after the second question and the function tool set are input into the trained target model, the trained target model can predict which function tool in the function tools can get the answer to the question, and then generate a second prediction instruction to call the function tool.

[0201] For example, the function tool set includes function tool c1, function tool c2 and function tool c3. It is predicted that the answer to the question is obtained through function tool c2, and a second prediction instruction for calling function tool c2 is generated.

[0202] The second prediction instruction includes a tool name, and the second instruction also includes a tool name. The function tool called by the second prediction instruction refers to the function tool corresponding to the tool name in the second prediction instruction, and the function tool called by the second instruction refers to the function tool corresponding to the tool name in the second instruction. The function tool called by the second instruction is equivalent to a label, and based on the label, it is determined whether the function tool called by the second prediction instruction is correct.

[0203] For the same problem, if the function tool called by the second prediction instruction is the same as the function tool called by the second instruction, it indicates that the target training model after training calls the function tool accurately. If the function tool called by the second prediction instruction is different from the function tool called by the second instruction, it indicates that the target training model after training calls the function tool incorrectly, and other function tools in the function tool set interfere with the call of this function tool.

[0204] For example, the evaluation sample includes question q1 and instruction d1. Instruction d1 is an instruction for calling function tool c3. The function tool set includes function tool c1, function tool c2 and function tool c3. Question q1 and the function tool set are input into the trained target model for processing to obtain the second prediction instruction d11. If the second prediction instruction d11 is a prediction instruction for calling function tool c3, it indicates that the call to function tool c3 is accurate. If the second prediction instruction d11 is a prediction instruction for calling function tool c2, it indicates that an error occurs in the call to function tool c3 and function tool c2 interferes with the call to function tool c3.

[0205] After receiving the second prediction instruction, the terminal can count the number of correct calls and the number of incorrect calls of each function tool based on the function tools called by the second prediction instruction and the function tools called by the second instruction. Finally, based on the statistical results of each function tool, the terminal evaluates the ability of the trained target model to call the function tools in the function tool set to process question and answer tasks.

[0206] In this embodiment, based on the function tools called by the second prediction instruction and the function tools called by the second instruction, the ability of the trained target model to call the function tools in the function tool set to process question and answer tasks is evaluated, thereby simplifying the evaluation task of the ability of the trained target model to process question and answer tasks into a classification task of function tools. This makes it possible to obtain the calling ability of the trained target model on each function tool, refine the granularity of evaluating the ability of the trained target model to process question and answer tasks, and improve the accuracy of evaluating the ability of the trained target model to process question and answer tasks.

[0207] In some embodiments, based on the function tool called by the second prediction instruction and the function tool called by the second instruction, evaluating the ability of the trained target model to call the function tool in the function tool set to process the question-answering task includes:

[0208] Determine, according to the function tool called by the second prediction instruction and the function tool called by the second instruction, an influence coefficient between the function tools, the influence coefficient being used to indicate the degree of mutual influence between the function tools;

[0209] Based on the influence coefficient, the ability of the trained target model to call the function tools in the function toolset to handle question answering tasks is evaluated.

[0210] Among them, when the function tool called by the second prediction instruction is different from the function tool called by the second instruction, it can be determined that the function tool called by the second prediction instruction interferes with the function tool called by the second instruction, thereby obtaining the degree of mutual influence between the various function tools, so that the training samples can be adjusted according to the degree of mutual influence between the various function tools, and the trained target model can continue to be fine-tuned based on the training samples. Therefore, the evaluation method of this embodiment can provide guidance for the fine-tuning of the trained target model and improve the efficiency of obtaining the target model for final use.

[0211] In some embodiments, based on the function tool called by the second prediction instruction and the function tool called by the second instruction, evaluating the ability of the trained target model to call the function tool in the function tool set to process the question-answering task includes:

[0212] Constructing a confusion matrix according to the function tool called by the second predicted instruction and the function tool called by the instruction, wherein the column data in the confusion matrix indicates the function tool called by the second instruction, and the row data in the confusion matrix indicates the function tool called by the second predicted instruction;

[0213] Based on the confusion matrix, the ability of the trained target model to call the function tools in the function toolset to handle question answering tasks is evaluated.

[0214] The confusion matrix can also be called the error matrix, which is used to display the classification results of the target model for the function tool after training.

[0215] For example, the confusion matrix can be written as Figure 3 As shown, from Figure 3 It can be seen that there are 11 evaluation samples for function tool c1. Among the second prediction instructions generated based on the problems in the 11 evaluation samples, the function tool in 8 second prediction instructions is function tool c1, indicating that 8 second prediction instructions are accurate and the call to function tool c1 is accurate. The function tool in 3 second prediction instructions is function tool c3, indicating that the call to function tool c1 is mistakenly identified as a call to function tool c3, indicating that an error occurs in the call to function tool c3.

[0216] In this embodiment, a confusion matrix is ​​constructed based on the function tools called by the second prediction instruction and the function tools called by the second instruction. The column data in the confusion matrix indicates the function tools called by the second instruction, and the row data in the confusion matrix indicates the function tools called by the second prediction instruction. Based on the confusion matrix, the ability of the trained target model to call the function tools in the function tool set to process question and answer tasks is evaluated, so that not only can the classification results of the trained target model for the function tools be intuitively viewed, but the trained target model can also be comprehensively evaluated.

[0217] In some embodiments, based on the confusion matrix, evaluating the ability of the trained target model to call a function tool in a function tool set to process a question-answering task includes:

[0218] Based on the row data in the confusion matrix, determine the influence coefficient between the function tools, and the influence coefficient is used to indicate the degree of mutual influence between the function tools;

[0219] Based on the influence coefficient, the ability of the trained target model to call the function tools in the function toolset to handle question answering tasks is evaluated.

[0220] Among them, the type of influence coefficient can be set according to actual conditions. For example, the influence coefficient can be at least one of a true positive example (True Positive, TP), a true negative example (True Negative, TN), a false positive example (False Positive, FP) and a false negative example (False Negative, FN), which is not limited in the embodiments of the present application.

[0221] True positives indicate the number of evaluation samples that predict a function tool as the function tool. True negatives indicate the number of evaluation samples that predict other function tools as other function tools. False positives indicate the number of evaluation samples that predict other function tools as the function tool, which can indicate the interference caused by the function tool to other function tools. False negatives indicate the number of evaluation samples that predict a function tool as other function tools, which can indicate the interference caused by other function tools to the function tool.

[0222] In an embodiment of the present application, based on the row data in the confusion matrix, the influence coefficient between the function tools is determined, and the influence coefficient is used to indicate the degree of mutual influence between the function tools. Based on the influence coefficient, the ability of the trained target model to call the function tools in the function tool set to process the question and answer task is evaluated, so that the training samples can be adjusted according to the degree of mutual influence between the various function tools, and the trained target model can continue to be fine-tuned based on the training samples. Therefore, the evaluation method of this embodiment can provide guidance for the fine-tuning of the trained target model and improve the efficiency of obtaining the target model for final use.

[0223] In some embodiments, based on the influence coefficient, evaluating the ability of the trained target model to call a function tool in a function tool set to process a question-answering task includes:

[0224] Based on the influence coefficient, determine a call capability indication parameter for each function tool, where the call capability indication parameter includes at least one of accuracy, precision, recall, F1 score, specificity, and area under the ROC curve;

[0225] Based on the calling capability indication parameter for each function tool, the ability of the trained target model to call each function tool is determined.

[0226] For example, based on the influence coefficient of function tool c1, the calling capability indication parameter of function tool c1 is determined, and based on the calling capability indication parameter of function tool c1, the ability of the target model to call function tool c1 after training is determined. Based on the influence coefficient of function tool c2, the calling capability indication parameter of function tool c2 is determined, and based on the calling capability indication parameter of function tool c2, the ability of the target model to call function tool c2 after training is determined.

[0227] In an embodiment of the present application, based on the influence coefficient, a calling capability indication parameter for each function tool is determined, and the calling capability indication parameter includes at least one of accuracy, precision, recall, F1 score, specificity, and the area under the ROC curve. Based on the calling capability indication parameter for each function tool, the ability of the trained target model to call the function tool is determined, thereby obtaining the performance ability of the trained target model on each function tool.

[0228] In some embodiments, based on the influence coefficient, evaluating the ability of the trained target model to call a function tool in a function tool set to process a question-answering task includes:

[0229] Based on the influence coefficient, determine the overall evaluation coefficient of the trained target model for each function tool;

[0230] Based on the overall evaluation coefficient, the ability of the trained target model to call the function tools in the function toolset to handle question answering tasks is evaluated.

[0231] Among them, based on the influence coefficient, the process of determining the overall evaluation coefficient of the trained target model for each function tool can be:

[0232] Based on the influence coefficient, determining a call capability indication parameter for each function tool, where the call capability indication parameter includes at least one of a precision rate and a recall rate;

[0233] The call capability indication parameters are weighted to obtain the overall evaluation coefficient of the trained target model for each function tool, or the call capability indication parameters are averaged to obtain the overall evaluation coefficient of the trained target model for each function tool.

[0234] For example, when the call capability indication parameter is precision, the overall evaluation coefficient may be weighted precision or average precision; when the call capability indication parameter is recall, the overall evaluation coefficient may be weighted recall.

[0235] Among them, the weighted precision can be calculated by the following formula:

[0236]

[0237] p_i represents the precision of the i-th function tool, Number_i represents the number of evaluation samples related to the i-th function tool, and Total represents the number of all evaluation samples in the evaluation set.

[0238] The average precision can be calculated by the following formula:

[0239]

[0240] p_i represents the precision of the i-th function tool, and N represents the number of function tools in the function tool set.

[0241] In an embodiment of the present application, based on the influence coefficient, the overall evaluation coefficient of the trained target model for each function tool is determined, and based on the overall evaluation coefficient, the ability of the trained target model to call the function tools in the function tool set to process question and answer tasks is evaluated, thereby evaluating the trained target model based on the overall evaluation coefficient, thereby further improving the accuracy of the evaluation of the trained target model.

[0242] In some embodiments, after obtaining the second prediction instruction, the method further includes:

[0243] Based on the target dimension, analyzing the correctness of the second prediction instruction to obtain an analysis result, the target dimension including at least one of a structure of the second prediction instruction, a parameter in the second prediction instruction, and an applicable scenario of the second prediction instruction;

[0244] Based on the function tools called by the second prediction instruction and the function tools called by the second instruction, evaluate the ability of the trained target model to process question-answering tasks by calling the function tools in the function toolset, including:

[0245] Based on the function tools called by the second prediction instruction, the function tools called by the second instruction, and the analysis results, the ability of the trained target model to call the function tools in the function toolset to handle question-answering tasks is evaluated.

[0246] When the target dimension is the structure of the second prediction instruction, the correctness of the second prediction instruction is analyzed according to the structure of the second prediction instruction, and the process of obtaining the analysis result may be:

[0247] Determining whether the second prediction instruction includes a tool name and parameters;

[0248] If the second prediction instruction includes the tool name and parameters, it means that the structure of the second prediction instruction is correct, and the analysis result is determined to be a correct result.

[0249] When the target dimension is a parameter in the second prediction instruction, the correctness of the second prediction instruction is analyzed according to the parameter of the second prediction instruction, and the process of obtaining the analysis result may be:

[0250] Whether the parameters in the second prediction instruction are missing, whether there are unnecessary parameters, whether the types of the parameters are wrong, and whether the parameters match are analyzed to obtain an analysis result.

[0251] When the target dimension is the applicable scenario of the second prediction instruction, the correctness of the second prediction instruction is analyzed according to the applicable scenario of the second prediction instruction, and the process of obtaining the analysis result may be:

[0252] Determine an applicable scenario of the second prediction instruction, where the applicable scenario indicates whether a function tool needs to be called or whether the function tool does not need to be called;

[0253] According to the applicable scenario of the second prediction instruction, the correctness of the second prediction instruction is analyzed to obtain an analysis result.

[0254] Among them, when the applicable scenario of the second prediction instruction is that a function tool needs to be called, it may be that a function tool that does not need to be called is mistakenly identified as a function tool that needs to be called. When the applicable scenario of the second prediction instruction is that a function tool does not need to be called, it may be mistakenly identified as a function tool that does not need to be called.

[0255] In an embodiment of the present application, based on the target dimension, the correctness of the second prediction instruction is analyzed to obtain an analysis result, the target dimension includes the structure of the second prediction instruction, the parameters in the second prediction instruction, and at least one of the applicable scenarios of the second prediction instruction, and according to the function tools called by the second prediction instruction, the function tools called by the instruction, and the analysis results, the ability of the trained target model to call the function tools in the function tool set to process question and answer tasks is evaluated, so as to evaluate the trained target model from multiple dimensions and further evaluate the accuracy of the evaluation of the trained target model.

[0256] In some embodiments, when the question-and-answer data includes real question-and-answer data and simulated question-and-answer data, the terminal can generate evaluation samples based on the real question-and-answer data and the simulated question-and-answer data, determine the number of correct prediction instructions obtained by the trained target model and the number of correct prediction instructions obtained by the second target model based on the evaluation samples, and obtain the accuracy of the trained target model and the accuracy of the second target model based on the number of correct prediction instructions. The accuracy of the trained target model is used to indicate the ability of the trained target model to process question-and-answer tasks, and the accuracy of the second target model is used to indicate the ability of the second target model to process question-and-answer tasks, thereby comparing the ability of the trained target model to process question-and-answer tasks with the ability of the second target model to process question-and-answer tasks, for example. Figure 4 shown.

[0257] From the above content, it can be seen that in an embodiment of the present application, the descriptive information of the auxiliary object is obtained, and the auxiliary object is used to assist the first target model in processing question and answer tasks; through the second target model, based on the descriptive information, the target information required to be generated when the first target model calls the auxiliary object to process the question and answer task is generated; through the second target model, the questions that need to be input into the first target model to trigger the generation of the target information are predicted, so as to realize automatic generation of question and answer data related to the auxiliary object without manual generation, thereby improving the generation efficiency of question and answer data related to the auxiliary object.

[0258] In order to better implement the above method, an embodiment of the present application further provides a data processing device, which can be specifically integrated into an electronic device, such as a computer device, which can be a terminal, a server, or other device.

[0259] Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop computer, a personal computer and other devices; the server can be a single server or a server cluster composed of multiple servers.

[0260] For example, in this embodiment, the method of the embodiment of the present application is described in detail by taking the data processing device specifically integrated in the terminal as an example. This embodiment provides a data processing device, such as Figure 5 As shown, the data processing device may include:

[0261] The information acquisition module 501 is used to obtain description information of the auxiliary object, where the auxiliary object is used to assist the first target model in processing the question-answering task.

[0262] The first generation module 502 is used to generate target information required to be generated when the first target model calls the auxiliary object to process the question and answer task based on the description information through the second target model.

[0263] The second generation module 503 is used to predict, through the second target model, questions that need to be input into the first target model to trigger the generation of target information. The target information and questions are used to generate training samples for the first target model.

[0264] In some embodiments, the auxiliary object is a function tool, the description information is a functional description of the function tool and a parameter description of the parameters required to input when calling the function tool, the target information is an instruction to call the function tool, and the problem includes the target parameters that need to be input.

[0265] In some embodiments, the first generating module 502 is specifically configured to execute:

[0266] Generate target parameters required to be input by the first target model for calling the function tool according to the function description and the parameter description through the second target model;

[0267] Assemble the target parameter and the tool name of the function tool to obtain an instruction for calling the function tool.

[0268] In some embodiments, the question includes a complete question and an incomplete question, and the second generating module 503 is further used to execute:

[0269] Generate supplementary questions based on the incomplete question through the second target model, where the supplementary questions are used to prompt the input of target parameters missing in the incomplete question;

[0270] Supplementary information is generated based on the supplementary questions through the second target model, and the supplementary information includes the missing target parameters. The supplementary questions and the supplementary information are used to generate training samples for the first target model.

[0271] In some embodiments, the second generating module 503 is specifically used to execute:

[0272] Get the set role information;

[0273] Through the second target model, based on the target information and the role information, it is predicted that the questions that need to be input into the first target model are used to trigger the generation of the target information.

[0274] In some embodiments, the data processing device further includes a training module, and the training module is used to execute:

[0275] Generate training samples of the first target model according to the target information and the question;

[0276] According to the training samples, the first target model is trained to obtain a trained target model.

[0277] In some embodiments, the training module is specifically configured to perform:

[0278] Generate a training sample of a first target model according to the partial instructions, the partial questions, and the function tool set where the function tool is located;

[0279] The order of the function tools in the function tool set is adjusted to obtain an adjusted function tool set;

[0280] Generate training samples for the first target model based on the additional instructions, the additional questions and the adjusted function tool set.

[0281] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation methods and corresponding beneficial effects of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0282] Accordingly, an embodiment of the present application further provides an electronic device, which may be a terminal, and the terminal may be a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer (PC), a personal digital assistant (PDA), or other terminal devices. Figure 6 As shown, Figure 6A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 600 includes a processor 601 having one or more processing cores, a memory 602 having one or more computer-readable storage media, and a computer program stored in the memory 602 and executable on the processor. The processor 601 is electrically connected to the memory 602. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or arrange components differently.

[0283] The processor 601 is the control center of the electronic device 600. It uses various interfaces and lines to connect various parts of the entire electronic device 600, executes various functions of the electronic device 600 and processes data by running or loading software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, thereby monitoring the electronic device 600 as a whole.

[0284] In the embodiment of the present application, the processor 601 in the electronic device 600 will load instructions corresponding to the processes of one or more application programs into the memory 602 according to the following steps, and the processor 601 will run the application programs stored in the memory 602 to implement various functions, such as:

[0285] Get the description information of the auxiliary object, which is used to assist the first target model in processing the question-answering task;

[0286] Generate target information required when the first target model calls the auxiliary object to process the question-answering task based on the description information through the second target model;

[0287] Through the second target model, the prediction needs to be input into the first target model to trigger the question of generating target information, and the target information and the question are used to generate training samples for the first target model.

[0288] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the detailed description of the data processing method above, which will not be elaborated here.

[0289] Optional, such as Figure 6 As shown, the electronic device 600 further includes: a touch screen 603, a radio frequency circuit 604, an audio circuit 605, an input unit 606, and a power supply 607. The processor 601 is electrically connected to the touch screen 603, the radio frequency circuit 604, the audio circuit 605, the input unit 606, and the power supply 607, respectively. Those skilled in the art can understand that Figure 6The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0290] The touch display screen 603 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 603 may include a display panel and a touch panel. Among them, the display panel may be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device, and these graphical user interfaces may be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light-emitting diode (OLED, Organic Light-Emitting Diode) and the like. The touch panel may be used to collect the user's touch operation on or near it (such as the user using any suitable object or attachment such as a finger, a stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 601, and can receive the command sent by the processor 601 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 601 to determine the type of touch event, and then the processor 601 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 603 to realize the input and output functions. However, in some embodiments, the touch panel and the touch panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 603 can also be used as a part of the input unit 606 to realize the input function.

[0291] The radio frequency circuit 604 may be used to send and receive radio frequency signals, so as to establish wireless communication with a network device or other electronic devices through wireless communication, and to send and receive signals between the network device or other electronic devices.

[0292] The audio circuit 605 can be used to provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 605 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 605 and converted into audio data, and then the audio data is output to the processor 601 for processing, and then sent to another electronic device through the radio frequency circuit 604, or the audio data is output to the memory 602 for further processing. The audio circuit 605 may also include an earplug jack to provide communication between an external headset and an electronic device.

[0293] The input unit 606 may be used to receive input numbers, character information or user feature information (such as fingerprint, iris, facial information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0294] The power supply 607 is used to supply power to various components of the electronic device 600. Optionally, the power supply 607 can be logically connected to the processor 601 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 607 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0295] although Figure 6 Not shown, the electronic device 600 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0296] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0297] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0298] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored, and the computer program can be loaded by a processor to execute any data processing method provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0299] Get the description information of the auxiliary object, which is used to assist the first target model in processing the question-answering task;

[0300] Generate target information required when the first target model calls the auxiliary object to process the question-answering task based on the description information through the second target model;

[0301] Through the second target model, the prediction needs to be input into the first target model to trigger the question of generating target information, and the target information and the question are used to generate training samples for the first target model.

[0302] The specific implementation methods and corresponding beneficial effects of the above operations can be found in the detailed description of the data processing method above, which will not be elaborated here.

[0303] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0304] Since the computer program stored in the computer-readable storage medium can execute any data processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any data processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0305] The above is a detailed introduction to a data processing method, device, electronic device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining description information of an auxiliary object, where the auxiliary object is used to assist the first target model in processing the question-answering task; Generate, by the second target model, target information required to be generated when the first target model calls the auxiliary object to process the question-answering task based on the description information; Through the second target model, it is predicted that a question needs to be input into the first target model to trigger the generation of the target information, and the target information and the question are used to generate training samples for the first target model.

2. The data processing method according to claim 1, characterized in that: The auxiliary object is a function tool, the description information is a functional description of the function tool and a parameter description of the parameters required to be input to call the function tool, the target information is an instruction to call the function tool, and the question includes the target parameters that need to be input.

3. The data processing method according to claim 2, characterized in that: The generating, by the second target model, target information required to be generated when the first target model calls the auxiliary object to process the question-answering task based on the description information includes: Generate, by means of the second target model, target parameters required to be input by the first target model to call the function tool according to the function description and the parameter description; The target parameter and the tool name of the function tool are assembled to obtain an instruction for calling the function tool.

4. The data processing method according to claim 2, characterized in that: The question includes a complete question and an incomplete question, and the method further includes: Generate a supplementary question based on the incomplete question through the second target model, wherein the supplementary question is used to prompt for input of target parameters missing from the incomplete question; Supplementary information is generated based on the supplementary question through the second target model, and the supplementary information includes the missing target parameters. The supplementary question and the supplementary information are used to generate training samples for the first target model.

5. The data processing method according to claim 1, characterized in that: The predicting, by the second target model, the problem that needs to be input into the first target model to trigger the generation of the target information includes: Get the set role information; Through the second target model, based on the target information and the role information, it is predicted that the questions that need to be input into the first target model to trigger the generation of the target information.

6. The data processing method according to claim 1, characterized in that: After predicting, by the second target model, the problem that needs to be input into the first target model to trigger the generation of the target information, the method further includes: Generate a training sample of the first target model according to the target information and the problem; The first target model is trained according to the training samples to obtain a trained target model.

7. The data processing method according to claim 6, characterized in that: The auxiliary object is a function tool, the target information is an instruction, and generating a training sample of the first target model according to the target information and the problem includes: Generate a training sample of the first target model according to part of the instructions, part of the questions and the function tool set where the function tool is located; Adjusting the order of the function tools in the function tool set to obtain an adjusted function tool set; According to the instructions in another part, the problems in another part and the adjusted function toolset, training samples of the first target model are generated.

8. A data processing device, characterized in that: The device comprises: An information acquisition module, used to acquire description information of an auxiliary object, where the auxiliary object is used to assist the first target model in processing the question-answering task; A first generating module is used to generate target information required to be generated when the first target model calls the auxiliary object to process the question-answering task based on the description information through the second target model; The second generation module is used to predict, through the second target model, questions that need to be input into the first target model to trigger the generation of the target information, and the target information and the questions are used to generate training samples for the first target model.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the data processing method according to any one of claims 1 to 7.

Images