Model training method, electronic equipment, storage medium and program product

Abstract

The invention relates to the technical field of computers, and discloses a model training method, electronic equipment, a storage medium and a program product. In the method, a first training sample used for training a first large language model comprises a user instruction and at least one tool name corresponding to the user instruction, and each tool name comprises at least one reference mark. After a user instruction is input into the first large language model, first loss of each reference mark is determined based on a prediction result output by the model. According to different mark categories of the reference marks, different loss weights are set for the reference marks, and then weighting calculation is carried out on the first loss of the reference marks to obtain target loss. And training the first large language model based on the target loss to obtain a second large language model. In the method, the reference marks of different mark categories correspond to different loss weights, so that the accuracy of the second large language model in predicting the reference mark of the mark category with a relatively large loss weight can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, electronic equipment, storage medium and program product. Background Art

[0002] Large language models play a key role in natural language processing tasks. For example, large language models can be used to perform tasks such as text generation, machine translation, speech recognition, and can also be applied to question-answering systems.

[0003] It can be understood that in the process of performing tasks, the large language model can not only directly understand and generate natural language text, but also use tools to interact with the external environment to improve the model's application capabilities and accuracy. For example, the large language model can use tools such as application programming interface (API) interfaces, databases, online services, image processing software, etc. to interact with the external environment to access the latest information related to the task (such as obtaining weather information and traffic information), interact with users in real time (such as feeding back the latest news accessed to users), and enhance logical reasoning and computing capabilities.

[0004] Usually, a large language model may need to use different tools when performing different tasks or different processing steps in the same task. Taking the application of a large language model to a question-answering system as an example, if a user requests to inquire about the current weather, the large language model needs to obtain the weather data of the time and place required by the user based on a tool (such as a weather data interface) to feedback the weather data to the user. At this time, if the large language model cannot accurately select weather-related tools (such as a weather data interface), it will not be able to accurately feedback weather data to the user.

[0005] Therefore, when large language models use tools to perform various tasks, how to accurately select tools to increase the accuracy of task execution is a technical problem that needs to be solved urgently. Summary of the invention

[0006] Embodiments of the present application provide a model training method, electronic device, storage medium, and program product.

[0007] In a first aspect, the present application provides a model training method, which is applied to an electronic device, and the method includes: obtaining a first training sample, the first training sample includes a user instruction and at least one tool name corresponding to the user instruction, wherein each tool name includes at least one reference mark, and each tool name indicates a reference tool; inputting the user instruction into a first large language model to be trained to obtain a prediction result of the user instruction; calculating a first loss of each reference mark of a target tool name in at least one tool name according to the prediction result; calculating a loss weight of each reference mark of the target tool name according to the mark category of the reference mark, and the loss weights of reference marks of different mark categories are different; according to the loss weight of each reference mark of the target tool name, weighted calculation is performed on the first loss of each reference mark in the target tool name to obtain a target loss; adjusting the parameters of the first large language model based on the target loss to obtain a second large language model.

[0008] The model training method provided in the present application sets different loss weights for reference tags of different tag categories, which can improve the accuracy of the trained second largest language model in predicting reference tags of tag categories with larger loss weights.

[0009] In a possible implementation of the first aspect, the marker category includes a first category and a second category, and the loss weight of the reference markers of the first category is greater than the loss weight of the reference markers of the second category.

[0010] In a possible implementation of the first aspect, the method also includes: determining the mark category of the reference mark in the target tool name that meets the preset conditions as the first category; determining the mark category of the reference mark in the target tool name that does not meet the preset conditions as the second category; wherein the preset conditions include at least one of the following: the reference mark is the first mark in the target tool name; the reference mark is the mark after the target mark in the target tool name, and the target mark is the same mark in the target tool name that is located at the same position in different tool names.

[0011] Taking the tool names get_1 and get_2 as examples, the first tag of the tool name get_1 is "get" and the second tag is "_1". The first tag of the tool name get_2 is "get" and the second tag is "_2". Since the first tags of the tool names get_1 and get_2 are both "get", the reference tag "get" belongs to the first category in both the tool names get_1 and get_2. In addition, since the first tags of the tool names get_1 and get_2 are both "get", the tool names get_1 and get_2 have the same tag (i.e., the target tag) at the same position, therefore, the reference tag "_1" belongs to the first category in the tool name get_1, and the reference tag "_2" also belongs to the first category in the tool name get_2.

[0012] In the model training method provided by the present application, the loss weight of the reference markers of the first category is higher than the loss weight of the reference markers of the second category. Since the reference markers of the first category are markers that are prone to errors in the inference process of the first language model, the method provided by the present application can make the first language model pay more attention to the prediction accuracy of the reference markers that are prone to errors during the training process by distinguishing the importance of reference markers of different marker categories, thereby improving the accuracy of the second language model obtained by training.

[0013] In a possible implementation of the first aspect, the loss weight of each reference mark of the target tool name is calculated according to the mark category of the reference mark, including: determining the ratio of the first number to the second number as the loss weight of the reference mark of the first category in the target tool name, wherein the first number is the number of reference marks belonging to the second category in the target tool name where the reference mark is located, and the second number is the number of reference marks belonging to the first category in the target tool name where the reference mark is located.

[0014] If the reference mark The mark category is the first category, then the reference mark The calculation method of the loss weight includes: Taking i as 1 and l as 3 as an example, the tool name t1 includes the reference mark If the reference mark Belongs to the first category, reference mark Belongs to the second category, then the first quantity is 3, the second quantity is 1, and the reference mark The loss weight of is equal to 3 / 1, which is 3.

[0015] In a possible implementation of the first aspect, the loss weight of each reference mark of the target tool name is calculated according to the mark category of the reference mark, including: determining the target value as the loss weight of the reference mark of the second category in the target tool name.

[0016] This application does not limit the value of the target numerical value. It can be set based on experience or flexibly adjusted according to actual application scenarios. For example, the target numerical value can be 1.

[0017] In a possible implementation of the first aspect, the first loss of each reference mark of a target tool name in at least one tool name is calculated based on a prediction result, including: determining a target tool name from at least one tool name, wherein the number of the target tool names is one or more; and calculating the first loss of each reference mark of the target tool name based on the prediction result.

[0018] It can be understood that in the embodiment of the present application, at least one tool name can be directly used as the target tool name, or the wrong name in at least one target name can be removed to obtain the target tool name.

[0019] In a possible implementation of the first aspect, the first training sample also includes reference feedback information of each tool name in at least one tool name, the reference feedback information includes calling result information after the reference tool indicated by the corresponding tool name is called, and determining the target tool name from at least one tool name, including: determining the tool name that matches the reference feedback information and the error feedback information in at least one tool name as an error name; and removing the error name from at least one tool name to obtain the target tool name.

[0020] The error feedback information may be obtained in advance. The present application does not limit the method for determining the error feedback information. For example, the error feedback information corresponding to the selection and use of the tool based on other models may be used as the error feedback information in the present application.

[0021] In the present application, the reference feedback information corresponding to at least one tool name can be matched with the acquired error feedback information, and the tool name corresponding to the reference feedback information matching the error feedback information is used as the error name. In addition, the reference feedback information matching the error feedback information can be the same reference feedback information as the error feedback information.

[0022] This method of training the first language model based on the tool name (i.e., the target tool name) with incorrect names removed can avoid the problem of low accuracy of the trained large language model caused by using incorrect tool names during the training of the first language model, and thus can improve the accuracy of the response results of the trained second language model to user instructions.

[0023] In a possible implementation of the first aspect, there are multiple target tool names, and the first losses of each reference mark in the target tool name are weightedly calculated according to the loss weights of each reference mark in the target tool name to obtain the target loss, including: weighted summing up the first losses of each reference mark in each target tool name according to the loss weights of each reference mark in the target tool name to obtain the second losses corresponding to each target tool name; summing up the second losses corresponding to each target tool name to obtain the target loss.

[0024] The formulas corresponding to the second loss and the target loss can be referred to in the following formulas (9) and (10), respectively.

[0025] In a possible implementation of the first aspect, the prediction result of the user instruction includes a predicted probability of each reference mark in each tool name.

[0026] It can be understood that in this case, the reference numerals The predicted probability p M and reference mark Labels, calculate reference markers The corresponding first loss L1. The label is usually a binary value, for example, 1. It can be understood that in this case, the first loss L1 is obtained based on the logarithmic loss function.

[0027] In a possible implementation of the first aspect, the method further includes: based on the first training sample, using a proximal strategy optimization algorithm to adjust parameters of the second largest language model to obtain a target large language model.

[0028] The proximal strategy optimization algorithm in this application adopts a reward function to perform reinforcement learning on the second largest language model. This method can make the response results of the target large language model to user instructions more accurate.

[0029] In a second aspect, the present application provides an electronic device, comprising: one or more processors; one or more memories; one or more memories storing one or more programs, and when the one or more programs are executed by one or more processors, the electronic device executes the model training method of the first aspect and any possible implementation of the first aspect.

[0030] In a third aspect, the present application provides a readable storage medium having instructions stored thereon, which, when executed on an electronic device, causes the electronic device to execute the model training method of the first aspect and any possible implementation of the first aspect.

[0031] In a fourth aspect, the present application provides a computer program product, which includes: a computer program or instructions, which, when the computer program or instructions are run on an electronic device, enables the electronic device to execute the model training method of the first aspect and any possible implementation of the first aspect.

[0032] Among them, the beneficial effects of the second to fourth aspects can refer to the beneficial effects of the first aspect and any possible implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 According to some embodiments of the present application, a schematic diagram of a scenario in which a large language model uses a tool to execute user instructions is shown;

[0034] Figure 2 According to some embodiments of the present application, a flow chart of a method for training a large language model in the related art is shown;

[0035] Figure 3 According to some embodiments of the present application, a flowchart of a first model training method provided by the present application is shown;

[0036] Figure 4 According to some embodiments of the present application, a flow chart of a second model training method provided by the present application is shown;

[0037] Figure 5 According to some embodiments of the present application, a schematic diagram showing the performance of a TL-CodeLLaMA-2 model and a RoTLLaMA model in five different noise environments is shown;

[0038] Figure 6 According to some embodiments of the present application, a schematic diagram of the generalization capability of a TL-CodeLLaMA-2 model is shown;

[0039] Figure 7 According to some embodiments of the present application, a schematic diagram of the hardware structure of an electronic device is shown. DETAILED DESCRIPTION

[0040] The illustrative embodiments of the present application include, but are not limited to, a model training method, an electronic device, a storage medium, and a program product.

[0041] The following first explains the specific terms involved in the embodiments of the present application.

[0042] Large language models (LLMs): A deep learning-based natural language processing (NLP) model designed to understand and generate natural language text. In this application, large language models can use tools to interact with the external environment to complete various tasks.

[0043] The tools that can be used by the large language model include, but are not limited to: API, database, online service, image processing software, etc. It is understood that in addition to the tools that can be used by the large language model, such as API, database, online service, image processing software, etc., the tools involved in this application can also be in various other possible forms, which are not limited in this application.

[0044] Taking the tool as an API as an example, by calling the API, the large language model can obtain data for a specific service or perform a specific function, such as weather data acquisition, online translation, etc. For example, in a question-and-answer scenario, the large language model can obtain relevant information by calling the API to execute the instructions issued by the user. For example, the large language model can obtain weather data by calling the weather data API to feedback weather data to the user.

[0045] Taking the tool as a database as an example, the large language model can access and query data by interacting with the database, thereby providing more accurate query results. For example, in the field of scientific research, the large language model can access professional databases to obtain the latest research results and other data.

[0046] Taking the tool as an online service as an example, the big language model can obtain the latest relevant data by interacting with online services such as search engines and social media platforms. For example, in a question-and-answer scenario, the big language model can call a search engine to find and integrate relevant data to execute the instructions issued by the user.

[0047] Taking image processing software as an example, in the image description generation task, the large language model can interact with the image processing software to extract image features and generate corresponding descriptions.

[0048] The method provided in the embodiment of the present application can be applied to any electronic device including a large language model, including but not limited to a mobile station (MS), a mobile terminal (MT), etc. For example, the electronic device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a desktop computer, a laptop computer, a virtual reality (VR) device, an augmented reality (AR) device, a terminal in industrial control, a terminal in self-driving, a terminal in remote medical surgery, a terminal in a smart grid, a terminal in transportation safety, a terminal in a smart city, a terminal in a smart home, etc. The embodiment of the present application does not limit the specific form of the electronic device.

[0049] In conjunction with the accompanying drawings, the application scenario of the large language model obtained by the model training method of the present application is described below by taking the electronic device as a mobile phone 100 and the application scenario of the large language model as a question-answering scenario as an example.

[0050] Figure 1 FIG. 1 shows a schematic diagram of a scenario in which a large language model uses a tool to execute user instructions. Figure 1 As shown, taking the user's interaction with the large language model in the question-answering system of the mobile phone 100 based on the smart assistant application (application, APP) as an example, when the mobile phone 100 starts the smart assistant APP, the interactive interface can be displayed. The user can interact with the mobile phone 100 through user commands (such as text commands, voice commands, and picture commands). Figure 1 As shown, after the mobile phone 100 detects the user instruction corresponding to the display frame 101 (for example, "What is the weather in Shanghai today"), the mobile phone 100 can parse the user instruction corresponding to the display frame 101 based on the large language model and use NLP technology to identify the corresponding keywords and intentions, etc. For example, the keywords such as "today", "Shanghai" and "weather" are identified, as well as the user's intention to query weather data.

[0051] Then, the large language model can determine the tool to be used based on the recognized intent. For example, the large language model can determine that the weather data required by the user needs to be obtained based on the weather data interface. Next, after the large language model obtains the weather data based on the weather data interface, the weather data can be fed back to the user, for example, displayed on the display screen of the mobile phone 100. Figure 1 As shown, the feedback information corresponding to the display box 102 is, for example, "Today's weather in Shanghai is cloudy, and the temperature is 2 to 15°. Please remember to add a coat."

[0052] It can be understood that in some embodiments, the execution process of a user instruction may correspond to at least one processing step, and each processing step may use a different tool. In addition, each tool will generate corresponding feedback information during use, and the feedback information is used to indicate the result information of the corresponding tool after use, such as the result obtained based on the tool, and whether the tool is successfully called.

[0053] Specifically, during the execution of user instructions, the large language model can determine the tool corresponding to the next processing step (i.e., the upcoming processing step) based on the tools and feedback information corresponding to the already determined processing steps, so as to determine all the tools required for the user instructions.

[0054] In some embodiments, each tool used by the large language model is distinguished in the form of a tool name, and the tool name of each tool is generally composed of at least one word. This application will later describe the tool name using one word as one tag as an example, that is, the tool name of each tool includes at least one tag. It can be understood that in the process of the large language model using a tool to execute a user instruction, each tag in the tool name of the tool can be determined to find and use the tool through the tool name.

[0055] However, in the process of outputting tool names by a large language model, the first token in the tool name and the next token of a target token are more likely to be mispredicted, where the target token is the same token in the tool name at the same position in different tool names.

[0056] For example, the tool name of tool 1 is get_1, and the tool name of tool 2 is get_2. Both tool names include two tags. The first tag of tool name get_1 is "get", and the second tag is "_1". The first tag of tool name get_2 is "get", and the second tag is "_2". In the process of predicting tool names get_1 and get_2, the first tag "get" is prone to error.

[0057] In addition, since the first token of the tool names get_1 and get_2 are both "get", that is, the tool names get_1 and get_2 have the same token (that is, the target token) at the same position, therefore, in the process of predicting the tool names get_1 and get_2, the next token of the target token (such as "_1" and "_2") is also prone to errors.

[0058] In order to solve the problem that the large language model is prone to errors in predicting tool names, an embodiment of the present application provides a model training method. Specifically, the method obtains a first training sample, which includes a user instruction for training and at least one tool name corresponding to the user instruction. Among them, each tool name includes at least one reference mark, and each tool name indicates a reference tool. Then, the user instruction is input into the first large language model to be trained to obtain the prediction result of the user instruction. According to the prediction result, the first loss of each reference mark of the target tool name in at least one tool name is calculated. Then, the loss weight of each reference mark of the target tool name is determined according to the mark category of each reference mark, and the weight loss of different mark categories is different. According to the loss weight of each reference mark, the first loss of each reference mark is weightedly calculated (such as weighted summation) to obtain the target loss, and the parameters of the first large language model are adjusted based on the target loss to obtain the trained second large language model.

[0059] The mark categories include the first category and the second category. The reference mark of the first category is the first mark in each target tool name, and / or the mark after the target mark in the target tool name, and the target mark is the same mark in the tool name at the same position in different tool names. The reference marks of the second category are reference marks other than the reference marks corresponding to the first category.

[0060] In some embodiments, the loss weight of the reference markers of the first category is higher than the loss weight of the reference markers of the second category. In this way, since the reference markers of the first category are markers that are prone to errors in the inference process of the first large language model, the method provided by the present application sets a loss weight for the reference markers of the first category greater than the loss weight corresponding to the reference markers of the second category, which can make the first large language model pay more attention to the prediction accuracy of the reference markers of the first category during the training process, thereby improving the accuracy of the trained second large language model.

[0061] Before introducing the model training method of the present application in detail, for ease of understanding, the model training method in the related art is first described in combination with a formula.

[0062] It can be understood that in order to realize the tool search capability of the large language model, the pre-acquired training sample set can be used to perform model training on the large language model to be trained. The training sample set includes a tool set and at least one training sample, each training sample includes a user instruction, at least one tool name corresponding to the user instruction, and at least one reference feedback information corresponding to the at least one tool name, wherein one tool name corresponds to one reference feedback information, and one tool name is used to indicate one reference tool.

[0063] The following is a detailed description of the user instructions, reference feedback information, tool names and tool sets in this application.

[0064] As an example, if a user instruction q and a tool set T including multiple tools are given, then at least one tool name t corresponding to the user instruction q can be expressed as the following formula (1): where t∈T.

[0065] t=(t0,t1,…,t s ) Formula (1)

[0066] In the above formula (1), t i Represents the tool name of the reference tool corresponding to the user instruction q in the i-th processing step, where i∈[0,s] and i and s are integers.

[0067] It can be understood that the tool name t in the above formula (1) i It is based on t 0…i-1 and 0…i-1 Therefore, the tool name t is determined based on the large language model N. i For example, the tool name is t i The determination process of can be shown as the following formula (2).

[0068] t i =N(|q,T,t 0…i-1 ,o 0…i-1 ) Formula (2)

[0069] In addition, at least one reference feedback information o corresponding to the user instruction q can be expressed as the following formula (3).

[0070] o=(o0,o1,…,o s ) Formula (3)

[0071] In the above formula (3), o i Indicates the tool name i Corresponding reference feedback information.

[0072] In some embodiments, since each tool name includes at least one reference mark, the at least one tool name t corresponding to the user instruction q is t=(t0, t1, ..., t s ), tool name t i It can be further expressed as the following formula (4).

[0073]

[0074] in, Indicates the tool name i The mth reference marker in , m∈[0,l], and m and l are integers.

[0075] Next, taking the training samples corresponding to the tool set T and the user instruction q as an example, the training process of the large language model M to be trained in some embodiments is described in detail. Figure 2 As shown, the training process may include the following steps:

[0076] 201: Obtain a training sample set.

[0077] The training sample set D may include a tool set T and a user instruction q, at least one tool name corresponding to the user instruction q, and at least one reference feedback information corresponding to the at least one tool name. It can be understood that one tool name corresponds to one reference feedback information, and one tool name is used to indicate one reference tool.

[0078] Based on the above, the name of at least one tool corresponding to the user instruction q is t=(t0, t1, …, t s ); at least one reference feedback information corresponding to the user instruction q is o=(o0,o1,…,o s ).

[0079] In addition, each tool name includes at least one reference mark. Based on the above, the i-th tool name is t i = in Indicates the tool name i The corresponding m-th reference marker.

[0080] The present application does not limit the method for determining at least one tool name t and at least one reference feedback information o. For example, the tool name t and the reference feedback information o can be obtained from an open source website, or can be obtained by inputting a user instruction q into another large language model (such as a large language model N). It can be understood that if the tool name t and the reference feedback information o are obtained by inputting a user instruction q into a large language model N, then the large language model N must be a trained model, and the accuracy of the tool name output by the large language model N based on the user instruction q must be high. In this way, when the tool name output by the large language model N is used as a reference tool name to train the large language model M, it can be ensured that the trained large language model M also has a high accuracy.

[0081] 202: Input the user instruction into the large language model to be trained to obtain the predicted probability of each reference mark corresponding to each tool name.

[0082] In some embodiments, each tool name t i Corresponding reference signs The predicted probability p M It can be expressed as

[0083] 203: Calculate the loss corresponding to each reference tag according to the predicted probability and the label of each reference tag.

[0084] Among them, each reference mark The label is usually a binary value, such as 1.

[0085] Reference signs The corresponding loss L1 can be expressed as the following formula (5).

[0086]

[0087] In formula (5), Indicates p M The logarithmic value of .

[0088] 204: Sum the losses corresponding to the reference marks in the tool names to obtain the losses corresponding to the tool names.

[0089] Among them, the tool names are i The corresponding loss L2 can be expressed as the following formula (6).

[0090]

[0091] 205: Sum the losses corresponding to the tool names to obtain the loss corresponding to the user command.

[0092] The loss L3 corresponding to the user instruction can be expressed as the following formula (7).

[0093]

[0094] 206: Adjust the parameters of the large language model according to the loss corresponding to the user instruction to obtain a trained large language model.

[0095] It is understandable that the parameters of the large language model M to be trained can be repeatedly adjusted with the goal of reducing the loss L3 to train the large language model M. Then, when the training end condition is met, the training is terminated to obtain the trained large language model M1. The present application does not limit the content of the training end condition. For example, the training end condition may include that the updated loss L3 is less than a preset loss threshold, or that the number of training times reaches a preset training number threshold, etc.

[0096] The above training process is described by taking the example that the training sample set D includes the training samples corresponding to the user instruction q. However, it should be understood that the training sample set D may also include training samples corresponding to multiple different user instructions.

[0097] Based on the above content, it can be seen that in the above step 203, that is, based on the reference mark output by the large language model M The predicted probability p M , determine the reference mark In the process corresponding to the loss L1, each reference marker The importance of the reference tags is the same. Therefore, since the importance of the reference tags is not distinguished during the training of the large language model M, the accuracy of the trained large language model M1 may be low. At this time, when the trained large language model M1 outputs the tool name, the accuracy of the first tag in the output tool name and the next tag of the target tag is low, which will lead to the low accuracy of the large language model M1 in correctly selecting and using the tool, and then the low accuracy of the response result of the user instruction.

[0098] Based on this, in order to improve the accuracy of tools using large language models, this application sets different loss weights for the losses of reference tags of different tag categories (such as loss L1) in the process of calculating the loss of the model to distinguish the importance of each reference tag.

[0099] Specifically, refer to Figure 3 As shown, the training process of the first large language model (such as large language model M) to be trained in this application is described. Among them, Figure 3 The model training process shown is similar to Figure 2 The differences in the model training process shown are mainly: Figure 3 During the model training process shown, different loss weights are set for reference tags of different categories.

[0100] Specifically, Figure 3 The process shown includes the following steps.

[0101] 301: Obtain a first training sample.

[0102] The first training sample includes a user instruction and at least one tool name corresponding to the user instruction, wherein each tool name includes at least one reference mark, and each tool name indicates a reference tool. In some embodiments, the first training sample also includes at least one reference feedback information corresponding to the at least one tool name.

[0103] It can be understood that the first training sample is the same as the training sample corresponding to the user instruction q in the previous text, that is, the first training sample may belong to the training sample set D. The relevant content of the first training sample can be specifically referred to the previous description, which will not be repeated here.

[0104] 302: Input the user command into the first language model to be trained to obtain a prediction result of the user command.

[0105] In some embodiments, the prediction result of the user instruction may include: the predicted probability of each reference mark in each tool name.

[0106] Based on the content shown in the previous step 202, the tool names t i Corresponding reference signs The predicted probability p M It can be expressed as

[0107] 303: Calculate a first loss of each reference mark of a target tool name in at least one tool name according to the prediction result.

[0108] It can be understood that in the embodiment of the present application, at least one tool name can be directly used as the target tool name, or the wrong name in at least one target name can be removed to obtain the target tool name.

[0109] The following first describes the model training method of the present application by taking at least one tool name as the target tool name as an example. The situation of removing the wrong name in at least one target name to obtain the target tool name will be described in detail later and will not be repeated here.

[0110] The prediction results include the names of various tools i Corresponding reference signs The predicted probability p M For example, in this step, based on the predicted probability p M and reference mark Labels, calculate reference marks The corresponding first loss L1.

[0111] The method for determining the first loss L1 is the same as the method for determining the loss L1 in the previous step 203, and will not be repeated here.

[0112] 304: Calculate the loss weight of each reference mark of the target tool name according to the mark category of the reference mark, and the loss weight of reference marks of different mark categories is different.

[0113] In an embodiment of the present application, the marker category includes a first category and a second category, wherein the loss weight of the reference marker of the first category is greater than the loss weight of the reference marker of the second category.

[0114] In some embodiments, the first category is determined by: determining the mark category of the reference mark in the target tool name that meets a preset condition as the first category. The preset condition includes at least one of the following: the reference mark is the first mark in the target tool name; the reference mark is the mark after the target mark in the target tool name, and the target mark is the same mark in the target tool name at the same position of different tool names.

[0115] Taking the tool names get_1 and get_2 as an example, the reference marks "get", "_1" and "_2" can all correspond to the first category.

[0116] In some other embodiments, the second category is determined by: determining the mark category of the reference mark in the target tool name that does not meet the preset condition as the second category. That is, the reference mark of the second category can be a reference mark in the target tool name other than the reference mark of the first category.

[0117] It can be understood that the set of reference marks corresponding to the first category is K i For example, if the reference mark Belongs to the set K i , then the reference mark The loss weight Set to the first weight w1. If the reference mark Does not belong to the set K i , which is the reference mark If the mark category is the second category, the reference mark The loss weight Set as the second weight w2.

[0118] Therefore, the reference mark The corresponding loss weight It can be expressed as the following formula (8).

[0119]

[0120] The embodiment of the present application does not limit the values ​​of the first weight and the second weight. They can be set based on experience or flexibly adjusted based on actual application scenarios, as long as the first weight is greater than the second weight.

[0121] In some embodiments, the method for determining w1 includes: determining the ratio of the first number to the second number as w1, wherein the first number is a reference mark The number of reference marks belonging to the second category in the target tool name, the second number being the reference mark The number of reference marks belonging to the first category in the target tool name.

[0122] Taking i as 1 and l as 3 as an example, the tool name t1 includes the reference mark If the reference mark Belongs to the first category, reference mark Belonging to the second category, the first number is 3 and the second number is 1. At this time, w1 is equal to 3 / 1, that is, 3.

[0123] In some other embodiments, w2 may be a preset target value, such as 1.

[0124] 305: According to the loss weights of the reference marks in the target tool name, weighted calculation is performed on the first loss of each reference mark in the target tool name to obtain the target loss.

[0125] For example, based on the reference mark The loss weight For each reference mark in the name of each target tool The first loss L1 is weighted summed to obtain the name of each target tool t i The corresponding second loss is L4.

[0126] The second loss L4 can be expressed as the following formula (9).

[0127]

[0128] Then, the second losses L4 corresponding to the target tool names are summed up to obtain the target loss L5 of the target tool name in at least one tool name.

[0129] Among them, the target loss L5 can be expressed as the following formula (10).

[0130]

[0131] 306: Adjust the parameters of the first language model based on the target loss to obtain the second language model.

[0132] It can be understood that the principle of step 306 is the same as the principle of step 206 above, and will not be repeated here.

[0133] In the model training method provided in the present application, the loss weight of the reference markers of the first category can be set higher than the loss weight of the reference markers of the second category. Since the reference markers of the first category are markers that are prone to errors in the inference process of the first language model, the method provided in the present application can make the first language model pay more attention to the prediction accuracy of the reference markers that are prone to errors during the training process by distinguishing the importance of reference markers of different marker categories, thereby improving the accuracy of the second language model obtained by training.

[0134] It can be understood that the above description uses the example of directly taking at least one tool name as the target tool name to describe the model training method of the present application. The following description will use the example of removing the wrong name in at least one target name to obtain the target tool name to describe the model training method of the present application.

[0135] It can be understood that in this case, the number of target tool names may be the same as the number of at least one tool name, or may be less than the number of at least one tool name, and the embodiments of the present application do not limit this.

[0136] Specifically, the method for determining the target tool name includes (not shown in the figure):

[0137] S1: Determine a tool name in which reference feedback information and error feedback information match in at least one tool name as an error name.

[0138] In the embodiment of the present application, the error feedback information may be obtained in advance. The present application does not limit the method for determining the error feedback information. For example, the error feedback information corresponding to the selection and use of the tool based on other models may be used as the error feedback information in the present application.

[0139] The error feedback information and the error type of the corresponding error name may be shown in Table 1.

[0140] Table 1

[0141]

[0142]

[0143] Then, at least one tool name (such as t=(t0, t1, …, t s )) respectively corresponds to the reference feedback information (such as o=(o0,o1,…,o s )) is matched with the obtained error feedback information, and the tool name corresponding to the reference feedback information matching the error feedback information is used as the error name.

[0144] The present application does not limit the method for determining whether a certain reference feedback information matches the error feedback information. For example, if a certain reference feedback information is the same as the error feedback information, then the reference feedback information is determined to match the error feedback information. For another example, if the similarity between a certain reference feedback information and the error feedback information is greater than a similarity threshold, then the reference feedback information is determined to match the error feedback information.

[0145] For example, if a certain reference feedback information is “name 'xxx' is not defined” in Table 1, it can be determined that the tool name corresponding to the reference feedback information is an incorrect name.

[0146] S2: Remove an incorrect name from at least one tool name to obtain a target tool name.

[0147] In some embodiments, the set corresponding to the error name in the at least one tool name may be represented as T e , and T e ∈(t0,t1,…,t s ), that is, T e ∈t.

[0148] It can be understood that the set T of error names is determined e After that, we can use the tool name t to exclude the set of error names T. e The tool names other than , i.e., the target tool names, are used to train the first language model to be trained. In other words, the target tool names used for training are t i Does not belong to the set T with the wrong name e .

[0149] In this case, the target tool names are i The corresponding second loss L4 can be expressed as the following formula (11).

[0150]

[0151] The remaining training steps for the first language model are the same as above. Figure 3 The principles of the steps shown are the same and will not be described in detail.

[0152] The model training method provided in the present application trains the first large language model based on the tool name (i.e., the target tool name) with incorrect names removed. This can avoid the problem of low accuracy of the trained large language model caused by using incorrect tool names during the training of the first large language model.

[0153] It can be understood that the above training process of the first language model adopts the supervised fine-tuning approach (SFT), so the obtained second language model can be represented as M sft .

[0154] In some other embodiments, after obtaining the second largest language model M sft Afterwards, the second language model M can be optimized based on the reward function combined with a reinforcement learning algorithm (such as the proximal policy optimization (PPO) algorithm). sft Reinforcement learning is performed to adjust the second language model M sft The parameters of maximizing the reward to improve the target large language model M* accuracy.

[0155] Among them, Figure 4 As shown, for the second largest language model M sft The reinforcement learning is carried out to obtain the target large language model M * The process may include the following steps:

[0156] 401: Input the user instruction in the first training sample into the second largest language model to obtain the prediction tool name corresponding to the output user instruction.

[0157] Among them, the second largest language model M sft The output prediction tool name can be expressed as t′=(t′0, t′1,…, t′ s ).

[0158] 402: Call the prediction tool corresponding to the prediction tool name based on the second largest language model to obtain corresponding prediction feedback information.

[0159] Among them, the prediction feedback information corresponding to the prediction tool name t′ can be expressed as o′=(o′0, o′1,…, o′ s ).

[0160] 403: Determine a reward function for each prediction tool name corresponding to each prediction feedback information based on each prediction feedback information.

[0161] In some examples, the reward function R(t′ i ) can be expressed as the following formula (12).

[0162]

[0163] Based on the above formula (12), it can be seen that if the predicted feedback information o i The content of ′ indicates the name of the corresponding prediction tool t i ′ cannot be resolved, then the reward function R(t i ′) is assigned a value of -2. If the predicted feedback information is o i The content of ′ indicates the name of the corresponding prediction tool t i ′ includes tool illusions, such as predicting tool names t i ′ is not defined, then the reward function R(t i ′) is assigned the value -2.

[0164] If the predicted feedback information o i The content of ′ indicates the name of the prediction tool t i ′ is an incorrect tool name, such as predictive feedback information o i ′ and reference feedback information o i Different, then the reward function R(ti ′) is assigned a value of -1.5. For example, if the reference feedback information o i Involves weather data corresponding to weather query requests, but forecast feedback information i ′ involves the navigation route, then the name of the prediction tool t i 'mistake.

[0165] If the predicted feedback information o i The content of ′ indicates the name of the prediction tool t i ′ has content filling problems, such as predicting feedback information o i ′ and reference feedback information o i If the parameter values ​​are different, then the reward function R(t i ′) is assigned a value of -0.25. For example, if the reference feedback information o i The value corresponding to parameter A is a, but the predicted feedback information is o i ′, the value corresponding to parameter A is b, then the name of the prediction tool t can be determined i ' has content filling problem.

[0166] If the predicted feedback information o i ′Content and reference feedback information i The content is exactly the same, that is, the name of the prediction tool t i ′ is correct, then the reward function R(t i ′) is assigned the value 1.

[0167] If the predicted feedback information o i ′Content and reference feedback information i The parameters of the prediction tool are not exactly the same, that is, the name of the prediction tool t i ′ has parameter problems (e.g., parameter hallucination, redundant parameters, missing parameters, etc.), then the reward function R(t i ′) Assign R p (t i ′).

[0168] Among them, R p (t i ′) can be expressed as the following formula (13).

[0169]

[0170] In the above formula (13), (t i ′ corresponds to the parameter illusion) indicates that if t i ′ corresponds to parameter hallucination, then (t i ′ corresponds to the parameter illusion) is assigned a value of 1, if t i′ does not correspond to the parameter illusion, it is assigned a value of 0. The rest of the principles are the same and will not be described in detail.

[0171] Specifically, if the predicted feedback information o i ′ indicates that a parameter is not defined, then t i ′ corresponds to parameter hallucination.

[0172] If the predicted feedback information o i ′Content and reference feedback information i Compared with the content of i ′ corresponds to redundant parameters. For example, if the reference feedback information o i Including parameter A, parameter B, but prediction feedback information o i ' includes parameters A, B and C, then determine t i ′ corresponds to redundant parameters.

[0173] If the predicted feedback information o i ′Content and reference feedback information i Compared with the missing parameters, it can be determined that t i 'The corresponding parameter is missing. For example, if the reference feedback information o i Including parameters A, B, C, but the prediction feedback information o i ′ only includes parameters A and B, then determine t i 'The corresponding parameter is missing.

[0174] In some embodiments, if t i ′ corresponds to parameter hallucination and redundant parameters, but not to parameter missing, so R in the above formula (13) p (t i ′) is equal to -0.8×1-0.5×1-0.5×0, which is equal to -1.3.

[0175] 404: Adjust the parameters of the second largest language model based on the reward function to obtain a target large language model with updated parameters.

[0176] It can be understood that in the process of adjusting the parameters of the second largest language model, multiple intermediate models can be obtained. The following takes the intermediate model M′ as an example to adjust the parameters of the target large language model M * Describe how to determine .

[0177] For example, the intermediate model M′ can be combined with the second largest language model M sft The parameter difference is defined as the first difference F. The first difference F can be expressed as the following formula (14).

[0178] F=βKL(M′(·)‖M sft(·)) Formula (14)

[0179] In the above formula (14), β is a hyperparameter that can be set based on experience; KL(M′(·)‖M sft (·)) is used to quantify the intermediate model M′ and the second largest language model M sft is a function of the difference in parameters.

[0180] In this application, each R(t i ′) and the sum of the differences between the first and second differences (hereinafter referred to as the first sum) as the target, adjust the second largest language model M sft Parameters of the target large language model M * .

[0181] For example, the second largest language model M can be calculated based on the following formula (15): sft Conduct training.

[0182]

[0183] It can be understood that in the above training method, when the first sum is the largest, the intermediate model M′ corresponding to the first sum is used as the trained target large language model M * That is, the target large language model M * The corresponding R(t i ′) and the sum of the differences between the first difference and the second difference (ie, the first sum) is the largest.

[0184] The model training method provided in this application performs reinforcement learning on the second largest language model based on the reward function to obtain the target large language model. This method can further improve the accuracy of the response results of the target large language model to user instructions.

[0185] The following describes the performance of the target large language model obtained based on the training method provided by the present application in the process of executing user instructions. It can be understood that the embodiment of the present application reflects the effect of the target large language model of the present application by comparing the performance of the target large language model involved in the present application and the large language model obtained without using the method provided by the present application (hereinafter collectively referred to as the control group model) on different test sets.

[0186] The following describes the test set used, the control group model and the target large language model, and the test process and results.

[0187] (I) Test set

[0188] This application does not limit the number of test sets, the source of acquisition, and the type of test sets. It can be determined based on experience or flexibly adjusted according to the actual application scenario. In the embodiments of this application, four open source test sets are used, such as ToolAlpaca, RoTBench, BFCL-v3, and ToolEyes. Among them, the test sets ToolAlpaca, RoTBench, and BFCL-v3 are used for single-round tool selection scenarios, and the test set ToolEyes is used for multi-round tool selection scenarios.

[0189] Based on the above, the execution process of a user instruction may correspond to at least one processing step, and each processing step may use different tools. In other words, during the execution process of a user instruction, the large language model may determine at least one prediction tool.

[0190] It can be understood that the large language model only needs to determine one prediction tool, that is, the large language model can realize the execution of the user instruction based on only one prediction tool, which is a single-round tool selection scenario. Therefore, each user instruction in the test sets ToolAlpaca, RoTBench, and BFCL-v3 corresponds to only one reference tool.

[0191] Similarly, a large language model needs to determine multiple prediction tools, that is, a large language model needs to be based on multiple prediction tools to implement the scenario of user command execution, which is a multi-round tool selection scenario. Therefore, each user command in the test set ToolEyes corresponds to multiple reference tools.

[0192] (II) Control group model and target large language model

[0193] 1. Control group model

[0194] This application does not limit the number of control group models, the source of acquisition, and the type of models, which can be determined based on experience or flexibly adjusted based on actual application scenarios. In the embodiments of this application, seven control group models are used, including tool-based models, open source models, and closed source models.

[0195] In some embodiments, the tool usage models used in this application include ToolLLaMA-2-7B and NexusRaven-2-13B. These two models are trained on the basis of a large amount of tool usage data, thereby enhancing the ability to interact with tools. For example, ToolLLaMA-2-7B is trained on more than 120,000 data covering more than 16,000 tools, and has a high tool prediction ability.

[0196] In some embodiments, the open source models used in this application include ChatGLM-4-chat-9B and Qwen-2-Instruct-7B.

[0197] In some other embodiments, the closed source models used in this application include GPT-3.5-turbo, GPT-4o and GPT-4-turbo. It can be understood that the generative pre-trained transformer (GPT) series represents the most advanced large language model, which not only performs well in general tasks, but also performs well in tool prediction, with significant generalization ability.

[0198] 2. Target Large Language Model

[0199] It can be understood that the target large language model obtained based on the model training method provided by this application can be defined as TL-CodeLLaMA-2, which is obtained by training with a custom training set based on the existing CodeLLaMA-2-7B model. Compared with the above seven control group models, the TL-CodeLLaMA-2 of this application is the smallest model, and the amount of data used in the training set during the training process is also the least.

[0200] In some embodiments, the user instructions in the training set can be artificially constructed based on the test set ToolEyes. Then, the user instructions in the training set can be input into the GPT-4o model, and the prediction tools corresponding to each user instruction output by the GPT-4o model are used as reference tools. It can be understood that since the GPT-4o model has a high accuracy in tool prediction, the prediction tool output by the GPT-4o model is used as a reference tool to train the large language model to be trained in this application, which can improve the accuracy of tool prediction of the large language model TL-CodeLLaMA-2 obtained in this application.

[0201] In some embodiments, in the SFT stage, an AdamW optimizer with cosine scheduling can be used. The large oracle model to be trained is trained, wherein the learning rate can be set to 1e-6, the warm-up rate is set to 0.01, the batch size is 4, and a total of 1 round of training, i.e., one epoch. In addition, the upper limit value of w1 mentioned above is set to 9.

[0202] In the reinforcement learning (RL) stage, 1194 training data can be selected from 1217 training data, and the large language model can be trained based on the reward function. The actor learning rate can be set to 2e-6, the critic learning rate can be set to 1e-6, the batch size is 8, and a total of 3 rounds of training, i.e. 3 epochs, are performed.

[0203] Table 2 shows information about a training set and a test set. As shown in Table 2, in the training process of the target large language model TL-CodeLLaMA-2 of the present application, a custom training set is used for training, and the training set includes 1217 user instructions for training, that is, 1217 training data. In addition, the custom training set is used for multiple rounds of tool selection scenarios.

[0204] Table 2

[0205]

[0206] Based on the above, in the test process of the target large language model TL-CodeLLaMA-2 of this application, four open source test sets are used, namely ToolAlpaca, RoTBench, BFCL-v3 and ToolEyes. Among them, the test set ToolAlpaca includes 114 test data, RoTBench includes 105 test data, BFCL-v3 includes 239 test data, and ToolEyes includes 382 test data. In addition, the test sets ToolAlpaca, RoTBench and BFCL-v3 are used for single-round tool selection scenarios, and the test set ToolEyes is used for multi-round tool selection scenarios.

[0207] (III) Testing process and results

[0208] 1. Testing process

[0209] During testing, the official tool call hint templates can be used and greedy search can be applied during inference (or tool selection) to maximize the tool selection ability for each model.

[0210] It can be understood that in order to compare the tool selection capabilities of the TL-CodeLLaMA-2 model of the present application with the seven control group models, the indicators for comparison can be preset. In addition, the indicators of the test sets ToolAlpaca, RoTBench and BFCL-v3 corresponding to the single-round tool selection scenario (hereinafter referred to as single-round indicators) are different from the indicators of the test set ToolEyes corresponding to the multi-round tool selection scenario (hereinafter referred to as multi-round indicators).

[0211] It can be understood that since the test sets ToolAlpaca, RoTBench and BFCL-v3 correspond to single-round tool selection scenarios, the above test sets can provide standardized interaction paths (i.e., the reference tools corresponding to each step). For example, for a certain user instruction, the tool corresponding to the user instruction in the test set is the only reference tool.

[0212] However, since the test set ToolEyes corresponds to multiple rounds of tool selection scenarios, the test set can only provide reference tools but cannot provide standardized interaction paths. For example, for a certain user instruction, the user instruction in the test set can correspond to multiple steps and multiple reference tools, but the selection order between multiple reference tools in the test machine is not unique. For example, a large language model may be able to accurately execute user instructions without selecting a reference tool based on the order of reference tools in the test set. Therefore, single-round indicators and multi-round indicators are different.

[0213] In some embodiments, the single-round indicators may include tool selection (TS), parameter identification (PI), and content filling (CF). Among them, the TS indicator is used to measure the accuracy of the model in the selected tool; the PI indicator is used to measure the accuracy of the relevant parameters required for the model to call; the CF indicator is used to measure the ability of the model to call the tool, for example, including the accuracy of the tool selected by the model, the accuracy of the relevant parameters called, and the accuracy of the values ​​of the filled parameters.

[0214] In some embodiments, multi-round indicators may include documentation understanding error (DE), tool call error (CE), and valid answers (VA). The DE indicator represents the percentage of errors caused by the model's failure to correctly interpret the tool documentation, including tool illusion, parameter illusion, and lack of necessary parameters. The CE indicator represents the proportion of errors caused by incorrect tool calls. The VA indicator represents the percentage of instances in which the model provides valid responses within nine rounds of dialogue, reflecting its ability to effectively meet needs.

[0215] 2. Test results

[0216] Table 3 shows the performance of each model (including the control group model and the target large language model) on the test sets ToolAlpaca, RoTBench and BFCL-v3 corresponding to the single-round tool selection scenario.

[0217] Table 3

[0218]

[0219]

[0220] As shown in Table 3, when the TL-CodeLLaMA-2 model is tested on the ToolAlpaca test set, the TS index is 87.72%, the PI index is 78.07%, and the CF index is 57.89%. The size of each index of each model on each test set is determined in the same way, so it will not be repeated one by one.

[0221] Based on Table 3, the TL-CodeLLaMA-2 model outperforms other control group models in terms of overall task completion (i.e., CF indicator). It is worth noting that on the ToolAlpaca and BFCL-v3 test sets, the TL-CodeLLaMA-2 model surpasses the best results of the GPT series (such as GPT-4-turbo) by 15.78% and 12.08%, respectively. In addition, on the ToolAlpaca test set, the TL-CodeLLaMA-2 model outperforms other control group models in all three indicators, while on the RoTBench test set, the TS indicator of the TL-CodeLLaMA-2 model is 9.52% higher than that of the Qwen-2-Instruct-7B model and 0.95% higher than that of GPT-4-turbo.

[0222] Table 4 shows the performance of each model (including the control group model and the target large language model) on the test set ToolEyes corresponding to the multi-round tool selection scenario.

[0223] Table 4

[0224]

[0225] As shown in Table 4, when the TL-CodeLLaMA-2 model is tested on the ToolEyes test set, the DE index is 0.82% and the CE index is 4.84%. The determination method of the size of each index of each model on the ToolEyes test set is similar, so it will not be repeated one by one.

[0226] Based on Table 4, we can see that in terms of VA indicators, the TL-CodeLLaMA-2 model outperforms ToolLLaMA-2-7B and ChatGLM-4-chat-9B. In terms of CE indicators, the performance of the TL-CodeLLaMA-2 model is second only to GPT-4o.

[0227] In some other embodiments, the present application also specifically analyzes the effects of the three methods used in the training process of the large language model. Figure 3The method of training a large language model by distinguishing the importance of reference tokens as shown can be called a prioritizing key tokens (PKT) method. The method of obtaining a target tool name by removing an incorrect name from at least one tool name as mentioned above can be called a mitigating adverse effects (MAE) method. Figure 4 The method of performing reinforcement learning on the second language model shown can be called introduction of a reward mechanism (IRM).

[0228] Table 5 shows the performance of a large language model obtained based on different training methods on different test sets. As shown in Table 5, the TS index of the model trained based on the standard SFT stage (i.e., without undergoing any of the three methods involved in this application) on the ToolAlpaca test set is 74.56%. The methods for determining the size of each index of the model trained based on each method on the ToolEyes, RoTBench and ToolEyes test sets are similar, so they will not be repeated one by one.

[0229] Table 5

[0230]

[0231] Based on Table 5, it can be seen that compared with the model trained based on the standard SFT stage, if the model adopts the MAE method during the training process, the VA index of the large language model of the present application on the ToolEyes test set is increased by 9.53%. This shows that removing the wrong path in the training sample can prevent the large language model from learning incorrect interaction paths, thereby improving the accuracy of the trained large language model on the test set corresponding to the multi-round selection scenario.

[0232] In addition, if the model adopts the PKT method during the training process, the ability of the large language model trained by the present application to distinguish similar tools (such as tool names) is enhanced, thereby improving the accuracy of tool selection and improving overall performance. In addition, if the model adopts the IRM method during the training process, the performance of the large language model trained by the present application is improved at all stages.

[0233] Continuing to refer to Table 5, MAE, PKT and IRM were used in the training process. The obtained TL-CodeLLaMA-2 model has comprehensive advantages. Only through training with 1217 user instructions, the selection performance of the model in single-round tool selection scenarios and multi-round tool selection scenarios can be significantly improved, and the tool selection accuracy can be improved, thereby improving the execution efficiency of user instructions.

[0234] In some other embodiments, the present application also experiments on the robustness of the TL-CodeLLaMA-2 model. It can be understood that during the execution of user instructions, the tool set (such as the tool set T mentioned above) often contains various types of noise (such as missing description information corresponding to a tool, etc.). Therefore, the TL-CodeLLaMA-2 model needs to be robust when selecting tools in order to accurately execute user instructions in different situations.

[0235] Figure 5 The performance of a TL-CodeLLaMA-2 model and a RoTLLaMA model in five different noise environments is shown. Figure 5 As shown in the figure, noise can be divided into no noise, mild noise, moderate noise, severe noise and very severe noise. It can be understood that in any noise environment, the TS, PI and CF indicators of the TL-CodeLLaMA-2 model are higher than the corresponding indicators of the RoTLLaMA model.

[0236] This shows that the method provided in this application can make the trained target large language model free from noise interference, so that it can better adapt to various tool selection scenarios and better execute user instructions.

[0237] In some other embodiments, the present application also evaluates the generalization ability of the TL-CodeLLaMA-2 model. Figure 6 As shown, this application evaluates the generalization ability of the TL-CodeLLaMA-2 model on three common test sets (i.e., MMLU, GSM8K, and HumanEval) and compares it with the CodeLLaMA-2 model. Figure 6 As shown in the figure, on any test set, the generalization ability of the TL-CodeLLaMA-2 model reaches or exceeds the generalization ability of the CodeLLaMA-2 model. In other words, the model training method provided in this application can make the trained target large language model have stronger generalization ability.

[0238] In some embodiments, the embodiments of the present application also provide a readable storage medium, on which instructions are stored, and when the instructions are executed on an electronic device, the electronic device executes the model training method described in the above embodiments.

[0239] In some embodiments, the embodiments of the present application also provide an electronic device, which includes: one or more processors; one or more memories; one or more memories storing one or more programs, and when one or more programs are executed by one or more processors, the electronic device executes the model training method described in the above embodiments.

[0240] In some embodiments, the embodiments of the present application also provide a computer program product, including: a computer program or instructions, when the computer program or instructions are run on an electronic device, the electronic device executes the model training method described in the above embodiments.

[0241] Figure 7 FIG. 1 is a schematic diagram showing the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0242] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0243] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0244] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0245] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it may be directly called from the above-mentioned memory. Repeated access is avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved. The processor may be used to execute the model training method mentioned in the present application.

[0246] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.

[0247] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0248] The SIM card interface 195 is used to connect a SIM card.

[0249] It will be understood that, as used herein, the term "module" may refer to or include an application-specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and / or memory that executes one or more software or firmware programs, a combinational logic circuit, and / or other appropriate hardware components that provide the described functionality, or may be part of these hardware components.

[0250] It is understood that in each embodiment of the present application, the processor may be a microprocessor, a digital signal processor, a microcontroller, etc., and / or any combination thereof. According to another aspect, the processor may be a single-core processor, a multi-core processor, etc., and / or any combination thereof.

[0251] The various embodiments disclosed in the present application may be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application may be implemented as a computer program or program code executed on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0252] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit, or a microprocessor.

[0253] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0254] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-read-only memories, CD-ROMs), magneto-optical disks, read-only memories (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signal digital signals, etc.) using the Internet in electrical, optical, acoustic or other forms of propagation signals. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0255] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0256] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0257] It should be noted that, in the examples and description of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the statement "comprise one" do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.

[0258] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present application.

Claims

1. A model training method, characterized in that: Applied to electronic equipment, the method comprises: Acquire a first training sample, wherein the first training sample includes a user instruction and at least one tool name corresponding to the user instruction, wherein each tool name includes at least one reference mark, and each tool name indicates a reference tool; Inputting the user instruction into a first language model to be trained to obtain a prediction result of the user instruction; Calculate a first loss of each reference mark of a target tool name in the at least one tool name according to the prediction result; Calculating the loss weight of each reference mark of the target tool name according to the mark category of the reference mark, where the loss weights of reference marks of different mark categories are different; According to the loss weights of the reference marks of the target tool name, weighted calculation is performed on the first loss of each reference mark in the target tool name to obtain a target loss; Parameters of the first largest language model are adjusted based on the target loss to obtain a second largest language model.

2. The method according to claim 1, characterized in that The marker categories include a first category and a second category, and the loss weight of the reference markers of the first category is greater than the loss weight of the reference markers of the second category.

3. The method according to claim 2, characterized in that The method further comprises: Determine the mark category of the reference mark that meets the preset condition in the target tool name as the first category; Determining the mark category of the reference mark in the target tool name that does not meet the preset condition as the second category; Among them, the preset conditions include at least one of the following: the reference mark is the first mark in the target tool name; the reference mark is the mark after the target mark in the target tool name, and the target mark is the same mark in the target tool name at the same position of different tool names.

4. The method according to claim 2 or 3, characterized in that: The step of calculating the loss weight of each reference mark of the target tool name according to the mark category of the reference mark includes: The ratio of the first quantity to the second quantity is determined as the loss weight of the reference mark of the first category in the target tool name, wherein the first quantity is the number of reference marks belonging to the second category in the target tool name where the reference mark is located, and the second quantity is the number of reference marks belonging to the first category in the target tool name where the reference mark is located.

5. The method according to claim 2 or 3, characterized in that: The step of calculating the loss weight of each reference mark of the target tool name according to the mark category of the reference mark includes: A target value is determined as a loss weight of the reference mark of the second category in the target tool name.

6. The method according to claim 1, characterized in that The calculating, according to the prediction result, a first loss of each reference mark of a target tool name in the at least one tool name comprises: Determining the target tool name from the at least one tool name, wherein the number of the target tool names is one or more; A first loss of each reference mark of the target tool name is calculated according to the prediction result.

7. The method according to claim 6, characterized in that The first training sample further includes reference feedback information of each tool name in the at least one tool name, the reference feedback information includes call result information after the reference tool indicated by the corresponding tool name is called, and, Determining the target tool name from the at least one tool name includes: Determine a tool name in the at least one tool name that matches the reference feedback information with the error feedback information as an error name; The wrong name is removed from the at least one tool name to obtain the target tool name.

8. The method according to claim 1, characterized in that The number of the target tool names is multiple, and the first loss of each reference mark in the target tool name is weightedly calculated according to the loss weight of each reference mark in the target tool name to obtain the target loss, including: According to the loss weights of the reference marks of the target tool names, weighted summation is performed on the first losses of the reference marks in the target tool names to obtain the second losses corresponding to the target tool names; The second losses corresponding to the names of the target tools are summed up to obtain the target loss.

9. The method according to claim 1, characterized in that: The prediction result of the user instruction includes the prediction probability of each reference mark in the tool name.

10. The method according to claim 1, characterized in that The method further comprises: Based on the first training sample, a proximal strategy optimization algorithm is used to adjust the parameters of the second large language model to obtain a target large language model.

11. An electronic device, characterized in that: include: one or more processors; One or more memories; the one or more memories store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device executes the model training method described in any one of claims 1 to 10.

12. A readable storage medium, characterized in that: The readable storage medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the model training method described in any one of claims 1 to 10.

13. A computer program product, characterized in that include: A computer program or instruction, which, when executed on an electronic device, enables the electronic device to execute the model training method described in any one of claims 1 to 10.

Images