Language model optimization, processing method, device, medium and program product

Abstract

The embodiments of the present application provide a method, device, medium and program product for optimizing and processing a language model. The method includes: obtaining path data of a calling tool corresponding to the language model, constructing a preference data set based on the path data, wherein the path data includes a success path and a failure path; obtaining input data based on the preference data set, inputting the input data into the language model, and obtaining a corresponding output result; determining a loss function based on the output result and a reward model; and adjusting the parameters of the language model based on the loss function to obtain an optimized language model. By introducing failed decision information, the preference data set can be enriched, data utilization can be improved, and the language model can be optimized accordingly to improve the accuracy of the language model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a language model optimization method, a language model-based processing method, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Traditional Large Language Models (LLMs) have demonstrated impressive capabilities in various areas such as language understanding, generation, interaction, and reasoning. Despite these significant advantages, these models still face significant challenges, such as the inability to obtain real-time information and difficulty in completing precise mathematical tasks. The development of tool-enhanced LLMs has greatly improved the performance of LLMs by facilitating effective interaction with external tools. This advancement enables them to act as an effective intermediary between users and a large ecosystem of applications. Tool-enhanced LLMs have been widely used in a variety of applications—including web browsing, multimodal reasoning, etc., and have achieved outstanding results.

[0003] Currently, most of the work on combining open source LLMs with downstream tasks used by tools relies on supervised fine-tuning using expert trajectory datasets, which trains LLMs to learn policies for subsequent operations based on observations and previous operations. In this research area, early studies often have some limitations, such as limited tool types, reliance on single tool scenarios, and the use of basic reasoning methods. Recently, a study proposed an instruction tuning dataset, which contains more than 16,000 real-world APIs and various real-world instructions, as well as expert annotations created based on depth-first search decision trees. In addition, they also used this dataset to perform supervised fine-tuning. However, the model still heavily relies on fine-tuning using expert trajectories, which can lead to insufficient exploration of the environment and suboptimal strategies. In addition, in the depth-first search decision, supervised fine-tuning only uses successful answers, ignoring the valuable information in failed attempts, resulting in low data utilization. Summary of the invention

[0004] An embodiment of the present application provides a method for optimizing a language model to improve data utilization.

[0005] Correspondingly, an embodiment of the present application also provides a processing method based on a language model, an electronic device, a computer-readable storage medium and a computer program product to ensure the implementation and application of the above system.

[0006] In order to solve the above problems, the embodiment of the present application discloses a method for optimizing a language model, wherein the method comprises:

[0007] Acquire path data of a calling tool corresponding to a language model, and construct a preference data set based on the path data, wherein the path data includes a successful path and a failed path;

[0008] Acquire input data based on the preference data set, input the input data into the language model, and obtain a corresponding output result;

[0009] Determine a loss function based on the output result and the reward model;

[0010] The parameters of the language model are adjusted based on the loss function to obtain an optimized language model.

[0011] Optionally, obtaining the path data of the calling tool corresponding to the language model includes:

[0012] Determine the path of the language model calling tool through the decision tree and obtain the corresponding success path and failure path;

[0013] constructing a path preference pair based on the success path and the failure path;

[0014] A preference dataset is constructed based on the path preference pairs.

[0015] Optionally, obtaining the path data of the calling tool corresponding to the language model includes:

[0016] Determine the successful path of the language model calling tool through a decision tree;

[0017] Constructing a corresponding failure path based on the success path and the decision tree, wherein at least a tail node of the constructed failure path is different from a tail node of the corresponding success path;

[0018] taking the successful path and the corresponding failed path as a path preference pair;

[0019] A preference dataset is constructed based on the path preference pairs.

[0020] Optionally, constructing a corresponding failure path based on the success path and the decision tree includes:

[0021] Traversing the successful path in the decision tree upwards from a leaf node;

[0022] For a node with a branch, another node of the branch and a path on the node of the branch are used to form a failure path, thereby obtaining at least one failure path.

[0023] Optionally, acquiring input data based on the preference data set includes:

[0024] Based on the preference data set, determining the instruction words corresponding to the path preference pair;

[0025] Corresponding input data is constructed based on the instruction words.

[0026] Optionally, the step of constructing the reward model is also included:

[0027] A preferred reward model is constructed based on the comparison model, and parameters of the reward model are estimated based on a setting algorithm.

[0028] Optionally, also include:

[0029] A preference optimization algorithm is determined, and the reward model is optimized based on the preference optimization algorithm, wherein the optimization goal of the preference optimization algorithm is to maximize the reward.

[0030] Optionally, also include:

[0031] A reinforcement learning algorithm is determined, and the reward model is optimized based on the reinforcement learning algorithm, wherein the optimization goal of the reinforcement learning algorithm is to maximize the reward.

[0032] Optionally, also include:

[0033] The language model is fine-tuned based on the successful path as training data.

[0034] The embodiment of the present application also discloses a processing method based on a language model, wherein the method comprises:

[0035] Receive service demand information obtained through the customer service page;

[0036] Analyze the service demand information through a language model to determine a tool path of a required service tool, wherein a loss function in a language model training process is determined based on a reward model, the training data of the language model is determined based on a preference data set, the preference data set is determined based on path data of a calling tool corresponding to the language model, and the path data includes a success path and a failure path;

[0037] Acquire corresponding service information based on the tool path;

[0038] generating a service demand result based on the service information;

[0039] The service demand result is sent to feedback the service demand result through the customer service page.

[0040] Optionally, the service tool includes at least one of the following tools: a logistics query tool, an order query tool, and a product query tool.

[0041] The embodiment of the present application also discloses a processing method based on a language model, wherein the method comprises:

[0042] Receive resource withdrawal information through the customer service page;

[0043] Analyzing the resource withdrawal information through a language model to determine a tool path of a service tool of the resource withdrawal corresponding to the associated information, wherein a loss function in the language model training process is determined based on a reward model, the training data of the language model is determined based on a preference data set, the preference data set is determined based on path data of a calling tool corresponding to the language model, and the path data includes a success path and a failure path;

[0044] Acquire corresponding association information based on the tool path;

[0045] Determine a resource withdrawal result based on the association information;

[0046] The resource withdrawal result is sent to feedback the resource withdrawal result through the customer service page.

[0047] Optionally, the service tool includes at least one of the following tools: a logistics query tool, an order query tool, a product query tool, a service information query tool, and a credit query tool.

[0048] The embodiment of the present application also discloses an electronic device, which includes: a processor, and a memory communicatively connected to the processor;

[0049] The memory stores computer-executable instructions;

[0050] The processor executes the computer-executable instructions stored in the memory to implement the method described in the embodiment of the present application.

[0051] The embodiment of the present application further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the embodiment of the present application.

[0052] The embodiment of the present application further discloses a computer program product, including a computer program / computer executable instructions, wherein the computer program / computer executable instructions, when executed by a processor in an electronic device, implement the method described in the embodiment of the present application.

[0053] Compared with the prior art, the embodiments of the present application have the following advantages:

[0054] In an embodiment of the present application, path data of a calling tool corresponding to a language model is obtained, and a preference data set is constructed based on the path data. The path data includes a success path and a failure path, so that the preference data set can be enriched by introducing failure decision information, thereby improving data utilization. Input data is then obtained based on the preference data set, and the input data is input into the language model to obtain a corresponding output result. A loss function is determined based on the output result and a reward model, and the parameters of the language model are adjusted based on the loss function to obtain an optimized language model, thereby improving the accuracy of the language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flowchart of the steps of an embodiment of a method for optimizing a language model of the present application;

[0056] Figure 2 is a schematic diagram of an example of a decision tree in an embodiment of the present application;

[0057] Figure 3 It is a flowchart of the steps of an embodiment of a method for constructing a reward model of the present application;

[0058] Figure 4 is a flowchart of another embodiment of a method for constructing a reward model of the present application;

[0059] Figure 5 It is a flowchart of the steps of an optional embodiment of a method for optimizing a language model of the present application;

[0060] Figure 6 is a flowchart of steps of another optional embodiment of a method for optimizing a language model of the present application;

[0061] Figure 7 is a flowchart of another embodiment of a method for optimizing a language model of the present application;

[0062] Figure 8 is a flowchart of a processing method embodiment based on a language model of the present application;

[0063] Fig. 9 is a flowchart of another embodiment of a processing method based on a language model of the present application;

[0064] Fig.10 It is a schematic diagram of the structure of an exemplary device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0066] The embodiment of the present application can be applied to the tool learning scenario of a large language model. By constructing a preference data set including success paths and failure paths, the failure trajectory information can be used to enhance the tool learning process. The loss function is calculated in combination with the reward model, thereby improving the utilization rate of expert data and broadening the learning space.

[0067] Reference Figure 1 , showing a step flow chart of an embodiment of a language model training method of the present application.

[0068] Step 102, obtaining path data of a calling tool corresponding to a language model, and constructing a preference data set based on the path data, wherein the path data includes a success path and a failure path.

[0069] In the embodiment of the present application, the language model may be a large language model LLM, which is a basic model with high application value in many fields such as natural language processing, computer vision, biology, etc. In specific fields, the LLM model can be processed by calling tools, such as calling professional tools in the e-commerce field, the financial field, etc. to solve corresponding problems and improve the processing effect of the LLM model.

[0070] The language model calling tool can usually be called through an API (Application Programming Interface). The corresponding call to the tool can form a call path. The call to the tool by the language model can be trained based on the path data of the tool interface call. The path data corresponding to the language model calling tool can be determined, and the path data includes a successful path and a failed path. The successful path refers to the path track of the successful call of the tool, that is, the result obtained based on the call data of the path is successful, such as an answer that the user is satisfied with. The failed path refers to the path track of the failed call of the tool, that is, the result obtained based on the call data of the path is a failure, such as an answer that the user is not satisfied with, no answer, etc.

[0071] In an embodiment of the present application, the call to the tool interface can constitute a decision tree, and the decision tree is constructed based on the dependency relationship between the tool interfaces, so that the path data for the tool interface call is determined based on the decision tree. In some optional embodiments, the expert trajectory can be determined in advance, and the expert trajectory is the path trajectory of the tool interface call determined based on the expert data. An expert trajectory tree is constructed based on the expert trajectory, and the expert trajectory tree is used as a decision tree. Among them, the expert trajectory data set is a data set for training the model. On this basis, the embodiment of the present application will expand the expert trajectory data to improve data utilization. In an embodiment of the present application, the decision tree can be a depth-first search decision tree, or it can be other decision links, such as a breadth-first search tree, a decision graph, etc.

[0072] The root node of the decision tree is the instruction for calling the interface, and the leaf nodes are the results of the call, such as success or failure. Therefore, the corresponding path can be obtained by traversing the decision tree, and then the preference path pair can be constructed based on the path to obtain the preference data set. The input data such as keywords, texts, and instruction words corresponding to the path can also be determined and added to the preference data set as the association information of the path. The preference path pair is determined for the input of the same instruction.

[0073] In an optional embodiment, the path data of the calling tool corresponding to the language model is obtained, including: determining the path of the language model calling tool through a decision tree, obtaining the corresponding successful path and failed path; constructing a path preference pair based on the successful path and failed path; and constructing a preference data set based on the path preference pair. The decision tree is traversed to determine the path of the language model calling tool, and when the leaf node is traversed, it can be determined whether the path is a successful path or a failed path. A path preference pair is constructed based on the successful path and failed path, and the preference data set is constructed using the path preference pair.

[0074] like Figure 2 In the decision tree shown, the corresponding path can be obtained by traversing the decision tree, where ⟨0,9, 12, 13, 14, 15> is the successful path of the decision tree, and ⟨0, 1, 2>, ⟨0, 3, 4, 5, 6>, ⟨0, 3, 7, 8>, ⟨0, 9, 10, 11> are four failed paths. The successful path and the failed path are used to form a path preference pair, where the user's preference information for the successful path is greater than the preference information for the failed path. The path preference pair is used to construct a preference data set. For example, the successful path and each failed path constitute a path preference pair, namely: “⟨0, 9, 12, 13,14, 15>>⟨0, 1, 2>”, “⟨0, 9, 12, 13, 14, 15>>⟨0, 3, 4, 5, 6>”, “⟨0, 9, 12, 13,14, 15>>⟨0, 3, 7, 8>”, and “⟨0, 9, 12, 13, 14, 15>>⟨0, 9, 10, 11>”, where “>” indicates large preference information.

[0075] This enables quick acquisition of preferred path pairs, and a model trained on this data can distinguish between correct and incorrect responses to specific instructions.

[0076] In another optional embodiment, the path data of the calling tool corresponding to the language model is obtained, including: determining the successful path of the language model calling tool through a decision tree; constructing a corresponding failure path based on the successful path, wherein at least the tail node of the constructed failure path is different from the tail node of the corresponding successful path; taking the successful path and the corresponding failure path as a path preference pair; and constructing a preference data set based on the path preference pair.

[0077] After determining the successful path of the language model calling tool through the decision tree, the failure path corresponding to the successful path can also be constructed, so that the successful path and the corresponding failure path are used to form a path preference pair and added to the preference data set. In an optional embodiment, part or all of the successful path can be used to construct the failure path. When constructing the failure path, the tail node of the successful path is replaced with other nodes, wherein the node of another branch corresponding to the second to last node in the successful path is searched in the decision tree to obtain the corresponding failure node. Thus, a preference pair is constructed using each branch node on the successful path in the decision tree and its corresponding pair of child nodes (which must include the child nodes on the successful path).

[0078] In an optional embodiment, the successful path in the decision tree is traversed upward from the leaf node; for a node with a branch, another node of the branch and the path above the node of the branch form a failed path to obtain at least one failed path. Then, the successful path and the failed path whose leaf nodes belong to the same branch node form a preferred path pair.

[0079] like Figure 2 In the decision tree, the complete success path is ⟨0, 9, 12, 13, 14, 15>. Traversing upward along the decision tree, each time a node with a branch is encountered, the node on the other side of the branch and the branch and its upper nodes can be used to form a failure path. The corresponding failure paths are ⟨0, 9, 10>, ⟨0, 3>, and ⟨0, 1>. Then, the success path and the failure path whose leaf nodes belong to the same branch node form a preference path pair, and the preference path pairs include: "⟨0, 9, 12>>⟨0, 9, 10>", "⟨0, 9>>⟨0, 3>", and "⟨0, 9>>⟨0, 1>".

[0080] Since the above method can more clearly understand the differences between each reasoning step, the fine-grained reasoning ability of the model is enhanced.

[0081] In the embodiment of the present application, any one of the above methods can be used to generate path preference pairs and construct a preference data set, or the two methods can be combined to generate path preference pairs and construct a preference data set.

[0082] Step 104: acquiring input data based on the preference data set, inputting the input data into the language model, and acquiring a corresponding output result.

[0083] After the construction of the preference data set is completed, the language model can be trained using the preference data set as training data, and input data can be obtained from the preference data set, wherein the input data can be constructed using the instruction words, keywords, etc. corresponding to the path, and the input data is input into the language model for processing, and the language model outputs the corresponding output result. The language model can call the tool interface when processing the input data, thereby obtaining the calling path of the corresponding model.

[0084] In an optional embodiment, the step of obtaining input data based on the preference data set includes: determining, based on the preference data set, an instruction word corresponding to a path preference pair; and constructing input data based on the instruction word, wherein the instruction word has a preference information for a successful path greater than a preference information for an unsuccessful path. The preference information may be represented based on a quantized preference value or the like.

[0085] In an optional embodiment of the present application, before training the language model based on the path preference pair and the reward model, the language model can also be fine-tuned based on the successful path. The pre-trained language model can be fine-tuned by fine-tuning instructions, and the pre-trained language model can be supervised by positive samples of the successful path to achieve full fine-tuning of the pre-trained language model. After the fine-tuning is completed, the language model is optimized based on the preference data set and the reward model.

[0086] Step 106, calculating loss information based on the output result and a loss function, wherein the loss function is determined according to a reward model.

[0087] Step 108: Adjust the parameters of the language model based on the loss information to obtain an optimized language model.

[0088] The loss information is calculated based on the output result loss function, and then the loss information is back-propagated to the language model to adjust the parameters of the language model to obtain an optimized language model. The calculated loss can be understood as maximizing the probability of the language model outputting a preferred path (i.e., a successful path) based on the input data, and minimizing the probability of the language model outputting a non-preferred path (i.e., a failed path). Therefore, the corresponding loss information can be determined based on the input data, output data, and loss function.

[0089] In summary, the path data of the calling tool corresponding to the language model is obtained, and a preference data set is constructed based on the path data. The path data includes a success path and a failure path, so that the preference data set can be enriched by introducing failure decision information, thereby improving data utilization. Input data is then obtained based on the preference data set, and the input data is input into the language model to obtain the corresponding output result. The loss function is determined based on the output result and the reward model, and the parameters of the language model are adjusted based on the loss function to obtain an optimized language model and improve the accuracy of the language model.

[0090] In an embodiment of the present application, a reward model is constructed to reward preferences, thereby improving the model's reasoning ability using external tools and making its preferences consistent with human preferences. Therefore, a preference reward model can be constructed based on a comparison model, the parameters of the reward model can be estimated based on a setting algorithm, and then the reward model can be optimized based on an optimization algorithm. Among them, the optimization algorithm includes: a preference optimization algorithm and a reinforcement learning algorithm, a preference optimization algorithm such as a direct preference optimization (Direct Preference Optimization, DPO) algorithm, a reinforcement learning algorithm such as a proximal policy optimization (Proximal Policy Optimization, PPO) algorithm, or other preference learning optimization algorithms can be used.

[0091] Reference Figure 3 A reward model construction method shown:

[0092] Step 302: construct a preferred reward model based on the comparison model.

[0093] Step 304: Estimate the parameter function of the reward model based on the set algorithm.

[0094] The goal of the reward model is to maximize the reward, so the reward model can be optimized by an optimization algorithm to determine the loss function for maximizing the reward. The reward model can be constructed by simulating the distribution of preferences through a comparison model such as the Bradley-Terry (BT) model. The parameters of the reward model are estimated by the maximum likelihood estimation algorithm to obtain a parameter function.

[0095] In some optional embodiments of the present application, the preference data set can be defined as ,in, The prompt word represents the i-th data. and represents the corresponding preferred output and non-preferred output. And, the symbol It means that for the prompt x, is More preferred output.

[0096] In the embodiment of this application, expert preference data is used to approximate human preferences. However, since the true distribution of human preferences cannot be directly obtained, it is assumed that the true distribution of human preferences is represented by a potential reward model Generate, where higher rewards represent stronger preferences. Based on the following comparative models such as the Bradley-Terry (BT) model Modeling the distribution of human preferences is shown in the following formula (1):

[0097] Formula (1)

[0098] in, is the logistic function, used to express the function of the preference distribution. is the preferred output of x, is the non-preferred output of x.

[0099] Then, the parameters of the reward model are estimated by maximum likelihood estimation, and the parameter function is as follows:

[0100]

[0101] Step 306: convert the parameter function of the reward model to determine the corresponding loss function.

[0102] The parameter function is transformed to be equivalent to minimizing the log-likelihood as the loss function.

[0103] The above function is equivalent to minimizing the log-likelihood, and the loss function is expressed by the following formula (2):

[0104] Formula (2)

[0105] in, is a parameterized reward model that needs to be learned.

[0106] Step 308, determine the reinforcement learning algorithm.

[0107] Step 310: Optimize the loss function of the reward model based on the reinforcement learning algorithm, wherein the optimization goal of the reinforcement learning algorithm is to maximize the reward.

[0108] Then, feedback learning is used to perform reinforcement learning of the language model. Therefore, the goal of the reward model can be set to maximize the reward. The reward model can be optimized through a direct reinforcement learning algorithm, such as the Proximal Policy Optimization (PPO) algorithm, to maximize the reward.

[0109] In order to optimize the reasoning trajectory of the LLM model by utilizing human preferences, the LLM model can perform reinforcement learning based on feedback, such as optimizing the language model by reinforcement learning from human feedback (RLHF). In the reinforcement learning stage of this method, the optimization target reward is maximized, which can be achieved by the following formula (3):

[0110] Formula (3)

[0111] in, is the learned reward model, It is a strategy model that needs to be optimized. is the model that controls the optimization With the base reference model (i.e., language model, such as the LLM model after fine-tuning training) weight parameter of the bias, represents the KL divergence.

[0112] In practice, It is also initialized as the LLM model after supervised fine-tuning training. This objective function essentially hopes to maximize the reward of the output y generated by the LLM model based on the input x, while using a regular term in the form of KL (Kullback-Leibler divergence, relative entropy) divergence to ensure that the optimized model is not much different from the model before optimization.

[0113] Based on this optimization goal, RLHF first learns the reward model based on formula (2): , and then use reinforcement learning methods (such as PPO) to optimize formula (3), where Used to provide reward feedback to LLM.

[0114] Reference Figure 4 Another way to build a reward model is shown below:

[0115] Step 402: construct a preferred reward model based on the comparison model.

[0116] Step 404: Estimate the parameter function of the reward model based on the set algorithm.

[0117] The goal of the reward model is to establish maximization, so a preference optimization algorithm can be determined, and the reward model is optimized based on the preference optimization algorithm, wherein the optimization goal of the preference optimization algorithm is to maximize the reward. The distribution of preferences can be simulated by comparing models such as the Bradley-Terry (BT) model to construct a reward model. The parameters of the established model are estimated by the maximum likelihood estimation algorithm to obtain a parameterized reward model. Step 306, determine the preference optimization algorithm.

[0118] Step 406: convert the parameter function of the reward model to determine the corresponding loss function.

[0119] The parameter function is transformed to be equivalent to minimizing the log-likelihood as the loss function.

[0120] The construction of the reward model and the determination of the parameter function are similar to the above steps 302 to 306, and therefore will not be described in detail.

[0121] Step 408, determining a preference optimization algorithm.

[0122] Step 410: Optimize the loss function of the reward model based on the preference optimization algorithm, wherein the optimization goal of the preference optimization algorithm is to maximize the reward.

[0123] Then, the feedback learning method is used to perform reinforcement learning of the language model, so the goal of the reward model can be set to maximize the reward, and the reward model can be optimized by a direct optimization algorithm to maximize the reward. In the embodiment of the present application, the preference optimization algorithm can adopt a direct preference optimization (DPO) algorithm.

[0124] In order to further reduce the computational overhead and training time and improve the stability of training, the embodiment of the present application can also use the Direct Preference Optimization (DPO) algorithm to optimize the reward model. First, it is proved that the optimal solution of formula (3) can be obtained, which is expressed by the following formula (4):

[0125] Formula (4)

[0126] in, It is called the partition function (also called the normalization factor). By transposing the terms, we get the following formula (5):

[0127] Formula (5)

[0128] Substituting formula (5) into The expression of, that is, formula (2), obtains the optimization objective of the DPO algorithm, and determines the loss function expressed by the following formula (6):

[0129] Formula (6)

[0130] in, It is a parameterized strategy model that optimizes This learning goal is to Close to the optimal solution This learning goal does not require explicit feedback from the reward model. By optimizing the goal through the DPO algorithm, we avoid the need to learn an additional reward model and the process of reinforcement learning while maximizing the reward.

[0131] After fine-tuning the pre-trained language model with a successful path, the embodiment of the present application uses a direct preference optimization (DPO) method to align a language model such as an LLM model with the task used by the tool on the constructed data set, thereby improving the utilization of data in the decision tree and broadening the learning space of the model, which can further reduce the risk of overfitting.

[0132] By optimizing the reward model in the above manner, the loss function can be calculated based on the reward model and the output result. The loss function can be calculated using the loss entropy function. Then, the loss function is back-propagated to the language model, the parameters of the language model are adjusted, and the path for calling the tool interface is optimized.

[0133] In the embodiments of the present application, the language model may be a large language model or other generative models, such as a large multimodal model.

[0134] Based on the above embodiments, the present application also provides a method for optimizing a language model. Figure 5 As shown:

[0135] Step 502, determine the path of the language model calling tool through the decision tree, and obtain the corresponding success path and failure path.

[0136] Step 504: construct a path preference pair based on the successful path and the failed path.

[0137] Step 506: construct a preference data set based on the path preference pair.

[0138] The root node of the decision tree is the instruction for calling the interface, and the leaf nodes are the results of the call, such as success or failure. The decision tree is traversed to determine the path of the language model calling tool. When traversing to the leaf node, it can be determined whether the path is a successful path or a failed path. A path preference pair is constructed based on the successful path and the failed path, and the preference data set is constructed using the path preference pair.

[0139] like Figure 2 In the decision tree shown, the corresponding path can be obtained by traversing the decision tree, where ⟨0,9, 12, 13, 14, 15> is the successful path of the decision tree, and ⟨0, 1, 2>, ⟨0, 3, 4, 5, 6>, ⟨0, 3, 7, 8>, ⟨0, 9, 10, 11> are four failed paths. The successful path and the failed path are used to form a path preference pair, where the user's preference information for the successful path is greater than the preference information for the failed path. The path preference pair is used to construct a preference data set. For example, the successful path and each failed path constitute a path preference pair, namely: “⟨0, 9, 12, 13,14, 15>>⟨0, 1, 2>”, “⟨0, 9, 12, 13, 14, 15>>⟨0, 3, 4, 5, 6>”, “⟨0, 9, 12, 13,14, 15>>⟨0, 3, 7, 8>”, and “⟨0, 9, 12, 13, 14, 15>>⟨0, 9, 10, 11>”, where “>” indicates large preference information.

[0140] Step 508: fine-tune the language model based on the successful path as training data.

[0141] Fine-tune the language model based on the successful path. You can fine-tune the pre-trained language model by fine-tuning the instructions. Perform supervised learning on the pre-trained language model through the positive samples of the successful path to achieve full fine-tuning of the pre-trained language model. After fine-tuning, optimize the language model based on the preference dataset and reward model.

[0142] Step 510: Determine the instruction words corresponding to the path preference pair based on the preference data set.

[0143] Step 512, construct corresponding input data based on the instruction word.

[0144] Step 514: input the input data into the language model to obtain a corresponding output result.

[0145] Step 516, calculating loss information based on the output result and the loss function.

[0146] Step 518: Adjust the parameters of the language model based on the loss information to obtain an optimized language model.

[0147] The input data is constructed using the instruction words and keywords associated with the path, and the input data is input into the language model for processing, and the language model outputs the corresponding output result. The loss information is calculated based on the input data, output result and loss function, and then back-propagated to the language model based on the loss information, and the parameters of the language model are adjusted to obtain an optimized language model.

[0148] By introducing failed decision information, a preference dataset construction strategy for mining decision tree information is implemented, and a new tool usage preference dataset is constructed based on it.

[0149] Based on the above embodiments, the present application also provides a method for optimizing a language model. Figure 6 As shown:

[0150] Step 602: Determine the successful path of the language model calling tool through a decision tree.

[0151] Step 604: traverse the successful path in the decision tree upward from the leaf node.

[0152] Step 606: For a node with a branch, another node of the branch and a path on the node of the branch are combined into a failure path to obtain at least one failure path.

[0153] Step 608: The successful path and the failed path whose leaf nodes belong to the same branch node form a preferred path pair.

[0154] Step 610: construct a preference data set based on the path preference pair.

[0155] After determining the successful path of the language model calling tool through the decision tree, the failure path corresponding to the successful path can also be constructed, so that the successful path and the corresponding failure path are used to form a path preference pair and added to the preference data set. In an optional embodiment, part or all of the successful path can be used to construct the failure path. When constructing the failure path, the tail node of the successful path is replaced with other nodes, wherein the node of another branch corresponding to the second to last node in the successful path is searched in the decision tree to obtain the corresponding failure node. Thus, a preference pair is constructed using each branch node on the successful path in the decision tree and its corresponding pair of child nodes (which must include the child nodes on the successful path).

[0156] like Figure 2In the decision tree, the complete success path is ⟨0, 9, 12, 13, 14, 15>. Traversing upward along the decision tree, each time a node with a branch is encountered, the node on the other side of the branch and the branch and its upper nodes can be used to form a failure path. The corresponding failure paths are ⟨0, 9, 10>, ⟨0, 3>, and ⟨0, 1>. Then, the success path and the failure path whose leaf nodes belong to the same branch node form a preference path pair, and the preference path pairs include: "⟨0, 9, 12>>⟨0, 9, 10>", "⟨0, 9>>⟨0, 3>", and "⟨0, 9>>⟨0, 1>".

[0157] Step 612: fine-tune the language model based on the successful path as training data.

[0158] Fine-tune the language model based on the successful path. You can fine-tune the pre-trained language model by fine-tuning the instructions. Perform supervised learning on the pre-trained language model through the positive samples of the successful path to achieve full fine-tuning of the pre-trained language model. After fine-tuning, optimize the language model based on the preference dataset and reward model.

[0159] Step 614: Determine the instruction words corresponding to the path preference pair based on the preference data set.

[0160] Step 616, construct corresponding input data based on the instruction word.

[0161] Step 618: input the input data into the language model to obtain a corresponding output result.

[0162] Step 620, calculating loss information based on the output result and the loss function.

[0163] Step 622: Adjust the parameters of the language model based on the loss information to obtain an optimized language model.

[0164] The input data is constructed using the instruction words and keywords associated with the path, and the input data is input into the language model for processing, and the language model outputs the corresponding output result. The loss information is calculated based on the input data, output result and loss function, and then back-propagated to the language model based on the loss information, and the parameters of the language model are adjusted to obtain an optimized language model.

[0165] In the embodiment of the present application, each pair of branch nodes on the successful path in the decision tree is parsed into a preference sample pair, thereby constructing a new step-by-step tool using a preference data set. The preference data comes directly from the decision tree, which can reduce the dependence on model training and more clearly understand the differences between each reasoning step, thereby reducing the computational and time overhead.

[0166] Based on the above embodiments, the present application also provides a method for optimizing a language model. Figure 7 As shown:

[0167] Step 702, determine the path of the language model calling tool through the decision tree, and obtain the corresponding success path and failure path.

[0168] Step 704: construct a path preference pair based on the successful path and the failed path.

[0169] Step 706: construct a preference data set based on the path preference pair.

[0170] Step 708: construct a corresponding failure path based on the success path and the decision tree, wherein at least the tail node of the constructed failure path is different from the tail node of the corresponding success path.

[0171] The method of constructing a corresponding failure path based on the success path and the decision tree includes: traversing the success path in the decision tree from a leaf node upward; for a node with a branch, forming a failure path with another node of the branch and a path above the node of the branch, and obtaining at least one failure path.

[0172] Step 710: The successful path and the corresponding failed path are used as a path preference pair.

[0173] Step 712: construct a preference data set using the path preference pairs.

[0174] Step 714: fine-tune the language model based on the successful path as training data.

[0175] Step 716, determining the instruction words corresponding to the path preference pair based on the preference data set.

[0176] Step 718, construct corresponding input data based on the instruction word.

[0177] Step 720: input the input data into the language model to obtain a corresponding output result.

[0178] Step 722, calculate loss information based on the output result and the loss function.

[0179] Step 724: Adjust the parameters of the language model based on the loss information to obtain an optimized language model.

[0180] The input data is constructed using the instruction words and keywords associated with the path, and the input data is input into the language model for processing, and the language model outputs the corresponding output result. The loss information is calculated based on the input data, output result and loss function, and then back-propagated to the language model based on the loss information, and the parameters of the language model are adjusted to obtain an optimized language model.

[0181] In the embodiment of the present application, by introducing failed decision information, a preference data set construction strategy for mining decision tree information is implemented, and a new tool usage preference data set is constructed thereby.

[0182] In addition, the DPO preference learning algorithm is used to optimize language models such as the LLM model based on the preference data set, thereby improving the language model's tool usage and decision-making capabilities.

[0183] There are many application scenarios, as long as it involves businesses that require autonomous decision-making and the use of multiple algorithm capabilities, external tools, etc. to solve certain needs. For example: 1. In the intelligent refund scenario, the intelligent agent can understand the problem based on the user's dispute description, and then decide how to determine whether the problem described by the user exists (for example, whether it is necessary to view his evidence pictures and videos, whether it is necessary to view the order product information, whether it is necessary to check the logistics, etc.).

[0184] Based on the language model obtained in the above embodiment, the language model can be applied to various scenarios to answer professional questions and needs in various fields. The language model can be applied to various scenarios that require autonomous decision-making and the use of multiple algorithm capabilities, external tools, etc. to solve certain needs.

[0185] Taking a question-answering method based on a language model as an example, the language model can call a tool interface in the process of answering user questions, so as to obtain information in the corresponding field through the tool interface and process it to obtain the required answer. Question information can be received through an interface or an application (Application, APP) page, and then the question information is input into the language model. The language model analyzes the question information to determine the corresponding task information; calls the tool interface based on the task information to receive the corresponding feedback information; and generates answer information based on the feedback information. For example, when asking questions about financial issues, promotional issues in the e-commerce field, etc., the language model can call the corresponding financial tools, e-commerce website interfaces, etc. to obtain information, and combine the corresponding answers for feedback.

[0186] In the field of e-commerce, it is applied to cross-border e-commerce websites, non-cross-border e-commerce websites, and other apps that support e-commerce. It can be used in customer service and other scenarios, such as automatically understanding the semantics of users' inquiries about products and orders and calling corresponding tools to solve users' problems. In after-sales scenarios, it can determine whether user refunds are feasible, whether they meet the refund standards, and what refund standards are met. It can also understand the semantics and call historical communication records to resolve disputes and determine the responsibility when disputes arise between merchants (or sellers) and buyers.

[0187] Taking a customer service scenario as an example, a language model can be called to analyze the service demand information received by the customer service page, and the required service tools can be called to obtain service information, so as to obtain the corresponding service demand results and feedback to each user, automatically solving various user needs, reducing manual intervention, and improving processing efficiency.

[0188] Reference Figure 8 , showing a step flow chart of an embodiment of a processing method based on a language model of the present application.

[0189] Step 802, receiving service demand information obtained through the customer service page.

[0190] When users are shopping on e-commerce websites or apps, they can consult customer service for various needs. The customer service page can be started to send service demand information. After the user terminal receives the service demand information through the customer service page, it can send the service demand information to the server for processing.

[0191] Step 804: Analyze the service demand information through a language model to determine a tool path of a required service tool.

[0192] Step 806: Obtain corresponding service information based on the tool path.

[0193] After receiving the service demand information, the server can call the language model to automatically analyze and feedback the service demand information. The language model can be trained in the manner of the above embodiment, so that it can automatically call the required tools to obtain information to solve user needs based on understanding user semantics.

[0194] The server inputs the service demand information into the language model, which can process the service demand information and understand the semantics corresponding to the service demand information through natural language processing, thereby determining at least one user demand, and based on the demand, determining the service tool to be called, and determining the tool path corresponding to the service tool, that is, the path of the tool interface to be called to solve the user demand. The corresponding demand information is transmitted to the corresponding service tool through the interface to obtain the service information fed back by the service tool. Among them, one or more service tool interfaces may be called to meet the user's service needs.

[0195] In an optional embodiment, the service tool includes at least one of the following tools: a logistics query tool, an order query tool, and a commodity query tool. The logistics query tool is used to query logistics-related information, and the order query tool is used to query order-related information, including information of the commodity object corresponding to the order, order status, etc. The commodity query tool is used to query commodity object information. For example, the user's service demand information is parsed through a language model to determine that the user needs to query the logistics information of the purchased commodity. The commodity object purchased by the user and its order can be determined through an order query interface, and the associated logistics order is determined based on the order. Then, the logistics status of the logistics order is queried through a logistics query interface, thereby using the logistics information as service information. For another example, the user's service demand information is parsed through a language model to determine that the user needs to query the size information of a specified commodity object. The commodity information details page of the commodity object can be queried through the commodity query tool, thereby obtaining the size information of the commodity object based on the details page as service information. For another example, if the commodity object purchased by the user is a handmade product, an item that needs to be assembled, etc., the assembly method, usage method, etc. can be consulted through the customer service page. The corresponding language model can call the interface of the commodity query tool on the basis of understanding the semantics, obtain the video, graphic tutorial, etc. of the assembly method and usage method corresponding to the commodity object, and provide feedback.

[0196] Therefore, the language model can automatically understand the semantics of user service requirements, analyze based on the semantics, call the corresponding service tool when necessary, and automatically analyze the tool path of the required service tool, so as to obtain the required service information through the interface of the service tool called by the tool path.

[0197] Step 808: Generate a service demand result based on the service information.

[0198] After the language model obtains the service information through the interface of the service tool, it can combine the service information with other information to solve the user's needs to obtain the service demand result. It can also obtain the corresponding feedback template and combine the service information with other information to solve the user's needs according to the service to generate the service demand result.

[0199] Step 810: Send the service demand result to feedback the service demand result through the customer service page.

[0200] The server sends the service demand result to the user terminal, so that the service demand result is fed back through the customer service page on the user terminal, and the user can obtain the required information. If there are other problems, they can also be automatically analyzed and processed through the customer service page interface and the language model.

[0201] Therefore, the language model can be used to automatically analyze user needs and call corresponding tools to solve them, minimizing manual intervention, improving processing efficiency and user satisfaction, and reducing labor costs.

[0202] Taking an after-sales customer service scenario as an example, the language model can be used to analyze the after-sales service requirements received on the customer service page and call tools to process them, such as analyzing the user's after-sales resources, refunds, etc., so as to automatically feedback the corresponding service results.

[0203] Reference Fig. 9 , showing a step flow chart of an embodiment of a processing method based on a language model of the present application.

[0204] Step 902: Receive resource withdrawal information obtained through the customer service page.

[0205] When users are shopping on e-commerce websites or apps, they can consult customer service for various needs. The customer service page can be started to send service demand information. After the user terminal receives the service demand information through the customer service page, it can send the service demand information to the server for processing.

[0206] Step 904: Analyze the resource withdrawal information through a language model to determine a tool path of a service tool corresponding to the associated information of the resource withdrawal.

[0207] Step 906: Acquire corresponding association information based on the tool path.

[0208] Step 908: Generate a resource withdrawal result based on the association information.

[0209] After receiving the service demand information, the server can call the language model to automatically analyze and feedback the service demand information. The language model can be trained in the manner of the above embodiment, so that it can automatically call the required tools to obtain information to solve user needs based on understanding user semantics.

[0210] For example, after purchasing and receiving a product object, if the user is not satisfied with the product object and applies for a refund, the corresponding terminal can obtain resource withdrawal information and send it to the server. For example, the buyer and seller users cannot reach an agreement on the refund issue. For example, the buyer believes that there is a quality problem with the product object, but the seller disagrees. At this time, the platform customer service needs to intervene. The server can use the language model to analyze the resource withdrawal information, determine the key points of the dispute issue by understanding the semantics, determine the required related information for the key points, and then determine the tool path of the service tool for the related information, that is, the path of the tool interface required to resolve the dispute. The corresponding demand information is transmitted to the corresponding service tool through the interface to obtain the service information fed back by the service tool. Among them, one or more service tool interfaces may be called to meet the user's service needs.

[0211] The service tool includes at least one of the following tools: logistics query tool, order query tool, product query tool, service information query tool, and credit query tool. The logistics query tool is used to query logistics-related information, and the order query tool is used to query order-related information, including information about the product object corresponding to the order, order status, etc. The product query tool is used to query product object information. The credit query tool is used to query the user's credit information. The service information query tool is used to query historical service information, such as historical communication records between seller users and buyer users.

[0212] For example, the user's resource withdrawal information can be parsed through the language model to determine the information category required for the resource withdrawal, and the corresponding service tool can be called according to the information category. For example, for the quality problems stated by the buyer user, the text, pictures, videos and other information related to the quality problems reported by the user can be obtained through the service information query tool and the order query tool, the information of the product object can be queried through the product query tool, the logistics information can be queried through the logistics query tool, and the credit information of the buyer user and the seller user can be queried through the credit query tool. A variety of information is combined to decide the corresponding judgment result for the dispute. The language model can also call the policy tool to obtain policy information, so as to determine the resource withdrawal result by combining the policy information and related information.

[0213] Step 910: Send the resource withdrawal result to feedback the resource withdrawal result through the customer service page.

[0214] The server sends the resource withdrawal result to the user terminal, which then feeds back the resource withdrawal result through the customer service page on the user terminal, and the user can determine the corresponding processing result, such as agreeing to the resource withdrawal, or returning the refund to the user's account if agreeing to the refund, etc. If the resource withdrawal is not agreed, the corresponding judgment reason is given.

[0215] Therefore, it is possible to automatically analyze resource withdrawal through language models and call corresponding tools to solve the problem, integrate various related information, minimize human intervention, improve processing efficiency and user satisfaction, and reduce labor costs.

[0216] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described in this article within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0217] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0218] On the basis of the above embodiment, this embodiment further provides a language model optimization device, which is applied to electronic devices such as server devices, and the device includes:

[0219] A preference acquisition module, used to acquire path data of a calling tool corresponding to a language model, and to construct a preference data set based on the path data, wherein the path data includes a successful path and a failed path;

[0220] A model processing module, used to obtain input data based on the preference data set, input the input data into the language model, and obtain a corresponding output result;

[0221] A loss determination module, configured to determine a loss function based on the output result and the reward model;

[0222] The optimization and adjustment module is used to adjust the parameters of the language model based on the loss function to obtain an optimized language model.

[0223] In summary, the path data of the calling tool corresponding to the language model is obtained, and a preference data set is constructed based on the path data. The path data includes a success path and a failure path, so that the preference data set can be enriched by introducing failure decision information, thereby improving data utilization. Input data is then obtained based on the preference data set, and the input data is input into the language model to obtain the corresponding output result. The loss function is determined based on the output result and the reward model, and the parameters of the language model are adjusted based on the loss function to obtain an optimized language model and improve the accuracy of the language model.

[0224] The preference acquisition module is used to determine the path of the language model calling tool through a decision tree, obtain the corresponding success path and failure path; construct a path preference pair based on the success path and failure path; and construct a preference data set based on the path preference pair.

[0225] The preference acquisition module is used to determine the successful path of the language model calling tool through a decision tree; construct a corresponding failure path based on the successful path and the decision tree, and at least the tail node of the constructed failure path is different from the tail node of the corresponding successful path; use the successful path and the corresponding failure path as a path preference pair; and construct a preference data set based on the path preference pair.

[0226] The preference acquisition module is used to traverse the successful path in the decision tree from the leaf node upward; for a node with a branch, another node of the branch and the path above the node of the branch constitute a failed path to obtain at least one failed path.

[0227] The model processing module is used to determine the instruction words corresponding to the path preference pairs based on the preference data set; and construct corresponding input data based on the instruction words.

[0228] It also includes: a reward model construction module, which is used to construct a preferred reward model based on the comparison model and estimate the parameters of the reward model based on a set algorithm.

[0229] The reward model optimization module is used to determine a preference optimization algorithm and optimize the reward model based on the preference optimization algorithm, wherein the optimization goal of the preference optimization algorithm is to maximize the reward.

[0230] The reward model optimization module is used to determine a reinforcement learning algorithm and optimize the reward model based on the reinforcement learning algorithm, wherein the optimization goal of the reinforcement learning algorithm is to maximize the reward.

[0231] A model fine-tuning module is used to fine-tune the language model based on the successful path as training data.

[0232] After fine-tuning the pre-trained language model with a successful path, the embodiment of the present application uses a direct preference optimization (DPO) method to align a language model such as an LLM model with the task used by the tool on the constructed data set, thereby improving the utilization of data in the decision tree and broadening the learning space of the model, which can further reduce the risk of overfitting.

[0233] In the embodiment of the present application, each pair of branch nodes on the successful path in the decision tree is parsed into a preference sample pair, thereby constructing a new step-by-step tool using a preference data set. The preference data comes directly from the decision tree, which can reduce the dependence on model training and more clearly understand the differences between each reasoning step, thereby reducing the computational and time overhead.

[0234] On the basis of the above embodiments, the embodiment of the present application further provides a processing device based on a language model, which is applied to an electronic device on the server side:

[0235] A demand receiving module is used to receive service demand information obtained through the customer service page;

[0236] A model processing module, used to analyze the service demand information through a language model to determine a tool path of a required service tool, wherein a loss function in the language model training process is determined based on a reward model, the training data of the language model is determined based on a preference data set, the preference data set is determined based on path data of a tool called corresponding to the language model, the path data includes a successful path and a failed path; obtain corresponding service information based on the tool path; and generate a service demand result based on the service information;

[0237] The result feedback module is used to send the service demand result so as to feedback the service demand result through the customer service page.

[0238] Optionally, the service tool includes at least one of the following tools: a logistics query tool, an order query tool, and a product query tool.

[0239] Therefore, the language model can be used to automatically analyze user needs and call corresponding tools to solve them, minimizing manual intervention, improving processing efficiency and user satisfaction, and reducing labor costs.

[0240] Taking an after-sales service scenario as an example, the device includes:

[0241] The demand receiving module is used to receive resource withdrawal information through the customer service page;

[0242] A model processing module is used to analyze the resource withdrawal information through a language model to determine a tool path of a service tool corresponding to the associated information of the resource withdrawal, wherein the loss function in the language model training process is determined based on a reward model, the training data of the language model is determined based on a preference data set, the preference data set is determined based on path data of a calling tool corresponding to the language model, and the path data includes a successful path and a failed path; obtain corresponding associated information based on the tool path; and determine a resource withdrawal result based on the associated information;

[0243] The result feedback module is used to send the resource withdrawal result so as to feedback the resource withdrawal result through the customer service page.

[0244] Optionally, the service tool includes at least one of the following tools: a logistics query tool, an order query tool, a product query tool, a service information query tool, and a credit query tool.

[0245] Therefore, it is possible to automatically analyze resource withdrawal through language models and call corresponding tools to solve the problem, integrate various related information, minimize human intervention, improve processing efficiency and user satisfaction, and reduce labor costs.

[0246] The embodiment of the present application further provides a non-volatile readable storage medium, in which one or more modules (programs) are stored. When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0247] The embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method described in the embodiment of the present application.

[0248] The embodiment of the present application also provides an electronic device, comprising: a processor, and a memory connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the embodiment of the present application. In the embodiment of the present application, the electronic device includes a server, a terminal device, and other devices.

[0249] The embodiment of the present application further discloses a computer program product, including a computer program / computer executable instructions, wherein the computer program / computer executable instructions, when executed by a processor in an electronic device, implement the method described in the embodiment of the present application.

[0250] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as a server (cluster), a terminal, etc. Fig.10 An exemplary apparatus 1000 that can be used to implement various embodiments described in this application is schematically illustrated.

[0251] For one embodiment, Fig.10An exemplary apparatus 1000 is shown having one or more processors 1002, a control module (chip set) 1004 coupled to at least one of the processor(s) 1002, a memory 1006 coupled to the control module 1004, a non-volatile memory (NVM) / storage device 1008 coupled to the control module 1004, one or more input / output devices 1010 coupled to the control module 1004, and a network interface 1012 coupled to the control module 1004.

[0252] The processor 1002 may include one or more single-core or multi-core processors, and the processor 1002 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1000 can be used as a server, terminal, or other device described in the embodiments of the present application.

[0253] In some embodiments, the apparatus 1000 may include one or more computer-readable media (e.g., memory 1006 or NVM / storage device 1008) having instructions 1014 and one or more processors 1002 combined with the one or more computer-readable media and configured to execute the instructions 1014 to implement a module to perform the actions described in the present disclosure.

[0254] For one embodiment, the control module 1004 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1002 and / or any suitable device or component in communication with the control module 1004 .

[0255] The control module 1004 may include a memory controller module to provide an interface to the memory 1006. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0256] The memory 1006 may be used, for example, to load and store data and / or instructions 1014 for the device 1000. For one embodiment, the memory 1006 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 1006 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0257] For one embodiment, control module 1004 may include one or more input / output controllers to provide an interface to NVM / storage device 1008 and input / output device(s) 1010 .

[0258] For example, NVM / storage 1008 may be used to store data and / or instructions 1014. NVM / storage 1008 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0259] NVM / storage device 1008 may include storage resources that are part of the device on which apparatus 1000 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 1008 may be accessed via (one or more) input / output devices 1010 over a network.

[0260] (One or more) input / output devices 1010 may provide an interface for the apparatus 1000 to communicate with any other appropriate device, and the input / output device 1010 may include a communication component, an audio component, a sensor component, etc. The network interface 1012 may provide an interface for the apparatus 1000 to communicate through one or more networks, and the apparatus 1000 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as Bluetooth, WiFi, 2G, 3G, 4G, 5G, 6G, etc., or a combination thereof for wireless communication.

[0261] For one embodiment, at least one of the processor(s) 1002 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1004. For one embodiment, at least one of the processor(s) 1002 may be packaged together with the logic of one or more controllers of the control module 1004 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 1002 may be integrated on the same die with the logic of one or more controllers of the control module 1004. For one embodiment, at least one of the processor(s) 1002 may be integrated on the same die with the logic of one or more controllers of the control module 1004 to form a system-on-chip (SoC).

[0262] In various embodiments, the apparatus 1000 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the apparatus 1000 may have more or fewer components and / or a different architecture. For example, in some embodiments, the apparatus 1000 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0263] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0264] The embodiment of the present application also provides an electronic device, including: a processor; and a memory, on which executable code is stored, and when the executable code is executed, the processor executes one or more methods as described in the embodiment of the present application. In the embodiment of the present application, the memory can store various data, such as target files, file and application association data, and user behavior data, so as to provide a data basis for various processing.

[0265] The embodiments of the present application also provide one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes one or more methods described in the embodiments of the present application.

[0266] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0267] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0268] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0269] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0270] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0271] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0272] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0273] The above is a detailed introduction to a language model optimization method, a language model-based processing method, an electronic device, a computer-readable storage medium and a computer program product provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for optimizing a language model, wherein: The method comprises: Acquire path data of a calling tool corresponding to a language model, and construct a preference data set based on the path data, wherein the path data includes path preference pairs consisting of a successful path and a failed path; Based on the preference data set, determining the instruction words corresponding to the path preference pair; Constructing corresponding input data based on the instruction word, inputting the input data into the language model, and obtaining a corresponding output result; Calculating loss information based on the output result and a loss function, wherein the loss function is determined according to a reward model, and the reward model is constructed based on a comparison model; Adjusting parameters of the language model based on the loss information to obtain an optimized language model; The obtaining of path data of a calling tool corresponding to the language model and constructing a preference data set based on the path data includes: Determine the successful path of the language model calling tool through a decision tree; Traversing the successful path in the decision tree upwards from a leaf node; For a node with a branch, another node of the branch and a path on the node of the branch are used to form a failure path, so as to obtain at least one failure path, wherein at least a tail node of the constructed failure path is different from a tail node of a corresponding success path; taking the successful path and the corresponding failed path as a path preference pair; A preference dataset is constructed based on the path preference pairs.

2. The method according to claim 1, wherein: The step of obtaining the path data of the calling tool corresponding to the language model and constructing the preference data set based on the path data further includes: Determine the path of the language model calling tool through the decision tree and obtain the corresponding success path and failure path; constructing a path preference pair based on the success path and the failure path; A preference dataset is constructed based on the path preference pairs.

3. The method according to claim 1, wherein: The steps of constructing the reward model are also included: A preferred reward model is constructed based on the comparison model, and a parameter function of the reward model is estimated based on a set algorithm.

4. The method according to claim 3, wherein: It also includes the step of determining the loss function based on the reward model: Converting the parameter function of the reward model to determine the corresponding loss function; Determine a preference optimization algorithm, and optimize the loss function of the reward model based on the preference optimization algorithm, wherein the optimization goal of the preference optimization algorithm is to maximize the reward.

5. The method according to claim 3, wherein: It also includes the step of determining the loss function based on the reward model: Converting the parameter function of the reward model to determine the corresponding loss function; A reinforcement learning algorithm is determined, and a loss function of the reward model is optimized based on the reinforcement learning algorithm, wherein the optimization goal of the reinforcement learning algorithm is to maximize the reward.

6. The method according to claim 1, wherein: Also includes: The language model is fine-tuned based on the successful path as training data.

7. A processing method based on a language model, wherein: The method comprises: Receive service demand information obtained through the customer service page; The service demand information is analyzed by a language model to determine a tool path of the required service tool, wherein the loss function in the language model training process is determined based on a reward model, the reward model is constructed based on a comparison model, the input data for the language model training is constructed based on instruction words, the instruction words are determined based on path preference pairs in a preference data set, the preference data set is determined based on path data of a tool called corresponding to the language model, the path data includes a path preference pair consisting of a successful path and a failed path; the preference data set is constructed in the following manner: a successful path for a tool called by a language model is determined by a decision tree, the successful path is traversed upward from a leaf node in the decision tree, and for a node with a branch, another node of the branch and the path above the node of the branch are used to form a failed path to obtain at least one failed path, at least the tail node of the constructed failed path is different from the tail node of the corresponding successful path, the successful path and the corresponding failed path are used as a path preference pair, and a preference data set is constructed based on the path preference pair; Acquire corresponding service information based on the tool path; generating a service demand result based on the service information; The service demand result is sent to feedback the service demand result through the customer service page.

8. The method according to claim 7, wherein: The service tool includes at least one of the following tools: a logistics query tool, an order query tool, and a product query tool.

9. A processing method based on a language model, wherein: The method comprises: Receive resource withdrawal information through the customer service page; The resource withdrawal information is analyzed by a language model to determine a tool path of a service tool for resource withdrawal corresponding to the associated information, wherein the loss function in the language model training process is determined based on a reward model, the reward model is constructed based on a comparison model, the input data for the language model training is constructed based on instruction words, the instruction words are determined based on path preference pairs in a preference data set, the preference data set is determined based on path data of a tool called corresponding to the language model, the path data includes a path preference pair consisting of a successful path and a failed path; the preference data set is constructed in the following manner: a successful path of a tool called by a language model is determined by a decision tree, the successful path is traversed upward from a leaf node in the decision tree, and for a node with a branch, another node of the branch and the path above the node of the branch are used to form a failed path, and at least one failed path is obtained, at least the tail node of the constructed failed path is different from the tail node of the corresponding successful path, the successful path and the corresponding failed path are used as a path preference pair, and a preference data set is constructed based on the path preference pair; Acquire corresponding association information based on the tool path; Determine a resource withdrawal result based on the association information; The resource withdrawal result is sent to feedback the resource withdrawal result through the customer service page.

10. The method according to claim 9, wherein: The service tool includes at least one of the following tools: a logistics query tool, an order query tool, a product query tool, a service information query tool, and a credit query tool.

11. An electronic device, wherein: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 10 when executed by a processor.

13. A computer program product comprising a computer program / computer executable instructions, wherein: When the computer program / computer executable instructions are executed by a processor in an electronic device, the method according to any one of claims 1 to 10 is implemented.

Images