Model training method, question processing method, device, equipment and medium

Abstract

Embodiments of the present disclosure disclose a model training method, a question processing method, an apparatus, a device and a medium. A specific implementation of the method comprises: obtaining target training data; inputting a reply question to an initial large language model multiple times to obtain multiple reply information; for each of the multiple reply information, performing a generation step: determining semantic accurate information according to standard reply information; performing segmentation processing on the reply information to obtain segmented information; generating segmented reward information and semantic accurate reward information according to the semantic accurate information and the segmented information; generating a reply loss according to the segmented reward information and the semantic accurate reward information; determining whether the initial large language model is trained to be completed according to the multiple reply losses; and in response to determining that the training is completed, determining the initial large language model as a large language model. The implementation is related to artificial intelligence, and the trained large language model can generate reply information with concise words and correct semantic reply.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of artificial intelligence technology, specifically to model training methods, problem-solving methods, apparatus, electronic devices, computer-readable media, and program products. Background Technology

[0002] Currently, with the continuous development of large language models, deep thinking capabilities have become a mainstream development direction. However, the phenomenon of model overthinking is gradually becoming a burden on online inference. Specifically, due to the limitations of the Transformers architecture, the longer the output sequence, the higher the storage cost and inference latency. To reduce the inference latency of online models, how to achieve efficient thinking has become an important issue. Existing methods are based on supervised fine-tuning (SFT) to help the model learn how to generate more concise response information. However, supervised fine-tuning (SFT) methods have a generalization upper limit problem and require manually constructing a concise dataset, which is costly and may result in response information that is not concise enough. Summary of the Invention

[0003] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0004] Some embodiments of this disclosure provide model training methods, problem-solving methods, apparatuses, electronic devices, computer-readable media, and program products to address the technical problems mentioned in the background section above.

[0005] In a first aspect, some embodiments of this disclosure provide a model training method, comprising: acquiring target training data, wherein the target training data includes: answer questions and standard answer information; inputting the answer questions multiple times into an initial large language model to obtain multiple answer information; for each of the multiple answer information, performing a generation step: determining semantic accuracy information corresponding to the answer information based on the standard answer information; segmenting the answer information to obtain segmentation information; generating segmentation reward information and semantic accuracy reward information based on the semantic accuracy information and the segmentation information; generating answer loss based on the segmentation reward information and semantic accuracy reward information; determining whether the initial large language model has finished training based on the multiple answer losses; and, in response to determining that training has finished, identifying the initial large language model as a large language model.

[0006] Optionally, the above segmentation information is the number of segments; and the above generation of segmentation reward information and semantic accuracy reward information based on the above semantic accuracy information and the above segmentation information includes: selecting target response information that corresponds to semantic accuracy information to represent semantic accuracy from the above multiple response information to obtain at least one target response information; selecting the segmentation number with the smallest value from the at least one segmentation number corresponding to the above at least one target response information as the minimum segmentation number; and generating segmentation reward information and semantic accuracy reward information based on the above semantic accuracy information, the above minimum segmentation number and the above segmentation number.

[0007] Optionally, generating segmentation reward information and semantic accuracy reward information based on the semantic accuracy information, the minimum number of segments, and the number of segments includes: in response to determining that the semantic accuracy information represents semantic accuracy, setting a first value as the semantic accuracy reward information corresponding to the response information; determining the numerical difference between the minimum number of segments and the number of segments as the segmentation reward information; in response to determining that the semantic accuracy information represents semantic inaccuracy, setting a second value as the semantic accuracy reward information corresponding to the response information; and determining the maximum value between the numerical difference and the second value as the segmentation reward information.

[0008] Optionally, generating the response loss based on the segmentation reward information and the semantically accurate reward information includes: performing a weighted summation on the segmentation reward information and the semantically accurate reward information to obtain weighted reward information; normalizing the weighted reward information to obtain normalized reward information; and inputting the normalized reward information into the loss function corresponding to the target reinforcement learning to obtain the response loss.

[0009] Optionally, the above-mentioned segmentation of the response information to obtain segmented information includes: using at least one of the following methods to segment the response information based on paragraph text similarity, paragraph segmentation, conjunction segmentation, and sentence segmentation to obtain segmented information.

[0010] Secondly, some embodiments of this disclosure provide a model training apparatus, comprising: a first acquisition unit configured to acquire target training data, wherein the target training data includes: answer questions and standard answer information; a first input unit configured to input the answer questions multiple times into an initial large language model to obtain multiple answer information; an execution unit configured to perform a generation step for each of the multiple answer information: determining semantic accuracy information corresponding to the answer information based on the standard answer information; segmenting the answer information to obtain segmentation information; generating segmentation reward information and semantic accuracy reward information based on the semantic accuracy information and the segmentation information; generating an answer loss based on the segmentation reward information and the semantic accuracy reward information; a first determination unit configured to determine whether the initial large language model has finished training based on the multiple answer losses; and a second determination unit configured to determine the initial large language model as a large language model in response to determining that training has ended.

[0011] Optionally, the above segmentation information is the number of segments; and the execution unit can be configured to: select the target response information that corresponds to the semantically accurate information from the above multiple response information to obtain at least one target response information; select the segment number with the smallest value from the at least one segment number corresponding to the at least one target response information as the minimum segment number; and generate segmentation reward information and semantically accurate reward information based on the above semantically accurate information, the above minimum segment number and the above segment number.

[0012] Optionally, the execution unit can be configured to: in response to determining that the semantically accurate information represents semantic accuracy, set a first value as the semantically accurate reward information corresponding to the response information; determine the difference between the minimum number of segments and the number of segments as the segmentation reward information; in response to determining that the semantically accurate information represents semantic incorrectness, set a second value as the semantically accurate reward information corresponding to the response information; and determine the maximum value between the difference between the numbers and the second value as the segmentation reward information.

[0013] Optionally, the execution unit can be configured to: perform weighted summation on the above segmentation reward information and the above semantic accuracy reward information to obtain weighted reward information; perform normalization on the above weighted reward information to obtain normalized reward information; and input the above normalized reward information into the loss function corresponding to the target reinforcement learning to obtain the response loss.

[0014] Optionally, the execution unit can be configured to segment the above-mentioned response information using at least one of the following methods: paragraph text similarity-based segmentation, paragraph segmentation, conjunction segmentation, and sentence segmentation, to obtain segmented information.

[0015] Thirdly, some embodiments of this disclosure provide a problem-solving method, including: acquiring a target logistics problem within a target logistics domain; inputting the target logistics problem into a pre-trained large language model to obtain logistics response information, wherein the large language model is generated based on a model training method; and controlling a target robot to execute a logistics processing strategy according to the logistics response information.

[0016] Fourthly, some embodiments of this disclosure provide a problem processing apparatus, including: a second acquisition unit configured to acquire a target logistics problem within a target logistics domain; a second input unit configured to input the target logistics problem into a pre-trained large language model to obtain logistics response information, wherein the large language model is generated based on a model training method; and a control unit configured to control a target robot to execute a logistics processing strategy according to the logistics response information.

[0017] Fifthly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any of the implementations of the first and third aspects.

[0018] Sixthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any of the implementations of the first and third aspects.

[0019] In a seventh aspect, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the implementations of the first and third aspects above.

[0020] The above embodiments of this disclosure have the following beneficial effects: By using the model training methods of some embodiments of this disclosure, and utilizing the trained large language model, concise and semantically correct response information can be generated. Specifically, the reason why the relevant response information is not concise enough is that the supervised fine-tuning (SFT) method has a generalization upper limit problem and requires the manual construction of a simplified dataset, which is costly and may result in the response information not being concise enough. Based on this, the model training method of some embodiments of this disclosure first obtains target training data, which includes: response questions and standard response information. Here, the target training data is used as training samples to train the subsequent initial large language model. Then, the above response questions are input into the initial large language model multiple times to obtain multiple response information. Here, the segmentation reward and semantic reward of the initial large language model under each response information output for the same response question are determined to ensure the accuracy and conciseness of the corresponding output of the initial large language model. Next, for each of the above multiple response information, a generation step is performed: First, based on the above standard response information, the semantic accuracy information corresponding to the above response information can be accurately determined to facilitate the subsequent generation of semantic accuracy reward information. The second step involves segmenting the response information to obtain segmented information, which is then used to evaluate the conciseness of the response. The third step, based on the semantic accuracy information and the segmented information, accurately generates segmentation reward information and semantic accuracy reward information. These are used to generate the response loss for subsequent response information generation. The fourth step, based on the segmentation reward information and semantic accuracy reward information, accurately generates the response loss to characterize the difference between the response information under textual conciseness and semantic accuracy features. Furthermore, based on multiple response losses, it can be accurately determined whether the initial large language model has finished training. Finally, in response to the determination of training completion, the initial large language model is established as the large language model. In summary, by determining the segmentation reward information and semantic accuracy reward information corresponding to multiple response information output by the initial large language model, it is possible to determine whether the initial large language model has finished training, ensuring that the subsequently trained large language model outputs concise and semantically accurate response information. Attached Figure Description

[0021] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0022] Figure 1 This is a schematic diagram illustrating an application scenario of a model training method according to some embodiments of the present disclosure;

[0023] Figure 2 This is a flowchart of some embodiments of the model training method according to this disclosure;

[0024] Figure 3 These are flowcharts of other embodiments of the model training method according to this disclosure;

[0025] Figure 4 These are flowcharts of some embodiments of the problem-solving methods according to this disclosure;

[0026] Figure 5 This is a schematic diagram of the structure of some embodiments of the model training apparatus according to the present disclosure;

[0027] Figure 6 These are schematic diagrams illustrating the structure of some embodiments of the problem-solving apparatus according to this disclosure;

[0028] Figure 7 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0029] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0030] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0031] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0032] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0033] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0034] Before performing any of the operations involving the collection, storage, or use of user personal information (such as user profiles and user historical behavior) disclosed in this disclosure, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, informing the personal information subjects, and obtaining prior authorization and consent from the personal information subjects.

[0035] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] Figure 1 This is a schematic diagram illustrating an application scenario of a model training method according to some embodiments of the present disclosure.

[0037] exist Figure 1In this application scenario, firstly, the electronic device 101 can acquire target training data. This target training data includes: a response question 102 and standard response information 103. Then, the electronic device 101 can input the response question 102 multiple times into the initial large language model 104 to obtain multiple response messages. In this application scenario, the multiple response messages can include: response information A, response information B, and response information C. Next, the electronic device 101 can perform the following generation steps for each of the multiple response messages: First, based on the standard response information 103, determine the semantic accuracy information corresponding to the response information. In this application scenario, for response information A, the corresponding semantic accuracy information is semantic accuracy information A. For response information B, the corresponding semantic accuracy information is semantic accuracy information B. For response information C, the corresponding semantic accuracy information is semantic accuracy information C. Second, perform segmentation processing on the response information to obtain segmentation information. In this application scenario, for response information A, the corresponding segmentation information is segmentation information A. For response information B, the corresponding segmentation information is segmentation information B. For response information C, the corresponding segmentation information is segmentation information C. The third step is to generate segmentation reward information and semantic accuracy reward information based on the aforementioned semantic accuracy information and segmentation information. In this application scenario, segmentation reward information A and semantic accuracy reward information A are generated based on segmentation information A and semantic accuracy information A. Segmentation reward information B and semantic accuracy reward information A are generated based on segmentation information B and semantic accuracy information B. Segmentation reward information A and semantic accuracy reward information C are generated based on segmentation information C and semantic accuracy information C. The fourth step is to generate response loss based on the aforementioned segmentation reward information and semantic accuracy reward information. In this application scenario, response loss A is generated based on segmentation reward information A and semantic accuracy reward information A. Response loss B is generated based on segmentation reward information B and semantic accuracy reward information B. Response loss C is generated based on segmentation reward information C and semantic accuracy reward information C. Furthermore, the electronic device 101 can determine whether the initial large language model 104 has finished training based on the multiple response losses. Finally, in response to determining that training has ended, the electronic device 101 can determine the initial large language model 104 as a large language model 105.

[0038] It should be noted that the aforementioned electronic device 101 can be either hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the electronic device is software, it can be installed in the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0039] It should be understood that Figure 1 The number of electronic devices shown is merely illustrative. Any number of electronic devices can be used depending on the implementation requirements.

[0040] Continue to refer to Figure 2 The diagram illustrates a flow 200 of some embodiments of a model training method according to the present disclosure. This model training method includes the following steps:

[0041] Step 201: Obtain target training data.

[0042] In some embodiments, the execution entity of the above model training method (e.g. Figure 1 The electronic device 101 shown can acquire target training data via a wired or wireless connection. The target training data includes: question answers and standard response information. The target training data can be training data to be used to train an initial large language model. The initial large language model can be a large language model that has not yet finished training. In practice, the large language model can support question-and-answer functions. The large language model can be a commercially available large language model. The question answer can be a question that requires a response from the large language model. For example, the question answer could be "How do I get from point A to point B?" The standard response information can be a semantically accurate answer to the question answer. For example, the standard response information could be "Take bus route 13 from point A to point B".

[0043] Step 202: Input the above-mentioned response questions into the initial large language model multiple times to obtain multiple response information.

[0044] In some embodiments, the executing entity can input the question into the initial large language model multiple times to obtain multiple response messages. Each time the question is input into the initial large language model, the corresponding response message from the multiple response messages is output. The response message can be the result of the initial large language model's answer to the question.

[0045] It should be noted that when the same question is input into the initial large language model multiple times, the output of the initial large language model may be different each time.

[0046] Step 203: For each of the above multiple response messages, perform the generation step:

[0047] Step 2031: Based on the above standard response information, determine the semantic accuracy information corresponding to the above response information.

[0048] In some embodiments, the executing entity can determine the semantically accurate information corresponding to the standard response information based on the standard response information. The semantically accurate information can be information indicating whether the content of the response is correct. Semantically accurate information can be one of the following: information indicating the semantic accuracy of the response content, or information indicating the semantic inaccuracy of the response content. In addition, the semantically accurate information can also be in numerical form. That is, the higher the numerical value of the semantically accurate information, the closer the content of the standard response information is to the content of the response information.

[0049] As an example, firstly, the aforementioned executing entity can determine a first response vector corresponding to the standard response information and a second response vector corresponding to the response information. The first response vector can represent the semantic feature information of the response content corresponding to the standard response information. The second response vector can also represent the semantic feature information of the response content corresponding to the response information. Then, the vector similarity between the first and second response vectors is determined. Finally, in response to determining that the vector similarity is higher than the target similarity value, semantically accurate information representing the semantic accuracy of the response content is generated. In response to determining that the vector similarity is not higher than the target similarity value, semantically accurate information representing the semantic inaccuracy of the response content is generated.

[0050] Step 2032: Segment the above response information to obtain segmented information.

[0051] In some embodiments, the executing entity may segment the response information to obtain segmented information. The segmented information may be processing information that segments the response information. For example, the segmented information may include individual response sub-information. Each response sub-information may be a segmented information block.

[0052] As an example, the aforementioned executing entity can segment the above response information using a sentence segmentation method to obtain segmented information. The sentence segmentation method can be a method that segments the information based on a complete sentence (e.g., using periods to determine if it is a complete sentence).

[0053] In some optional implementations of certain embodiments, the aforementioned executing entity can utilize at least one of the following methods to segment the response information, namely, a paragraph text similarity-based segmentation method, a paragraph segmentation method, a conjunction segmentation method, and a sentence segmentation method, to obtain segmented information. Specifically, the paragraph text similarity-based segmentation method can be a method that groups paragraphs with high similarity into one category based on the similarity between paragraph texts, thereby achieving text segmentation. The paragraph segmentation method can be a method of segmenting text according to paragraphs. The conjunction segmentation method can be a method of segmenting text according to conjunctions in the text. The sentence segmentation method can be a method of segmenting text by treating the entire sentence as a subtext.

[0054] Step 2033: Based on the above semantic accuracy information and the above segmentation information, generate segmentation reward information and semantic accuracy reward information.

[0055] In some embodiments, the aforementioned executing entity can generate segmentation reward information and semantically accurate reward information based on the aforementioned semantically accurate information and segmentation information. The segmentation reward information can be a reward value that provides feedback on the segmentation result corresponding to the segmentation information. In practice, in reinforcement learning, the reward value is the immediate feedback signal obtained when the agent interacts with the environment; its core function is to define the quality of behavior and guide policy optimization. The semantically accurate reward information can be a reward value that provides feedback on the accuracy of the semantic content of the response information. Both segmentation reward information and semantically accurate reward information can be in numerical form.

[0056] As an example, firstly, the accuracy value corresponding to the semantically accurate information is determined as the semantically accurate reward information. Then, the various response sub-information included in the segmentation information are identified. The semantic similarity between each response sub-information is determined, resulting in a semantic similarity set. Semantic similarity characterizes the degree of similarity in the semantic content of the corresponding information between two response sub-information pieces. Finally, the highest and lowest values ​​in the semantic similarity set are removed, resulting in a semantic similarity set after removal. The similarities in the semantic similarity set after removal are summed to obtain the segmentation reward information.

[0057] Step 2034: Generate response loss based on the segmentation reward information and semantic accuracy reward information described above.

[0058] In some embodiments, the aforementioned executing entity can generate a response loss based on the segmentation reward information and the semantic accuracy reward information. The response loss can characterize the loss of the response information under the semantic accuracy feature and the textual conciseness feature.

[0059] As an example, the aforementioned executing entity can perform a weighted summation of the segmented reward information and the aforementioned semantically accurate reward information to obtain the response loss.

[0060] In some optional implementations of certain embodiments, the aforementioned execution entity can generate a response loss based on the aforementioned segmentation reward information and semantically accurate reward information, including the following steps:

[0061] The first step is to perform a weighted summation of the above segmentation reward information and the above semantic accuracy reward information to obtain weighted reward information.

[0062] The second step is to normalize the weighted reward information to obtain normalized reward information.

[0063] As an example, firstly, the implementing entity can determine the average reward and standard deviation of each weighted reward information. Then, the average reward is subtracted from the weighted reward information to obtain the subtraction result. Finally, the subtraction result is divided by the standard deviation of the reward to obtain the normalized reward information.

[0064] The third step involves inputting the normalized reward information into the loss function corresponding to the objective reinforcement learning method to obtain the response loss. The objective reinforcement learning method can be the GRPO (Group Relative Policy Optimization) training model method. GRPO is a policy optimization-based reinforcement learning algorithm that optimizes the parameters of the policy network to enable the agent to obtain higher cumulative rewards in the environment. The loss function can be the conventional loss function corresponding to the GRPO training model method.

[0065] Step 204: Based on the multiple response losses, determine whether the initial large language model has finished training.

[0066] In some embodiments, the aforementioned execution entity may determine whether the initial large language model has finished training based on multiple response losses.

[0067] As an example, firstly, the aforementioned execution entity can weight multiple response loss information to obtain weighted loss information. Then, in response to determining that the values ​​of each item in the weighted loss information sequence corresponding to the weighted loss information tend towards the target loss value, the initial large language model training is determined to be complete. In response to determining that the values ​​of each item in the weighted loss information sequence corresponding to the weighted loss information do not tend towards the target loss value, the initial large language model training is determined to be complete. Here, the weighted loss information sequence consists of various weighted loss information generated within the current time and historical time periods prior to the current time. The weighted loss information items in the weighted loss information sequence are trained in chronological order.

[0068] Step 205: In response to determining the end of training, the initial large language model is identified as the large language model.

[0069] In some embodiments, in response to determining the end of training, the aforementioned execution entity may identify the aforementioned initial large language model as a large language model.

[0070] The above embodiments of this disclosure have the following beneficial effects: By using the model training methods of some embodiments of this disclosure, and utilizing the trained large language model, concise and semantically correct response information can be generated. Specifically, the reason why the relevant response information is not concise enough is that the supervised fine-tuning (SFT) method has a generalization upper limit problem and requires the manual construction of a simplified dataset, which is costly and may result in the response information not being concise enough. Based on this, the model training method of some embodiments of this disclosure first obtains target training data, which includes: response questions and standard response information. Here, the target training data is used as training samples to train the subsequent initial large language model. Then, the above response questions are input into the initial large language model multiple times to obtain multiple response information. Here, the segmentation reward and semantic reward of the initial large language model under each response information output for the same response question are determined to ensure the accuracy and conciseness of the corresponding output of the initial large language model. Next, for each of the above multiple response information, a generation step is performed: First, based on the above standard response information, the semantic accuracy information corresponding to the above response information can be accurately determined to facilitate the subsequent generation of semantic accuracy reward information. The second step involves segmenting the response information to obtain segmented information, which is then used to evaluate the conciseness of the response. The third step, based on the semantic accuracy information and the segmented information, accurately generates segmentation reward information and semantic accuracy reward information. These are used to generate the response loss for subsequent response information generation. The fourth step, based on the segmentation reward information and semantic accuracy reward information, accurately generates the response loss to characterize the difference between the response information under textual conciseness and semantic accuracy features. Furthermore, based on multiple response losses, it can be accurately determined whether the initial large language model has finished training. Finally, in response to the determination of training completion, the initial large language model is established as the large language model. In summary, by determining the segmentation reward information and semantic accuracy reward information corresponding to multiple response information output by the initial large language model, it is possible to determine whether the initial large language model has finished training, ensuring that the subsequently trained large language model outputs concise and semantically accurate response information.

[0071] Further reference Figure 3 The diagram illustrates flow 300 of some other embodiments of the model training method according to this disclosure. This model training method includes the following steps:

[0072] Step 301: Obtain target training data.

[0073] Step 302: Input the above-mentioned response questions into the initial large language model multiple times to obtain multiple response information.

[0074] Step 303: For each of the above multiple response messages, perform the generation step:

[0075] Step 3031: Determine the semantic accuracy information corresponding to the above-mentioned standard response information based on the above-mentioned standard response information.

[0076] Step 3032: Segment the above response information to obtain segmented information.

[0077] Step 3033: Select the target response information that corresponds to the semantically accurate information representation from the above multiple response information to obtain at least one target response information.

[0078] In some embodiments, the executing entity (e.g. Figure 1 The electronic device 101 shown can filter out the target response information that corresponds to the semantically accurate information from the above multiple response information to obtain at least one target response information. That is, the semantics corresponding to the target response information are the same as the semantics corresponding to the standard response information.

[0079] Step 3034: Select the smallest number of segments from the at least one number of segments corresponding to the at least one target response information mentioned above, and use it as the minimum number of segments.

[0080] In some embodiments, the executing entity may select the smallest number of segments from the at least one number of segments corresponding to the at least one target response information, and use this as the minimum number of segments. The segmentation information refers to the number of segments. The number of segments can be the number of times the response information is segmented. For example, segmenting the response information 5 times yields 6 response sub-information. The number of segments corresponding to the response information is "5". Each target response information has a corresponding number of segments. For example, at least one target response information may include: target response information A, target response information B, and target response information C. The number of segments corresponding to target response information A is 3. The number of segments corresponding to target response information B is 2. The number of segments corresponding to target response information C is 4. Therefore, the minimum number of segments is "2".

[0081] Step 3035: Based on the above semantically accurate information, the above minimum number of segments, and the above number of segments, generate segmentation reward information and semantically accurate reward information.

[0082] In some embodiments, the execution entity may generate segmentation reward information and semantically accurate reward information based on the semantically accurate information, the minimum number of segments, and the number of segments.

[0083] As an example, the aforementioned executing entities determine the semantically accurate information as the segmentation reward information. In response to determining the difference interval corresponding to the numerical difference between the minimum number of segments and the number of segments, the reward value corresponding to the difference interval is determined as the segmentation reward information. Here, there are multiple numerically consecutive difference intervals, and each difference interval has a unique corresponding segmentation reward information.

[0084] In some optional implementations of certain embodiments, the execution entity can generate segmentation reward information and semantically accurate reward information based on the semantically accurate information, the minimum number of segments, and the number of segments, including the following steps:

[0085] The first step, in response to determining that the aforementioned semantically accurate information represents semantic accuracy, is to set a first value as the semantically accurate reward information corresponding to the aforementioned response information. The first value can be a pre-set value representing semantic accuracy. For example, the first value can be the value "1".

[0086] The second step is to determine the difference between the minimum number of divisions and the number of divisions mentioned above, which will be used as the division reward information.

[0087] As an example, the aforementioned executing entity can subtract the number of segments from the minimum number of segments to obtain the difference in number, which can be used as the segment reward information.

[0088] Third, in response to determining that the semantically accurate information indicates semantic inaccuracy, a second value is set as the semantically accurate reward information corresponding to the above response information. The second value can be a pre-set value representing semantic inaccuracy. For example, the second value can be the value "0".

[0089] The fourth step is to determine the maximum value between the above numerical difference and the above second value as the split reward information.

[0090] Step 3036: Generate response loss based on the segmentation reward information and semantic accuracy reward information described above.

[0091] Step 304: Based on the multiple response losses, determine whether the initial large language model has finished training.

[0092] Step 305: In response to determining the end of training, the above initial large language model is identified as the large language model.

[0093] In some embodiments, the specific implementation of steps 301-302, 3031-3032, 3036, and 304-305, and their resulting technical effects, can be found in [reference needed]. Figure 2 Steps 201-202, 2031-2032, 2034, and 204-205 in the corresponding embodiments will not be repeated here.

[0094] from Figure 3 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 3 In some corresponding embodiments, the model training method process 300, by setting the number of segments, keeps the model with the smallest semantic units, thus shortening the model output while maintaining model performance.

[0095] Continue to refer to Figure 4 The diagram illustrates a flow 400 of some embodiments of a problem-solving method according to the present disclosure. The problem-solving method includes the following steps:

[0096] Step 401: Obtain the target logistics problem within the target logistics domain.

[0097] In some embodiments, the entity executing the above-described problem-solving method (e.g., a question-and-answer terminal) can acquire a target logistics problem within a target logistics domain. The target logistics domain can be a logistics domain supported by the question-and-answer terminal for question answering. The question-and-answer terminal can be a terminal deployed with a large language model to implement question responses. The target logistics problem can be a logistics-related question to be answered. For example, the target logistics problem could be "How should item A be processed so that it can be delivered to location A before the end of tomorrow's workday?"

[0098] Step 402: Input the above target logistics problem into the pre-trained large language model to obtain logistics response information.

[0099] In some embodiments, the aforementioned executing entity can input the target logistics problem into a pre-trained large language model to obtain logistics response information. The logistics response information can be the result of answering the target logistics problem. The large language model can be trained using a model training method. For example, the target logistics problem is "How to process item A so that it can be delivered to location A before the end of tomorrow's workday?" The corresponding logistics response information could be "Change the express delivery channel for item A from channel A to channel B, and print 'urgent' on the package corresponding to item A." The logistics response information can be in concise text format.

[0100] Step 403: Control the target robot to execute the logistics processing strategy according to the above logistics response information.

[0101] In some embodiments, the aforementioned executing entity can control the target robot to execute a logistics processing strategy according to the aforementioned logistics response information. The target robot can be a robot that executes a strategy based on the logistics response information. The logistics processing strategy can be various action execution information generated in response to the logistics response information. For example, the logistics response information could be "change the express delivery channel corresponding to item A from channel A to channel B, and print 'urgent' on the package corresponding to item A," and the logistics processing strategy could be "first adjust the channel, then print 'urgent'."

[0102] The above embodiments of this disclosure have the following beneficial effects: Through the problem-solving methods of some embodiments of this disclosure, a pre-trained large language model can be used to output concise and accurate logistics response information for the target logistics problem pair.

[0103] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a model training apparatus, which are similar to... Figure 2 Corresponding to the method embodiments shown, this model training device can be specifically applied to various electronic devices.

[0104] like Figure 5 As shown, a model training device 500 includes: a first acquisition unit 501, a first input unit 502, an execution unit 503, a first determination unit 504, and a second determination unit 505. The first acquisition unit 501 is configured to acquire target training data, which includes: answer questions and standard answer information; the first input unit 502 is configured to input the answer questions multiple times into an initial large language model to obtain multiple answer information; the execution unit 503 is configured to perform the following generation steps for each of the multiple answer information: determining the semantic accuracy information corresponding to the answer information based on the standard answer information; segmenting the answer information to obtain segmentation information; generating segmentation reward information and semantic accuracy reward information based on the semantic accuracy information and the segmentation information; generating an answer loss based on the segmentation reward information and the semantic accuracy reward information; the first determination unit 504 is configured to determine whether the initial large language model has finished training based on the multiple answer losses; and the second determination unit 505 is configured to determine the initial large language model as a large language model in response to the determination that training has ended.

[0105] In some optional implementations of some embodiments, the segmentation information is the number of segments; and the execution unit 503 may be further configured to: filter out target response information that corresponds to semantically accurate information from the plurality of response information to obtain at least one target response information; filter out the smallest segmentation number from the at least one segmentation number corresponding to the at least one target response information as the minimum segmentation number; and generate segmentation reward information and semantically accurate reward information based on the semantically accurate information, the minimum segmentation number, and the segmentation number.

[0106] In some optional implementations of certain embodiments, the execution unit 503 may be further configured to: in response to determining that the semantically accurate information represents semantic accuracy, set a first value as the semantically accurate reward information corresponding to the response information; determine the difference between the minimum number of segments and the number of segments as the segmentation reward information; in response to determining that the semantically accurate information represents semantic incorrectness, set a second value as the semantically accurate reward information corresponding to the response information; and determine the maximum value between the difference between the numbers and the second value as the segmentation reward information.

[0107] In some optional implementations of some embodiments, the execution unit 503 may be further configured to: perform weighted summation on the segmentation reward information and the semantic accuracy reward information to obtain weighted reward information; perform normalization on the weighted reward information to obtain normalized reward information; and input the normalized reward information into the loss function corresponding to the target reinforcement learning to obtain the response loss.

[0108] In some optional implementations of some embodiments, the execution unit 503 may be further configured to: segment the above-mentioned response information using at least one of a segmentation method based on paragraph text similarity, a paragraph segmentation method, a conjunction segmentation method, and a sentence segmentation method to obtain segmented information.

[0109] It is understandable that the units described in the model training device 500 are related to the reference... Figure 2 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the model training device 500 and the units contained therein, and will not be repeated here.

[0110] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a problem-solving apparatus, which are similar to... Figure 4 Corresponding to the method embodiments shown, this problem-solving apparatus can be specifically applied to various electronic devices.

[0111] like Figure 6 As shown, a problem processing device 600 includes: a second acquisition unit 601, a second input unit 602, and a control unit 603. The second acquisition unit 601 is configured to acquire a target logistics problem within a target logistics domain; the second input unit 602 is configured to input the target logistics problem into a pre-trained large language model to obtain logistics response information, wherein the large language model is generated based on a model training method; and the control unit 603 is configured to control a target robot to execute a logistics processing strategy according to the logistics response information.

[0112] It is understandable that the units described in the problem-solving apparatus 600 are related to the reference... Figure 4 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the problem processing apparatus 600 and the units contained therein, and will not be repeated here.

[0113] The following is for reference. Figure 7 It illustrates electronic devices suitable for implementing some embodiments of this disclosure (e.g., Figure 1 A schematic diagram of the structure of electronic device 101)700 in the middle. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0114] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory 702 or a program loaded from a storage device 708 into a random access memory 703. The random access memory 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, the read-only memory 702, and the random access memory 703 are interconnected via a bus 704. An input / output interface 705 is also connected to the bus 704.

[0115] Typically, the following devices can be connected to the input / output interface 705: input devices 706 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 707 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 708 including, for example, magnetic tape, hard disk, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 7 Each box shown can represent a device or multiple devices as needed.

[0116] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 709, or installed from a storage device 708, or installed from a read-only memory 702. When the computer program is executed by the processing device 701, it performs the functions defined in the methods of some embodiments of this disclosure.

[0117] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0118] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0119] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire target training data, wherein the target training data includes: answer questions and standard answer information; input the answer questions multiple times into an initial large language model to obtain multiple answer information; for each of the multiple answer information, perform the generation steps: determine the semantic accuracy information corresponding to the answer information based on the standard answer information; segment the answer information to obtain segmentation information; generate segmentation reward information and semantic accuracy reward information based on the semantic accuracy information and the segmentation information; generate answer loss based on the segmentation reward information and semantic accuracy reward information; determine whether the initial large language model has finished training based on the multiple answer losses; and, in response to determining that training has ended, identify the initial large language model as a large language model. The second acquisition unit is configured to acquire a target logistics problem within the target logistics domain; the second input unit is configured to input the target logistics problem into a pre-trained large language model to obtain logistics response information, wherein the large language model is generated based on a model training method; and the control unit is configured to control the target robot to execute a logistics processing strategy according to the logistics response information.

[0120] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0122] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a first acquisition unit, a first input unit, an execution unit, a first determination unit, and a second determination unit. The names of these units do not necessarily limit the specific unit; for example, the first acquisition unit may also be described as a "unit for acquiring target training data".

[0123] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0124] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any of the model training methods or problem-solving methods described above.

[0125] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A model training method, comprising: Obtain target training data, wherein the target training data includes: answer questions and standard answer information; The question to be answered is input into the initial large language model multiple times to obtain multiple answer messages; For each of the multiple response messages, perform the generation step: Based on the standard response information, determine the semantic accuracy information corresponding to the response information; The response information is segmented to obtain segmented information; Based on the semantic accuracy information and the segmentation information, segmentation reward information and semantic accuracy reward information are generated, wherein the segmentation information is the number of segments, including: selecting target response information that corresponds to semantic accuracy information from the plurality of response information to obtain at least one target response information; selecting the segmentation number with the smallest value from the at least one segmentation number corresponding to the at least one target response information as the minimum segmentation number; and generating segmentation reward information and semantic accuracy reward information based on the semantic accuracy information, the minimum segmentation number, and the segmentation number. Based on the segmentation reward information and semantically accurate reward information, a response loss is generated; Based on the loss of multiple responses, determine whether the initial large language model has finished training; In response to the determination that training has ended, the initial large language model is identified as the large language model.

2. The method according to claim 1, wherein, The step of generating segmentation reward information and semantic accuracy reward information based on the semantic accuracy information, the minimum number of segments, and the number of segments includes: In response to determining that the semantically accurate information represents semantic accuracy, a first value is set as the semantically accurate reward information corresponding to the response information; The difference between the minimum number of segments and the number of segments is determined as the segmentation reward information; In response to determining that the semantically accurate information represents semantically incorrect information, a second value is set as the semantically accurate reward information corresponding to the response information; The maximum value between the difference in the number and the second value is determined as the segmented reward information.

3. The method according to claim 1, wherein, The step of generating a response loss based on the segmentation reward information and the semantically accurate reward information includes: The segmentation reward information and the semantic accuracy reward information are weighted and summed to obtain weighted reward information; The weighted reward information is normalized to obtain normalized reward information; The normalized reward information is input into the loss function corresponding to the target reinforcement learning to obtain the response loss.

4. The method according to claim 1, wherein, The segmentation process of the response information to obtain segmented information includes: The response information is segmented using at least one of the following methods: paragraph text similarity-based segmentation, paragraph segmentation, conjunction segmentation, and sentence segmentation, to obtain segmented information.

5. A problem-solving method, comprising: Identify target logistics issues within the target logistics domain; The target logistics problem is input into a pre-trained large language model to obtain logistics response information, wherein the large language model is generated based on the method described in any one of claims 1-4; Control the target robot to execute the logistics processing strategy according to the logistics response information.

6. A model training device, comprising: The first acquisition unit is configured to acquire target training data, wherein the target training data includes: answer questions and standard answer information; The first input unit is configured to input the response question multiple times into the initial large language model to obtain multiple response messages; An execution unit is configured to perform the following generation steps for each of the plurality of response messages: determining the semantic accuracy information corresponding to the response message based on the standard response message; segmenting the response message to obtain segmentation information; generating segmentation reward information and semantic accuracy reward information based on the semantic accuracy information and the segmentation information, wherein the segmentation information is a number of segments, including: selecting target response messages that correspond to semantic accuracy information from the plurality of response messages to obtain at least one target response message; selecting the smallest segmentation number from at least one segmentation number corresponding to the at least one target response message as the minimum segmentation number; generating segmentation reward information and semantic accuracy reward information based on the semantic accuracy information, the minimum segmentation number, and the segmentation number; and generating a response loss based on the segmentation reward information and semantic accuracy reward information. The first determining unit is configured to determine whether the initial large language model has finished training based on multiple response losses; The second determining unit is configured to determine the initial large language model as a large language model in response to determining the end of training.

7. A problem processing apparatus, comprising: The second acquisition unit is configured to acquire the target logistics problem within the target logistics domain; The second input unit is configured to input the target logistics problem into a pre-trained large language model to obtain logistics response information, wherein the large language model is generated based on the method described in any one of claims 1-4; The control unit is configured to control the target robot to execute a logistics processing strategy in accordance with the logistics response information.

8. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

9. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.

Images