Model training method, task processing method, computer program product and equipment

Abstract

The invention provides a model training method, a task processing method, a computer program product and electronic equipment, and relates to the technical field of artificial intelligence, the method comprises the following steps: obtaining sample input information of a reasoning task, and splicing the sample input information with specified information represented by a non-reasoning prefix to obtain intermediate input information; analyzing the intermediate input information through a plurality of reasoning operation modes of a reasoning model, and determining a training sample; grouping the training samples, and when the reasoning accuracy of each group of training samples is greater than an accuracy threshold, triggering reasoning length penalty of the reasoning model to determine a reward function; adjusting model parameters of the reasoning model based on the reward function to obtain a trained reasoning model; the trained reasoning model is used for analyzing and processing the text information and / or the image information through a reasoning operation mode matched with the to-be-processed task to generate a processing result. The reasoning accuracy can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular, to a model training method, a task processing method, a computer program product, and an electronic device. BACKGROUND

[0002] In view of the overthinking problem of large language inference models in inference tasks, in the related art, the problem is generally handled by means of supervised learning optimization, reinforcement learning optimization, or prompt word control.

[0003] However, the above-mentioned manner reduces the inference accuracy, leading to premature shortening of inference and affecting the inference integrity of the model; in addition, prompt word control needs to be designed according to the task and the scene, and has poor generalization. SUMMARY

[0004] The present disclosure aims to provide a model training method, a task processing method, a computer program product, and an electronic device, thereby at least partially overcoming the problem of low inference accuracy caused by the limitations and defects of the related art.

[0005] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a model training method is provided, comprising: obtaining sample input information of an inference task, and splicing the sample input information with specified information represented by a no-inference prefix to obtain intermediate input information; the specified information is used to reduce redundant inference of an inference model during execution of the inference task; analyzing the intermediate input information through multiple inference running modes of the inference model to determine training samples; grouping the training samples, and triggering an inference length penalty of the inference model in the case that the inference accuracy of each group of training samples meets an accuracy threshold, to determine a reward function; adjusting model parameters of the inference model based on the reward function to obtain a trained inference model; wherein the trained inference model is used to analyze and process text information and / or image information of a to-be-processed task through an inference running mode matched with the to-be-processed task, to generate a processing result.

[0007] In an exemplary embodiment of the present disclosure, the analyzing the intermediate input information through multiple inference running modes of the inference model to determine training samples comprises: analyzing and processing the intermediate input information according to multiple inference running modes of the inference model to generate multiple answer information corresponding to the multiple inference running modes, to determine training samples; wherein the inference running modes include a zero-thinking sampling mode and a self-recovery inference sampling mode, and the inference chain lengths of the multiple inference running modes are different.

[0008] In an example embodiment of the present disclosure, the inference length penalty of the trigger inference model is determined to determine the reward function, including: for each training sample, obtaining the longest answer information and the shortest answer information of the sample input information in each training sample from the multiple answer information included in each training sample; determining the length difference between the longest answer information and the shortest answer information, and calculating the super-long ratio corresponding to the training sample according to the ratio of the length difference to the pre-configured penalty window length; determining the penalty coefficient of the training sample, and determining the penalty based on the super-long ratio and the penalty coefficient; determining the reward function according to the answer correctness indication function and the penalty.

[0009] In an example embodiment of the present disclosure, the determination of the penalty coefficient of the training sample includes: in the case that the inference accuracy is greater than or equal to the accuracy threshold, determining the penalty coefficient according to the inference accuracy, a fixed parameter and the accuracy threshold.

[0010] In an example embodiment of the present disclosure, the adjustment of the model parameters of the inference model based on the reward function to obtain a trained inference model includes: determining a reward value through the reward function; performing reinforcement learning on the inference model according to the reward value to determine the trained inference model.

[0011] In an example embodiment of the present disclosure, the reinforcement learning on the inference model according to the reward value to determine the trained inference model includes: determining an advantage according to the reward value, and determining a policy gradient according to the advantage; adjusting a policy parameter according to the gradient direction of the policy gradient to obtain the trained inference model.

[0012] According to an aspect of the present disclosure, a task processing method is provided, including: obtaining text information and / or image information of a task to be processed; inputting the text information and / or image information into a trained inference model, and performing model analysis on the text information and / or image information through an inference running mode matched with the task to be processed to obtain a processing result corresponding to the input information; wherein the trained inference model is trained according to the model training method of any one of the above.

[0013] In an example embodiment of the present disclosure, the model analysis on the text information and / or image information according to the inference running mode matched with the task to be processed to obtain a processing result corresponding to the input information comprises: determining a task difficulty of the task to be processed; determining an inference running mode matched with the task to be processed according to the task difficulty, and performing model analysis on the text information and / or image information of the task to be processed according to an inference chain corresponding to the inference running mode to determine a processing result corresponding to the text information and / or image information.

[0014] According to an aspect of the present disclosure, a computer program product is provided, which, when executed by a processor, implements the model training method of any one of the above or the task processing method of any one of the above.

[0015] According to an aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the model training method of any one of the above or the task processing method of any one of the above via execution of the executable instructions.

[0016] In the technical solution provided by the embodiments of the present disclosure, on the one hand, the model training is implemented by concatenating specified information for sample input information and triggering inference length penalty when the inference accuracy is greater than the accuracy threshold, which avoids the problem of model training in related technologies that all tasks are subjected to inference chain reduction, can realize adaptive compression and recovery of inference model inference chain, avoids redundant inference, and improves the accuracy of model training. On the other hand, the trained inference model only triggers inference length penalty to impose a penalty reward on the inference length of the inference model after the inference accuracy reaches the accuracy threshold, thereby preventing the inference model from prematurely shortening the inference chain when the accuracy has not yet met the standard, and the inference model can select an inference running mode according to a task to be processed and adjust inference resource allocation, thereby improving inference integrity and inference accuracy. On the other hand, manual modification of prompt words is avoided, which can be applied to any inference scene and model, increases universality, expansibility and flexibility, and improves generalization ability.

[0017] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure. It is to be understood that the drawings are only schematic, and that they do not necessarily represent a limiting case of the embodiments of the present disclosure. For the purpose of explanation and clearness, elements in the drawings have not necessarily been drawn on scale. In the drawings:

[0019] Figure 1 A schematic diagram of a system architecture of a model training method or a task processing method to which embodiments of the present disclosure can be applied is shown.

[0020] Figure 2 A flowchart of a model training method in an embodiment of the present disclosure is shown schematically.

[0021] Figure 3 A flowchart of a method for determining a reward function in an embodiment of the present disclosure is shown schematically.

[0022] Figure 4 A flowchart of a method for adaptive automatic reply inference training in an embodiment of the present disclosure is shown schematically.

[0023] Figure 5 A flowchart of a task processing method in an embodiment of the present disclosure is shown schematically.

[0024] Figure 6 A schematic diagram of a simple instruction distribution and a difficult instruction distribution for generating an inference model in an embodiment of the present disclosure is shown schematically. DETAILED DESCRIPTION

[0025] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided as non-limiting examples so that this disclosure will be thorough and complete, and will fully convey the scope of the example implementations to those skilled in the art. The described features, structures, or characteristics can be combined in one or more implementations. In the following description, numerous specific details are provided to give a thorough understanding of implementations of the present disclosure. One skilled in the relevant art will recognize, however, that the implementations of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures have not been described in detail so as not to obscure the aspects of the present disclosure.

[0026] Further, the accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification. The drawings are not necessarily to scale, the dimensions of certain features being exaggerated from others for illustrative purposes. Same reference numerals in different drawings represent same or similar elements unless otherwise specified. Some of the blocks in the drawings represent functional entities that can be realized in software, hardware, or a combination of both. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0027] In the related art, for the problem of "overthinking" of large language reasoning models (LRMs) in reasoning tasks, generally, the problem is handled through supervised learning optimization, reinforcement learning optimization or prompt word control. Among them, supervised learning optimization refers to that through supervised fine-tuning and direct preference optimization, in combination with a specially designed short chain reasoning data set, fine-grained suppression of model output reasoning length is realized. Supervised learning optimization reduces the reply length of the model on different instruction distributions without distinction, resulting in a decrease in the accuracy rate of instructions that require deep thinking and poor adaptability; the training form of supervised fine-tuning reduces the reply diversity of the model when sampling, making it more difficult to sample better solutions. Reinforcement learning optimization refers to introducing a reasoning length-based reward in reinforcement learning training under the guidance of accuracy reward, to encourage the model to generate concise reasoning chains while maintaining a certain accuracy rate, trying to strike a balance between performance and efficiency. However, the length reward in reinforcement learning often leads to premature shortening of reasoning when the accuracy rate has not yet met the requirements, affecting the reasoning integrity of complex problems. Prompt word control refers to using different prompt word designs to guide the model to generate more concise reasoning chains to reduce unnecessary reasoning steps. However, prompt word control relies on artificially designed control mechanisms or prompt word engineering, and it is difficult to realize the adaptive reasoning resource allocation of the model for different task types and task difficulties, and the generalization and practicality are limited.

[0028] To solve the technical problems in the related art, in some embodiments of the present disclosure, a model training method is provided, which can be applied to training a reasoning model used for any reasoning task, to realize optimization of redundant reasoning.

[0029] Figure 1 A schematic diagram of a system architecture of a model training method, a task processing method and apparatus to which some embodiments of the present disclosure can be applied is shown.

[0030] In some embodiments, the system architecture 100 relied on by the model training and the task processing can include a terminal 110 and a server 120. The terminal 110 can include any type of terminal. The terminal 110 and the server 120 can transmit information through a network 130. In the model training phase, the server 120 is used to analyze the sample input information uploaded by the terminal 110 to output answer information, and then obtain training samples based on the sample input information and the answer information, determine a reward function according to the training samples, and implement the training process of the inference model according to the reward function. In the task processing phase, the text information and / or image information of the task to be processed can be analyzed, so that the trained inference model calls the target inference running mode related to the task difficulty of the task to be processed to analyze and process the text information and / or image information, and obtains the corresponding processing result.

[0031] It should be noted that in the above model training method and task processing method, the server 120 can be any type of server or server cluster. The terminal 110 can be any type of device. The above model training method and task processing method can be completely executed by the server; or partially executed by the server and partially executed by the terminal, which is not limited here.

[0032] Next, each step in the model training method in the embodiments of the present disclosure will be described in detail with reference to FIG. 2. Figure 2

[0033] In step S210, sample input information of an inference task is obtained, and the sample input information is spliced with specified information represented by a non-inference prefix to obtain intermediate input information; the specified information is used to reduce redundant inference of the inference model in the process of executing the inference task.

[0034] In some embodiments, the inference task can be any type of inference task, which refers to a task of extracting features from input information and generating predicted output based on the features. The inference task can be a classification task, a detection task, a generation task, a multi-modal inference task, etc. The sample input information can be input data corresponding to the inference task, which can be determined according to the inference task itself, for example, it can be sample text information or sample image information, or multi-modal information composed of sample text information and sample image information, etc. The number of inference tasks can be at least one, and can be a task that has been executed for training the model. In the embodiments of the present disclosure, the input of each inference task can be structured to obtain sample input information. Each sample input information of the inference task can be represented as x i ​In addition, specified information can be injected into the sample input information to obtain intermediate input information. The specified information can be information for reducing redundant reasoning of the inference model during execution of the inference task. The specified information can be a No-Thinking prefix, which can be represented as p term The redundant reasoning can be unnecessary reasoning or repeated reasoning of the inference model during the reasoning process, and the like.

[0035] For the sample input information of each inference task, the sample input information can be spliced with the specified information represented by the No-Thinking prefix to obtain intermediate input information For example, the No-Thinking prefix can be spliced at any position of the sample input information, and the splicing position is not specifically limited. For example, for each inference task input x i , the No-Thinking prefix p term is spliced to form a new input as intermediate input information The intermediate input information can be, for example, “Okay, I have finished thinking.”

[0036] In step S220, the intermediate input information is analyzed by multiple reasoning running modes of the inference model to determine the training sample.

[0037] In the embodiments of the present disclosure, the intermediate input information obtained by splicing the intermediate input information and the specified information can be input into the inference model, and the intermediate input information can be analyzed and processed by multiple reasoning running modes of the inference model to generate a training sample. The reasoning chain lengths of the multiple reasoning running modes are different. The inference model can perform multiple model analyses on the intermediate input information to obtain multiple answer information, and determine the training sample according to the multiple answer information. The reasoning running modes used in the multiple model analyses can be the same or different.

[0038] The inference model includes multiple reasoning running modes, which can include a zero-thinking sampling mode and a self-recovery reasoning sampling mode. The self-recovery reasoning sampling mode can also be understood as a self-recovery supplemental thinking mode. The zero-thinking sampling mode refers to prompting the inference model to directly output an answer by using the No-Thinking prefix, thereby reducing the redundant reasoning steps to the greatest extent and generating a zero-thinking answer output. The self-recovery reasoning sampling mode refers to implicitly activating an internal reasoning path when generating answer information, automatically supplementing the reasoning steps required for the inference task, and generating a self-recovery thinking supplemented answer output.

[0039] For the inference task, inference prediction can be performed according to two inference running modes of the inference model, to predict a plurality of answer information corresponding to the sample input information. The inference task can be performed multiple times, and after multiple inferences, at least one answer information can be generated for each inference running mode, and the answer information generated each time can be the same or different.

[0040] The sample input information can be a sample question, and the generated answer information can be answer information predicted for the sample question. Based on this, the training sample can include sample input information and answer information, that is, the training sample can include a sample question and a plurality of answer information corresponding to the sample question. The plurality of answer information can contain different numbers of words, and the lengths of the answer information can also be different.

[0041] In the embodiments of the present disclosure, by utilizing the No-Thinking prefix and the inherent ability of the inference model, adaptive compression and recovery of the inference chain of the inference model are realized, and the perception and adaptability of the inference model to task difficulty are improved.

[0042] Next, in step S230, the training samples are grouped, and in the case that the inference accuracy of each group of training samples meets the accuracy threshold, the inference length penalty of the inference model is triggered to determine the reward function.

[0043] In the embodiments of the present disclosure, after the training samples are determined, the training samples can be grouped according to the same sample input information, and the answer information corresponding to the same sample question is divided into a group. Further, the answer information obtained by using different inference running modes for the same sample input information in the training samples can be compared with the actual answer information of the sample input information, and in the case that the comparison result is consistent or similar, the answer information is determined to be correct. The inference accuracy of each group of training samples is determined according to the proportion of correct answer information, and the inference accuracy of each group of training samples is statistically determined in real time

[0044] The inference accuracy meets the accuracy threshold, which means that the inference accuracy is greater than or equal to the accuracy threshold. In the case that the inference accuracy of each group of training samples is greater than or equal to the accuracy threshold, the inference length penalty of the inference model can be activated, thereby preventing the inference chain of the inference model from being compressed too early when the inference accuracy is not up to standard, and improving the accuracy. The accuracy threshold can be a value set according to actual needs, for example, it can be 0.6 or 0.8, etc., which is not limited here.

[0045] In some embodiments, by activating the inference length penalty, the reward function can be determined, and then the model parameters of the inference model are adjusted according to the reward function to obtain a trained inference model.

[0046] Figure 3 A flowchart for determining the reward function is shown in FIG. 1, and the steps are described in detail below with reference to Figure 3 The steps mainly include the following steps:

[0047] In step S310, for each training sample, a super-length ratio corresponding to the training sample is calculated.

[0048] In step S320, a penalty coefficient of the training sample is determined, and a penalty is determined based on the super-length ratio and the penalty coefficient.

[0049] In step S330, the reward function is determined according to the answer correctness indication function and the penalty.

[0050] In the embodiments of the present disclosure, for each training sample, a super-length ratio of the training sample can be calculated The super-length ratio is used to represent the proportion of the length difference between the longest answer information and the shortest answer information in the training sample. For example, the longest answer information and the shortest answer information corresponding to the sample input information can be determined from the multiple answer information contained in the training sample, and the lengths of the longest answer information and the shortest answer information can be determined according to the number of words contained in the longest answer information and the number of words contained in the shortest answer information. It should be noted that the longest answer information and the shortest answer information here refer to the correct answer information. Further, the length difference between the longest answer information and the shortest answer information can be calculated, and the super-length ratio can be determined according to the ratio of the length difference to a pre-configured penalty window length. The pre-configured penalty window length can be determined according to actual needs, for example, it can be 100 or other numerical values. Based on this, the super-length ratio can be calculated in the following way:

[0051]

[0052] wherein, L correct_shortest is the shortest length of the answer information in the group of training samples, i.e., the length of the shortest answer information; L i is the length of the longest answer information in the group of training samples; L window is a fixed penalty window length.

[0053] Further, a penalty coefficient of the training sample can be determined. The penalty coefficient can be used to represent the adjustment range of the regularization process to the error, or can be used to represent the penalty intensity, which mainly functions to limit the overfitting of the model to the training data by controlling the weight of the regularization term, thereby improving the generalization ability of the model.

[0054] In the embodiments of the present disclosure, the penalty coefficient can be determined as 0 based on a comparison result of the inference accuracy and the accuracy threshold, or the penalty coefficient can be determined according to the inference accuracy, a fixed parameter and the accuracy threshold. For example, if the comparison result is that the inference accuracy is less than the accuracy threshold, the penalty coefficient is 0. In the case that the inference accuracy is greater than or equal to the accuracy threshold, the penalty coefficient is determined according to the inference accuracy, the fixed parameter and the accuracy threshold. For example, the fixed parameter can include a first parameter and a second parameter, the first parameter can be a scaling factor, and the second parameter can be a constant. Based on this, the difference between the inference accuracy and the accuracy threshold can be added to the second parameter to obtain a first addition result. The first parameter is multiplied by the first addition result to obtain a multiplication result, and the ratio between the multiplication result and a second addition result is further calculated to determine the penalty coefficient. The second addition result can be obtained by adding the difference between a preset value and the accuracy threshold to the second parameter, and the preset value can be 1. The penalty coefficient can be positively correlated with the inference accuracy, for example, as the inference accuracy increases, the penalty coefficient increases. The penalty coefficient can be calculated according to formula (2):

[0055]

[0056] After the super-long ratio and the penalty coefficient are determined, the penalty can be determined based on the super-long ratio and the penalty coefficient. The penalty generally refers to a regularization term, which functions to suppress the excessive value of the model parameter by adding an additional cost term, thereby preventing the model from overfitting the training data. In the embodiments of the present disclosure, the penalty can be determined according to the product of the super-long ratio and the penalty coefficient.

[0057] Further, the reward function can be determined according to the answer correctness indication function and the calculated penalty. Specifically, the answer correctness indication function and the penalty can be subtracted to obtain the reward function. The answer correctness indication function refers to a function for indicating whether certain answer information is correct. The output of the answer correctness indication function can be 1 (indicating correct) or 0 (indicating incorrect). The calculation method of the reward function can be represented as follows:

[0058]

[0059] wherein, represents the answer correctness indication function, and a is a dynamically adjusted penalty coefficient, represents the penalty.

[0060] In step S240, the model parameters of the inference model are adjusted based on the reward function to obtain a trained inference model; the trained inference model is used to analyze and process the text information and / or image information of a to-be-processed task through an inference running mode matched with the to-be-processed task, and generate a processing result.

[0061] In the embodiments of the present disclosure, the reward function guides the inference model to make the optimal decision in a complex environment by defining the reward values of different states and actions. The reward function directly determines the goal of the inference model, and the reward is the feedback obtained by the inference model immediately after performing an action, which is used to adjust the behavior strategy of the inference model to make it more optimized in decision-making.

[0062] After the reward function is determined, the reward value can be determined according to the reward function. Further, the inference model can be trained by reinforcement learning algorithm according to the reward value to obtain a trained inference model.

[0063] In some embodiments, the process of reinforcement learning of the inference model according to the reward value mainly includes: determining the advantage according to the reward value, and determining the policy gradient according to the advantage; adjusting the model parameters of the inference model according to the gradient direction of the policy gradient to obtain a trained inference model. Illustratively, the inference model can obtain the current state from the environment, select an action according to the current policy, return the action to the environment, and substitute it into the reward function to obtain the reward value. The target value can be calculated according to the reward value, which can be the advantage. The advantage is used to measure the good or bad degree of the current action relative to the average level. The advantage can be a generalized advantage estimation, which can be determined according to the reward value, the discount factor and the state value function. The policy gradient is determined according to the target value, which is used to measure the influence of the change of the policy parameter on the reward value. The policy parameter refers to the learnable model parameter of the inference model, such as the weight and bias of the inference model.

[0064] The policy parameter is adjusted according to the gradient direction until the reward mean or the change of the policy gradient is less than a threshold. After multiple iterations, the policy parameter is optimized to a state that can maximize the long-term reward, and a trained inference model can be obtained. By adjusting the model parameters of the inference model using the reward function, the redundant inference can be maximally compressed under the premise of ensuring the inference accuracy, the computational complexity is reduced, and the inference efficiency is improved.

[0065] In the embodiments of the present disclosure, the accuracy reward is used to guide the training, and the dynamic inference length penalty based on the accuracy rate is used. By adjusting the grouping accuracy rate and the dynamic penalty coefficient, the dynamic optimal balance of the inference efficiency and the accuracy rate is achieved, and the "short but wrong" or "accurate but lengthy" phenomenon is prevented. By combining the explicit inference inhibition and the implicit self-recovery mechanism with the dynamic inference length penalty based on the accuracy rate, the inference model can adaptively allocate inference resources for different task difficulties. Experimental results show that in multiple benchmark tests, the ASRR framework significantly reduces the inference overhead (up to 32.5%), and the accuracy loss is very small (less than 1.2%).

[0066] It should be noted that since the input training samples contain answer information of different inference running modes, the trained inference model can also automatically identify the inference running mode, so as to select the inference running mode matched with the to-be-processed task to perform model analysis processing on the text information and / or image information of the to-be-processed task, generate a processing result matched with the text information and / or image information and conforming to the inference running mode, so as to realize the function of the to-be-processed task.

[0067] Figure 4 An illustrative diagram of adaptive automatic recovery of inference training is shown in FIG. 1, which mainly includes a display inhibition prefix injection stage, a model generation stage, a reinforcement learning training stage, and a model inference stage. In the display inhibition prefix injection stage, the input of the inference task is structured and a No-Thinking prefix is injected to generate sample input information, and the specified information represented by the No-Thinking prefix is used to explicitly inhibit the redundant inference of the model. Figure 4

[0068] In the model generation stage, the sample input information obtained by splicing the sample input information with the specified information is input into the inference model, which can include multiple inference running modes such as zero-thinking sampling mode and self-recovery inference sampling mode. For the sample input information of the inference task, if the sample input information is a sample question, at least one answer output corresponding to the zero-thinking sampling mode can be output, and at least one answer output corresponding to the self-recovery inference sampling mode can be output. In the reinforcement learning training stage, the inference accuracy of each training sample composed of the sample question and the answer output can be determined. In the case where the inference accuracy of each training sample is greater than or equal to the accuracy threshold, the dynamic inference length penalty is enabled. Illustratively, for each training sample, the length difference between the longest answer information and the shortest answer information in the multiple answer information of the sample question is determined, and the ratio of the length difference to the preconfigured penalty window length is determined to obtain the super-long ratio corresponding to the training sample. Further, in the case where the inference accuracy is greater than or equal to the accuracy threshold, the penalty coefficient is determined according to the logical processing result of the inference accuracy, the fixed parameter, and the accuracy threshold. And the penalty can be determined according to the product of the super-long ratio and the penalty coefficient. After obtaining the penalty, the reward function can be determined according to the answer correctness indication function and the penalty, and the reward value can be determined based on the reward function. Based on this, the advantage can be calculated according to the reward value, the strategy gradient can be determined according to the advantage, the strategy parameter can be adjusted according to the gradient direction to maximize the reward, and the model parameters of the inference model can be determined when the reward mean or the strategy gradient changes reaches a threshold, thereby obtaining the trained inference model.

[0069] After obtaining the trained inference model, in the model inference stage, the text information and / or image information corresponding to the to-be-processed task can be input into the trained inference model to obtain a processing result corresponding to the text information and / or image information.​

[0070] The technical solutions in the embodiments of the present disclosure significantly reduce the reasoning overhead of simple tasks by accurately gating rewards, while ensuring the reasoning integrity and accuracy of complex tasks, achieving a balance between reasoning efficiency and reasoning accuracy. The diversity loss caused by supervised fine-tuning is avoided, and better performance is achieved in safety testing, reducing the potential risks brought by redundant reasoning, increasing diversity, and improving safety. It can be seamlessly integrated into the training process of mainstream large language models, suitable for various reasoning scenarios and model architectures, and has good universality and scalability. Without relying on artificially designed prompt words or control structures, the reasoning model can adaptively allocate reasoning resources matching the task type and task difficulty of the task to be processed, improving the generalization ability and actual application effect, and has strong adaptability.

[0071] In the embodiments of the present disclosure, a task processing method is also provided, which can include the following steps as shown in Figure 5

[0072] Step S510, obtaining text information and / or image information of a task to be processed;

[0073] Step S520, inputting the text information and / or image information into the trained reasoning model, and performing model analysis on the text information and / or image information through a reasoning running mode matching the task to be processed, to obtain a processing result corresponding to the text information and / or image information.

[0074] In the embodiments of the present disclosure, the task to be processed can be any type of reasoning task to be executed. The task to be processed can be a task instruction. The text information and / or image information of the task to be processed can be text information or image information, or multi-modal information composed of text information and image information.

[0075] After obtaining the text information and / or image information, the trained reasoning model can automatically identify the task difficulty of the task to be processed, and then automatically select and switch a reasoning running mode matching the task difficulty of the task to be processed from a plurality of reasoning running modes corresponding to the trained reasoning model. Simple tasks are directly output, and complex tasks are automatically supplemented with reasoning chains. The reasoning running mode matching the task to be processed can be a zero-thinking sampling mode or a self-recovery reasoning sampling mode. After determining the reasoning running mode matching the task to be processed, the text information and / or image information can be analyzed based on the reasoning chain corresponding to the reasoning running mode matching the task to be processed, to obtain a processing result.

[0076] ​In the embodiments of the present disclosure, the inference model can dynamically adjust the inference resource allocation according to the task difficulty, suppress redundant inference in simple tasks, and ensure the inference integrity of complex tasks, thereby effectively reducing the computational overhead while maintaining or even improving the inference accuracy and security of the model.

[0077] Exemplarily, the text information and / or image information of the to-be-processed task is input into the trained inference model, and the trained inference model selects the zero-thought sampling mode to obtain a simple instruction distribution as the processing result according to the task difficulty of the to-be-processed task represented by the text information and / or image information, or adopts the self-recovery inference sampling mode to obtain a difficult instruction distribution as the processing result.

[0078] For the inference running mode matching the to-be-processed task being the self-recovery inference sampling mode, the text information and / or image information can be subjected to word segmentation processing to obtain text features. Further, the text features can be encoded to obtain encoded features. The encoded features are subjected to attention weight matrix calculation, weighted summation and multi-head parallel calculation to generate multi-head attention output. Then, the multi-head attention output is subjected to feedforward neural network, combined with residual connection and layer normalization to realize feature extraction. Finally, the extracted features are decoded by the decoder to generate answer information corresponding to the input information as the processing result according to the features. The simple instruction distribution generated according to the zero-thought sampling mode and the difficult instruction distribution generated according to the self-recovery inference sampling mode can be as shown in Figure 6

[0079] ​In the embodiments of the present disclosure, the adaptive automatic recovery mechanism utilizes the self-recovery capability of the inference model when generating answers, allowing the model to automatically supplement necessary implicit reasoning for complex tasks without explicit complete reasoning, further improving the performance of the model on complex problems. Through the perception training of the model on the difficulty of the problem and the reinforcement of implicit automatic recovery, the length of the inference chain is dynamically adjusted, so that the model actively reduces the inference steps to improve efficiency when facing simple tasks, and retains sufficient inference process to ensure accuracy when facing complex tasks, avoiding the problem of inference length compression for all tasks under the control of supervised learning and prompt words. In reinforcement learning training, only when the model reaches the preset accuracy threshold in the current problem group, the penalty reward is applied to the inference length, so as to prevent the model from shortening the inference chain too early when the accuracy has not yet met the standard, and to balance the inference efficiency and the problem solving ability of complex tasks. Without relying on artificially designed prompt words or control structures, the model can adaptively allocate inference resources according to different task types and difficulties, improving the generalization ability and actual application effect. Through the accuracy gating reward, the inference overhead of simple tasks is significantly reduced, while the inference integrity and accuracy of complex tasks are guaranteed. The diversity loss caused by supervised fine-tuning is avoided, and better performance is achieved in safety testing, reducing the potential risks caused by redundant reasoning. The framework can be seamlessly integrated into the training process of mainstream large language models, and is suitable for various inference scenarios and model architectures, with good universality and expandability.

[0080] In some embodiments of the present disclosure, a model training apparatus is provided, which comprises an input splicing module, a training sample determination module, a reward function determination module, and a parameter adjustment module, wherein:

[0081] The input splicing module is configured to obtain sample input information of an inference task, splice the sample input information with specified information represented by a non-inference prefix to obtain intermediate input information, and the specified information is used to reduce redundant reasoning of the inference model during execution of the inference task.

[0082] The training sample determination module is configured to analyze the intermediate input information through multiple inference running modes of the inference model, and determine training samples.

[0083] The reward function determination module is configured to group the training samples, trigger an inference length penalty of the inference model when the inference accuracy of each group of training samples meets an accuracy threshold, and determine a reward function.

[0084] The parameter adjustment module is configured to adjust model parameters of the inference model based on the reward function to obtain a trained inference model, wherein the trained inference model is configured to analyze and process text information and / or image information of a to-be-processed task through an inference running mode matched with the to-be-processed task, and generate a processing result.

[0085] In an example embodiment of the present disclosure, the analyzing the intermediate input information through the plurality of inference running modes of the inference model to determine the training sample includes: processing the intermediate input information according to the plurality of inference running modes of the inference model to generate a plurality of answer information corresponding to the plurality of inference running modes, so as to determine the training sample; wherein the inference running mode includes a zero thinking sampling mode and a self-recovery inference sampling mode, and the inference chain lengths of the plurality of inference running modes are different.

[0086] In an example embodiment of the present disclosure, the triggering the inference length penalty of the inference model to determine the reward function includes: for each training sample, obtaining the longest answer information and the shortest answer information of the sample input information from a plurality of answer information included in each training sample; determining the length difference between the longest answer information and the shortest answer information, and calculating the overlong ratio corresponding to the training sample according to the ratio of the length difference to a pre-configured penalty window length; determining the penalty coefficient of the training sample, and determining the penalty based on the overlong ratio and the penalty coefficient; and determining the reward function according to the answer correctness indication function and the penalty.

[0087] In an example embodiment of the present disclosure, the determining the penalty coefficient of the training sample includes: in a case where the inference accuracy is greater than or equal to an accuracy threshold, determining the penalty coefficient according to the inference accuracy, a fixed parameter, and the accuracy threshold.

[0088] In an example embodiment of the present disclosure, the adjusting the model parameters of the inference model based on the reward function to obtain a trained inference model includes: determining a reward value through the reward function; and performing reinforcement learning on the inference model according to the reward value to determine the trained inference model.

[0089] In an example embodiment of the present disclosure, the performing reinforcement learning on the inference model according to the reward value to determine the trained inference model includes: determining an advantage according to the reward value, and determining a policy gradient according to the advantage; adjusting a policy parameter according to the gradient direction of the policy gradient to obtain the trained inference model.

[0090] According to an aspect of the present disclosure, a task processing apparatus is provided, which comprises an input information acquisition module and a model inference module, wherein the input information acquisition module is configured to acquire text information and / or image information of a task to be processed; the model inference module is configured to input the text information and / or image information into a trained inference model, perform model analysis on the text information and / or image information through an inference running mode matched with the task to be processed, and obtain a processing result corresponding to the text information and / or image information; and the trained inference model is trained according to the model training method in any one of the above aspects.

[0091] In an exemplary embodiment of the present disclosure, the model analysis on the text information and / or image information through the inference running mode matched with the task to be processed to obtain the processing result corresponding to the text information and / or image information comprises: determining a task difficulty of the task to be processed; determining the inference running mode matched with the task to be processed according to the task difficulty, and performing model analysis on the text information and / or image information of the task to be processed according to an inference chain corresponding to the inference running mode to determine the processing result corresponding to the text information and / or image information.

[0092] It should be noted that the specific details of each part of the above model training apparatus and task processing apparatus have been described in detail in the part of the embodiments of the method, and the undisclosed details can be referred to the embodiment content of the method part, and thus will not be described again.

[0093] The exemplary embodiments of the present disclosure also provide an electronic device. The electronic device can be the terminal device or the server described above. Generally, the electronic device can comprise a processor and a memory, the memory is configured to store executable instructions of the processor, and the processor is configured to execute the above task processing method by executing the executable instructions. In addition, the electronic device can further comprise a display for displaying an operation interface.

[0094] In the following, the electronic device is exemplarily illustrated in the form of a general computing device. The electronic device is only an example, and should not bring any limitation to the function and use range of the embodiments of the present disclosure.

[0095] The components of the electronic device can include but are not limited to the above at least one processing unit, the above at least one storage unit, a bus connecting different system components including the storage unit and the processing unit, and a display unit.

[0096] The storage unit stores program codes which can be executed by the processing unit, so that the processing unit executes the steps described in the above "Exemplary Method" section according to various exemplary embodiments of the present disclosure. For example, the processing unit can execute the steps as shown in the above "Exemplary Method" section. Figure 2

[0097] The storage unit can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) and / or a cache storage unit, and can further include a read-only memory (ROM).

[0098] The storage unit can further include programs / utilities having a set of (at least one) program modules, such as an operating system, one or more application programs, other program modules, and program data, each of which can include implementation of a network environment or some combination of these examples.

[0099] The bus can be one or more of several types of bus structures including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit bus, or a local bus using any of a variety of bus architectures.

[0100] The electronic device can also communicate with one or more external devices such as a keyboard or a pointing device, through an I / O interface. Furthermore, the electronic device can communicate with one or more devices that enable a user to interact with the electronic device, and / or one or more devices (e.g., a router, a modem, a switch, a bridge, a server, etc.) that enable the electronic device to communicate with one or more other computing devices. Such communication can occur via an I / O interface. Still yet, the electronic device can communicate with one or more networks (such as one or more local area networks (LANs) and / or wide area networks (WANs) such as the Internet) through a network adapter. As depicted, the network adapter is communicatively coupled to the other components of the electronic device through a bus. It should be appreciated that the network adapter can be directly coupled to the other components in the electronic device, rather than being communicatively coupled via a bus. The network adapter can implement one or more communication protocols over one or more networks to communicate with other computing devices.

[0101] It should be noted that some embodiments of the present disclosure further provide a computer program product. The computer program product includes a computer program, and the computer program is executed by a processor to implement the above method.

[0102] ​In an embodiment, the computer program product can be a tangible product that contains the computer program, such as a computer-readable storage medium storing the computer program. The computer-readable storage medium can be a storage medium based on electric, magnetic, optical, electromagnetic, infrared, etc. signals, including but not limited to random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, Flash memory (Flash), mechanical hard disk (HDD), solid-state disk (SSD), etc. For example, the computer program product can be implemented as a non-volatile storage medium storing the computer program, such as a read-only memory (ROM), a Nand Flash, etc.

[0103] In an embodiment, the computer program product can be an intangible product that contains the computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, an installation package, etc. digital file storing the computer program.

[0104] The code of the computer program can be written in one or more programming languages. The programming language can be, for example, C, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (for example, through an Internet connection provided by an operator).

[0105] The computer program can be carried or transmitted by electric, magnetic, optical, electromagnetic, infrared, etc. signals. The electronic device can convert the signals carrying the computer program into digital signals, and then run the computer program. When the computer program is running on the electronic device, the code thereof is used to make the electronic device execute (more specifically, can make the processor of the electronic device execute) the method steps of various exemplary embodiments of the present disclosure.

[0106] From the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to make a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) execute the method according to the embodiments of the present disclosure.

[0107] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the concepts disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

[0108] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A model training method, characterized in that, include: Obtain sample input information for the reasoning task, and concatenate the sample input information with the specified information represented by the no-reasoning prefix to obtain intermediate input information; The specified information is used to reduce redundant reasoning in the reasoning model during the execution of reasoning tasks; The intermediate input information is analyzed through multiple inference operation modes of the inference model to determine the training samples; The training samples are grouped, and if the inference accuracy of each group of training samples meets the accuracy threshold, the inference length penalty of the inference model is triggered to determine the reward function. The model parameters of the inference model are adjusted based on the reward function to obtain a trained inference model; wherein, the trained inference model is used to analyze and process the text information and / or image information of the task to be processed through an inference operation mode that matches the task to be processed, and generate processing results.

2. The model training method according to claim 1, characterized in that, The step of analyzing the intermediate input information through multiple inference operation modes of the inference model to determine training samples includes: Based on the multiple reasoning operation modes of the reasoning model, the intermediate input information is analyzed and processed to generate multiple answer information corresponding to the multiple reasoning operation modes, so as to determine the training samples; The inference operation modes include zero-thinking sampling mode and self-recovering inference sampling mode, and the inference chain lengths of the multiple inference operation modes are different.

3. The model training method according to claim 2, characterized in that, The inference length penalty that triggers the inference model, used to determine the reward function, includes: For each training sample, extract the longest and shortest answer information from the multiple answer information included in each training sample. Determine the length difference between the longest and shortest answer information, and calculate the overlength ratio corresponding to the training sample based on the ratio of the length difference to the pre-configured penalty window length; Determine the penalty coefficient for the training samples, and determine the penalty based on the ultra-long ratio and the penalty coefficient; The reward function is determined based on the correctness indicator function of the answer and the penalty.

4. The model training method according to claim 3, characterized in that, Determining the penalty coefficient for the training samples includes: If the inference accuracy is greater than or equal to the accuracy threshold, the penalty coefficient is determined based on the inference accuracy, the fixed parameter, and the accuracy threshold.

5. The model training method according to claim 2, characterized in that, The step of adjusting the model parameters of the inference model based on the reward function to obtain a trained inference model includes: The reward value is determined using the reward function. The inference model is reinforced based on the reward value to determine the trained inference model.

6. The model training method according to claim 5, characterized in that, The step of performing reinforcement learning on the inference model based on the reward value to determine the trained inference model includes: The advantage is determined based on the reward value, and the strategy gradient is determined based on the advantage. The model parameters of the inference model are adjusted according to the gradient direction of the policy gradient to obtain a trained inference model.

7. A task processing method, characterized in that, include: Obtain text and / or image information of the task to be processed; The input information is input into the trained inference model, and the text information and / or image information are analyzed by the model through an inference operation mode that matches the task to be processed, so as to obtain the processing result corresponding to the text information and / or image information; wherein, the trained inference model is trained by the model training method according to any one of claims 1-6.

8. The task processing method according to claim 7, characterized in that, The step of performing model analysis on the text information and / or image information through a reasoning operation mode matched with the task to be processed, to obtain the processing result corresponding to the text information and / or image information, includes: Determine the task difficulty of the task to be processed; Based on the task difficulty, a reasoning operation mode matching the task to be processed is determined. Based on the reasoning chain corresponding to the reasoning operation mode, the text information and / or image information of the task to be processed are analyzed by the model to determine the processing result corresponding to the input information.

9. A computer program product, characterized in that, When the computer program is executed by the processor, it implements the model training method according to any one of claims 1-6 or the task processing method according to any one of claims 7-8.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the model training method of any one of claims 1-6 or the task processing method of any one of claims 7-8 by executing the executable instructions.

Images