Model processing method and device and electronic equipment

Abstract

The invention provides a model processing method and device and electronic equipment. The model processing method comprises the steps of obtaining sample data; the sample data comprises sample tasks and reasoning steps corresponding to the sample tasks; inputting the sample data into a preset model to obtain a first sample task processing result corresponding to the sample task; determining task reward information based on the difference between the first sample task processing result and the real task processing result; adjusting the reasoning step to obtain an adjusted reasoning step; step reward information is determined according to the influence degree of the adjustment reasoning step on the first sample task processing result; generating target reward information corresponding to the sample task based on the step reward information and the task reward information; and adjusting an inference strategy of the preset model based on the target reward information. According to the embodiment of the invention, the step reward information and the task reward information are combined, so that the reasoning accuracy and reasoning efficiency of the upmodel are improved.

Description

Technical Field

[0001] This application belongs to the field of computer technology, specifically relating to a model processing method, apparatus, and electronic device. Background Technology

[0002] In related technologies, a single traditional reward mechanism is usually relied upon, which calculates the difference between the processed result and the actual result to obtain a reward, and then adjusts the model's inference strategy based on the reward.

[0003] However, models that rely on a single traditional reward mechanism will assume that the reasoning process is reasonable as long as the final result is correct, lacking flexibility and diversity, thereby reducing the model's reasoning accuracy and efficiency. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a model processing method, apparatus, and electronic device.

[0005] According to one aspect of the embodiments of this application, a model processing method is provided, the method comprising: Acquire sample data; the sample data includes sample tasks and the inference steps corresponding to the sample tasks. The sample data is input into a preset model, and the sample task is processed according to the inference steps based on the preset model to obtain the first sample task processing result corresponding to the sample task; the task reward information is determined based on the difference between the first sample task processing result and the actual task processing result corresponding to the sample task. The reasoning steps are adjusted to obtain adjusted reasoning steps; based on the degree of influence of the adjusted reasoning steps on the processing result of the first sample task, the step reward information corresponding to the adjusted reasoning steps is determined. Based on the step reward information and the task reward information, the target reward information corresponding to the sample task is generated; The reasoning strategy of the preset model is adjusted based on the target reward information to obtain the target model.

[0006] In an optional embodiment, the number of inference steps is at least two, and adjusting the inference steps to obtain adjusted inference steps includes: When the reasoning strategy of the preset model is adjusted, that is, the reasoning path of the preset model is adjusted, each reasoning step is modified to obtain the modified reasoning step corresponding to each reasoning step; Based on the preset model, the sample task is processed according to the modified inference steps corresponding to each inference step to obtain the second sample task processing result corresponding to the sample task. If the difference between the processing result of the first sample task and the processing result of the second sample task is greater than or equal to a first threshold, each inference step is adjusted to a corresponding modified inference step to obtain the adjusted inference step corresponding to each inference step.

[0007] In an optional embodiment, determining the step reward information corresponding to the adjusted inference step based on the degree of influence of the adjusted inference step on the processing result of the first sample task includes: Based on the degree of influence of the adjusted inference step corresponding to each inference step on the processing result of the first sample task, the reverse inference confidence of the adjusted inference step corresponding to each inference step is determined. The confidence scores of the reverse reasoning are weighted to obtain the reward information for the step.

[0008] In an optional embodiment, determining the reverse inference confidence of the adjusted inference step corresponding to each inference step based on the degree of influence of the adjusted inference step on the processing result of the first sample task includes: Determine the first partial derivative of the adjusted inference step corresponding to each inference step, and the second partial derivative corresponding to the processing result of the first sample task; The ratio of the second partial derivative to the first partial derivative is determined to obtain the reverse reasoning confidence of the adjusted reasoning step corresponding to each reasoning step.

[0009] In an optional embodiment, generating the target reward information corresponding to the sample task based on the step reward information and the task reward information includes: Determine a first reward weight; the first reward weight is used to balance the step reward information and the task reward information. Based on the first reward weight, the step reward information, and the task reward information, the first target reward information corresponding to the sample task is generated.

[0010] In an optional embodiment, the sample task further includes inference depth and background knowledge, wherein the inference depth characterizes the inference complexity of the inference step, and the background knowledge characterizes the background knowledge and / or inference assumptions required to perform the sample task; The adjustment of the reasoning steps to obtain the adjusted reasoning steps includes: When the inference strategy of the preset model is adjusted to adjust the inference depth of the preset model, the complexity of the sample task is evaluated based on the number of inference steps, the inference depth, and the background knowledge to obtain the task complexity corresponding to the sample task. If the task complexity is greater than or equal to the second threshold, a new reasoning step is added to at least two of the reasoning steps to obtain the adjusted reasoning step; If the task complexity is less than the second threshold, the inference steps are reduced from at least two inference steps to obtain the adjusted inference steps.

[0011] In an optional embodiment, the number of adjustment inference steps is at least two, and determining the step reward information corresponding to the adjustment inference step based on the degree of influence of the adjustment inference step on the processing result of the first sample task includes: Determine the reasoning complexity of each of the adjusted reasoning steps, and the degree of impact of each of the adjusted reasoning steps on the processing result of the first sample task; Based on the reasoning complexity of each of the adjusted reasoning steps and the degree of influence of each of the adjusted reasoning steps on the processing result of the first sample task, the reward information corresponding to each of the adjusted reasoning steps is determined. The sum of the reward information corresponding to each of the aforementioned adjustment reasoning steps is determined to obtain the step reward information.

[0012] In an optional embodiment, generating the target reward information corresponding to the sample task based on the step reward information and the task reward information includes: Determine a second reward weight; the second reward weight is used to balance the step reward information and the task reward information. Based on the second reward weight, the step reward information, and the task reward information, the second target reward information corresponding to the sample task is generated.

[0013] In an optional embodiment, the method further includes: First comprehensive reward information is generated based on the first target reward information and the second target reward information; the first target reward information is determined based on the task reward information and the step reward information determined by the back reasoning confidence, wherein the back reasoning confidence is the back reasoning confidence of the adjustment reasoning step corresponding to each reasoning step; the adjustment reasoning step corresponding to each reasoning step is to adjust each reasoning step to the corresponding modified reasoning step, etc., and the modified reasoning step corresponding to each reasoning step is obtained by modifying each reasoning step. The step of adjusting the inference strategy of the preset model based on the target reward information to obtain the target model includes: The reasoning strategy of the preset model is adjusted based on the first comprehensive reward information to obtain the target model.

[0014] In an optional embodiment, the method further includes: In cases where the sample task includes at least two modalities, determine the modal correlation between the at least two modalities; Cross-modal reward information is generated based on the modal correlation; Based on the cross-modal reward information and the task reward information, a third target reward information is generated; the third target reward information is used to adjust the inference strategy of the preset model.

[0015] In an optional embodiment, generating cross-modal reward information based on the modal correlation includes: If the modal correlation is greater than or equal to a third threshold, determine the degree of fusion of the preset model with the at least two modal information; A third reward weight is generated based on the degree of fusion. The cross-modal reward information is generated based on the third reward weight and the modal correlation.

[0016] In an optional embodiment, generating the target reward information corresponding to the sample task based on the step reward information and the task reward information includes: Based on the step reward information, the task reward information, and the third target reward information, generate the second comprehensive reward information corresponding to the sample task; The step of adjusting the inference strategy of the preset model based on the target reward information to obtain the target model includes: The reasoning strategy of the preset model is adjusted based on the second comprehensive reward information to obtain the target model.

[0017] According to one aspect of the embodiments of this application, a task processing method is provided, the method comprising: Obtain the target task; The target task is input into the target model for processing to obtain the task processing result corresponding to the target task. The target model is trained based on the model processing method described in any of the above embodiments.

[0018] According to one aspect of the embodiments of this application, a model processing apparatus is provided, the apparatus comprising: A sample data acquisition module is used to acquire sample data; the sample data includes sample tasks and the inference steps corresponding to the sample tasks. The model prediction module is used to input the sample data into a preset model, process the sample task according to the inference steps based on the preset model, and obtain a first sample task processing result corresponding to the sample task; and determine task reward information based on the difference between the first sample task processing result and the actual task processing result corresponding to the sample task. An adjustment module is used to adjust the inference steps to obtain adjusted inference steps; and to determine the step reward information corresponding to the adjusted inference steps based on the degree of influence of the adjusted inference steps on the processing result of the first sample task. The target reward information generation module is used to generate target reward information corresponding to the sample task based on the step reward information and the task reward information; The strategy adjustment module is used to adjust the inference strategy of the preset model based on the target reward information to obtain the target model.

[0019] In an optional embodiment, the number of inference steps is at least two, and the adjustment module is further configured to: When the reasoning strategy of the preset model is adjusted, that is, the reasoning path of the preset model is adjusted, each reasoning step is modified to obtain the modified reasoning step corresponding to each reasoning step; Based on the preset model, the sample task is processed according to the modified inference steps corresponding to each inference step to obtain the second sample task processing result corresponding to the sample task. If the difference between the processing result of the first sample task and the processing result of the second sample task is greater than or equal to a first threshold, each inference step is adjusted to a corresponding modified inference step to obtain the adjusted inference step corresponding to each inference step.

[0020] In an optional embodiment, the adjustment module is further configured to: Based on the degree of influence of the adjusted inference step corresponding to each inference step on the processing result of the first sample task, the reverse inference confidence of the adjusted inference step corresponding to each inference step is determined. The confidence scores of the reverse reasoning are weighted to obtain the reward information for the step.

[0021] In an optional embodiment, the adjustment module is further configured to: Determine the first partial derivative of the adjusted inference step corresponding to each inference step, and the second partial derivative corresponding to the processing result of the first sample task; The ratio of the second partial derivative to the first partial derivative is determined to obtain the reverse reasoning confidence of the adjusted reasoning step corresponding to each reasoning step.

[0022] In an optional embodiment, the target reward information generation module is further configured to: Determine a first reward weight; the first reward weight is used to balance the step reward information and the task reward information. Based on the first reward weight, the step reward information, and the task reward information, the first target reward information corresponding to the sample task is generated.

[0023] In an optional embodiment, the sample task further includes inference depth and background knowledge, wherein the inference depth characterizes the inference complexity of the inference step, and the background knowledge characterizes the background knowledge and / or inference assumptions required to perform the sample task; The adjustment module is also used for: When the inference strategy of the preset model is adjusted to adjust the inference depth of the preset model, the complexity of the sample task is evaluated based on the number of inference steps, the inference depth, and the background knowledge to obtain the task complexity corresponding to the sample task. If the task complexity is greater than or equal to the second threshold, a new reasoning step is added to at least two of the reasoning steps to obtain the adjusted reasoning step; If the task complexity is less than the second threshold, the inference steps are reduced from at least two inference steps to obtain the adjusted inference steps.

[0024] In an optional embodiment, the number of adjustment inference steps is at least two, and the adjustment module is further configured to: Determine the reasoning complexity of each of the adjusted reasoning steps, and the degree of impact of each of the adjusted reasoning steps on the processing result of the first sample task; Based on the reasoning complexity of each of the adjusted reasoning steps and the degree of influence of each of the adjusted reasoning steps on the processing result of the first sample task, the reward information corresponding to each of the adjusted reasoning steps is determined. The sum of the reward information corresponding to each of the aforementioned adjustment reasoning steps is determined to obtain the step reward information.

[0025] In an optional embodiment, the target reward information generation module is further configured to: Determine a second reward weight; the second reward weight is used to balance the step reward information and the task reward information. Based on the second reward weight, the step reward information, and the task reward information, the second target reward information corresponding to the sample task is generated.

[0026] In an optional embodiment, the apparatus further includes: A first comprehensive reward information generation device is used to generate first comprehensive reward information based on first target reward information and second target reward information; the first target reward information is determined based on the task reward information and step reward information determined by back-inference confidence, wherein the back-inference confidence is the back-inference confidence of the adjustment inference step corresponding to each inference step; the adjustment inference step corresponding to each inference step is to adjust each inference step to the corresponding modified inference step, etc., and the modified inference step corresponding to each inference step is obtained by modifying each inference step. Accordingly, the strategy adjustment module is also used to: adjust the reasoning strategy of the preset model based on the first comprehensive reward information to obtain the target model.

[0027] In an optional embodiment, the above-described apparatus further includes: A modal correlation determination module is used to determine the modal correlation between at least two modalities when the sample task includes at least two modal information. A cross-modal reward information generation module is used to generate cross-modal reward information based on the modal correlation; The third objective reward information generation module is used to generate third objective reward information based on the cross-modal reward information and the task reward information; the third objective reward information is used to adjust the inference strategy of the preset model.

[0028] In an optional embodiment, the cross-modal reward information generation module is further configured to: If the modal correlation is greater than or equal to a third threshold, determine the degree of fusion of the preset model with the at least two modal information; A third reward weight is generated based on the degree of fusion. The cross-modal reward information is generated based on the third reward weight and the modal correlation.

[0029] In an optional embodiment, the target reward information generation module is further configured to: Based on the step reward information, the task reward information, and the third target reward information, generate the second comprehensive reward information corresponding to the sample task; Accordingly, the aforementioned strategy adjustment module is also used for: The reasoning strategy of the preset model is adjusted based on the second comprehensive reward information to obtain the target model.

[0030] According to one aspect of the embodiments of this application, a task processing apparatus is provided, the apparatus comprising: The target task acquisition module is used to acquire target tasks; The task processing module is used to input the target task into the target model for processing and obtain the task processing result corresponding to the target task. The target model is trained based on the model processing method described in any of the above embodiments.

[0031] According to one aspect of the embodiments of this application, an electronic device is provided, the electronic device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the model processing method or task processing method as described above.

[0032] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the model processing method or task processing method as described above.

[0033] According to one aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the model processing method or task processing method as described above.

[0034] The beneficial effects of the technical solutions provided in this application include at least the following: In this embodiment, sample data is input into a preset task processing model. Based on the preset model, the sample tasks in the sample data are processed according to the inference steps in the sample data to obtain the first sample task processing result. Then, the difference between the first sample task processing result and the actual task processing result corresponding to the sample task is calculated to determine the task reward information. To avoid relying on a single traditional reward mechanism and a preset inference path, the server does not directly use the task reward information to adjust the inference strategy of the preset model. Instead, it further adjusts the inference steps to obtain adjusted inference steps. Based on the degree of influence of the adjusted inference steps on the first sample task processing result, the step reward information corresponding to the adjusted inference steps is determined. Then, based on the step reward information and the task reward information, the target reward information corresponding to the sample task is generated. Finally, based on the target reward information, the inference strategy of the preset model is adjusted to obtain the target model.

[0035] Therefore, by combining step reward information and task reward information, the model not only looks at the final result during training, but also considers whether the final answer will change after certain steps are adjusted. That is, it provides detailed rewards for the intermediate links in the reasoning process, avoiding the model from relying on incorrect reasoning steps. This effectively solves the problems that the model is prone to getting stuck in fixed reasoning paths during training, missing the training and optimization of the reasoning process, and lacking flexibility and diversity, thereby improving the model's reasoning accuracy and efficiency. Attached Figure Description

[0036] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the implementation environment of the model processing method or task processing method provided in an exemplary embodiment of this application.

[0038] Figure 2 This is a flowchart illustrating a model processing method provided in an exemplary embodiment of this application. Figure 1 .

[0039] Figure 3 This is a flowchart illustrating a model processing method provided in an exemplary embodiment of this application. Figure 2 .

[0040] Figure 4 This is a flowchart illustrating the model processing method provided in an exemplary embodiment of the application. Figure 3 .

[0041] Figure 5 This is a flowchart illustrating the model processing method provided in an exemplary embodiment of the application. Figure 4 .

[0042] Figure 6 This is a flowchart illustrating the model processing method provided in an exemplary embodiment of the application. Figure 5 .

[0043] Figure 7 This is a flowchart illustrating the model processing method provided in an exemplary embodiment of the application. Figure 6 .

[0044] Figure 8 This is a flowchart illustrating the model processing method provided in an exemplary embodiment of the application. Figure 7 .

[0045] Figure 9 This is a block diagram of a model processing apparatus provided in an exemplary embodiment of this application.

[0046] Figure 10 An exemplary embodiment of the application provides a block diagram of a task processing apparatus.

[0047] Figure 11 This is a hardware structure block diagram of a server provided according to an exemplary embodiment. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0049] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the present application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0050] It should be noted that, in the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0051] Figure 1 This is a schematic diagram illustrating the implementation environment of the model processing method or task processing method provided in an exemplary embodiment of this application. For example... Figure 1As shown, the implementation environment may include at least a computer device 110 and a server 120. The computer device 110 and the server 120 may be directly or indirectly connected via wired or wireless communication, and this embodiment of the application does not impose any limitations on this.

[0052] The model processing method or task processing method provided in this application embodiment can be executed by computer device 110 alone, by server 120 alone, or by computer device 110 and server 120 through data interaction and collaboration.

[0053] In one example, the model processing method or task processing method can be performed by server 120. For this model processing method, server 120 acquires sample data; the sample data includes sample tasks and the corresponding inference steps; the sample tasks are input into a preset model, and the sample tasks are processed according to the inference steps based on the preset model to obtain a first sample task processing result; task reward information is determined based on the difference between the first sample task processing result and the actual task processing result corresponding to the sample task; the inference steps are adjusted to obtain adjusted inference steps; step reward information corresponding to the adjusted inference steps is determined based on the degree of influence of the adjusted inference steps on the first sample task processing result; target reward information corresponding to the sample task is generated based on the step reward information and the task reward information; the inference strategy of the preset model is adjusted based on the target reward information to obtain the target model.

[0054] For this task processing method, server 120 obtains the target task; inputs the target task into the target model for processing, and obtains the task processing result corresponding to the target task.

[0055] In the above example, the server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0056] The computer device 110 may include, but is not limited to, personal computers (PCs), mobile phones, tablets, wearable devices, smart voice interaction devices, smart home appliances, vehicle terminals, aircraft and other electronic devices, virtual reality devices, etc. This application does not limit the types of computer device 110 and server 120.

[0057] While existing large language models perform well on certain tasks, they typically have limitations. A significant issue is their lack of flexibility in reasoning, often relying on pre-defined inference paths and failing to adapt to changes in task complexity. For example, a model might easily arrive at the answer to a simple math problem, but it may err or fail to delve deeply enough into a problem requiring multiple steps and complex background information. Furthermore, models are prone to taking detours in complex tasks and lack effective self-correction capabilities.

[0058] Furthermore, while existing multimodal reasoning models can perform well on some simple tasks, they typically process information from each modality (such as images and text) separately, lacking deep fusion. Often, the models fail to fully utilize the synergistic effects between modalities, leading to imprecise inference results. For instance, given an image and a piece of text, the model needs to understand the information in the image and text and combine them to deduce the answer. However, existing models do not effectively combine these two types of information; they may simply process the image and text separately and then piece them together to reach a conclusion, ignoring the deeper relationships between the image and text.

[0059] Furthermore, most existing reasoning models rely on traditional outcome rewards, meaning the model only focuses on whether the final answer is correct, neglecting the quality of the reasoning process. For a math problem, as long as the result is correct, the reasoning process is irrelevant. The result is that the model may learn incorrect reasoning methods, believing its reasoning is reasonable as long as it can arrive at a correct answer in certain situations. For example, if a crucial step in the reasoning process goes wrong, the model may not recognize this error, but as long as it arrives at the correct result, it will reward itself. The disadvantages of this approach are: it cannot effectively guide the model to improve each step of the reasoning process, and may instead lead the model to develop incorrect reasoning habits.

[0060] To address the problem of relying solely on traditional result rewards, which lacks flexibility and diversity and thus reduces model inference accuracy and efficiency, this application proposes a model processing method. This method inputs sample data, including sample tasks and their corresponding inference steps, into a preset model. Based on the preset model, the sample tasks are processed according to the inference steps to obtain a first sample task processing result. Then, the difference between the first sample task processing result and the actual task processing result corresponding to the sample task is calculated to determine the task reward information. To avoid relying on a single traditional reward mechanism and preset inference path, the server does not directly use the task reward information to adjust the inference strategy of the preset model. Instead, it adjusts the inference steps to obtain adjusted inference steps. Based on the degree of influence of the adjusted inference steps on the first sample task processing result, the step reward information corresponding to the adjusted inference steps is determined. Then, based on the step reward information and the task reward information, target reward information corresponding to the sample task is generated. Finally, based on the target reward information, the inference strategy of the preset model is adjusted to obtain the target model. Therefore, by combining step reward information and task reward information, the model not only looks at the final result during training, but also considers whether the final answer will change after certain steps are adjusted. That is, it provides detailed rewards for the intermediate links in the reasoning process, avoiding the model from relying on incorrect reasoning steps. This effectively solves the problems that the model is prone to getting stuck in fixed reasoning paths during training, missing the training and optimization of the reasoning process, and lacking flexibility and diversity, thereby improving the model's reasoning accuracy and efficiency.

[0061] Furthermore, this application introduces a reverse reasoning reward mechanism. This means that during the reasoning process, the model not only looks at the final result but also considers whether the final answer would change if certain steps were altered. This is similar to solving a math problem, where you not only focus on the final answer but also consider what would happen if a certain step of the reasoning were wrong. The advantage of this approach is that the model can pay more attention to the rationality of the reasoning process, rather than solely relying on the final answer.

[0062] Furthermore, this application also introduces a hierarchical thinking reward mechanism, which allows the model to flexibly adjust its reasoning strategy when facing tasks of varying complexity. For example, if the task is relatively simple, the model can focus on basic logical reasoning; while for complex tasks, the model can focus on higher-level abstraction and reasoning. Hierarchical rewards allow the model to exhibit more appropriate thought processes in different tasks, thereby improving task-solving performance.

[0063] Furthermore, this application also introduces a cross-modal collaborative thinking reward mechanism, which is particularly suitable for tasks that process multimodal information. For example, in image question answering tasks, the model not only needs to understand the image content, but also the text information related to the image, and effectively combine the two. Through this cross-modal collaborative reward mechanism, the model can better understand the relationship between images and text, no longer simply processing them separately, but giving full play to their synergistic effect.

[0064] In summary, the embodiments of this application, by combining at least one of reverse reasoning rewards, hierarchical rewards, and cross-modal collaboration, can effectively solve the problems of inflexibility, lack of rigor, and insufficient multimodal information collaboration in the reasoning process of existing technologies. This allows the model to perform more intelligently, flexibly, and efficiently in various tasks, thereby effectively solving the problems that the model is prone to getting stuck in fixed reasoning paths and lacks flexibility and diversity during training, and improving the model's reasoning accuracy and efficiency.

[0065] First, the technical terms used in the embodiments of this application will be explained: Large Language Models (LLM): These are deep learning models trained on large amounts of text data, capable of performing tasks such as natural language understanding, generation, and reasoning, such as GPT and BERT.

[0066] Counterfactual Thinking Rewards: A mechanism that adjusts model rewards by backtracking the reasoning process and evaluating the impact of reasoning steps on the final result, aiming to optimize the rationality and stability of the reasoning path.

[0067] Hierarchical Thinking Rewards: A reward mechanism that adaptively adjusts the depth and level of reasoning based on task complexity, helping the model to quickly arrive at answers in simple tasks and perform multi-level reasoning in complex tasks.

[0068] Cross-modal Collaborative Thinking Rewards: A mechanism designed to enhance reasoning ability in multimodal tasks by evaluating and rewarding collaboration between different modalities (such as images and text).

[0069] Reasoning Step: refers to each reasoning operation or logical judgment performed sequentially by the model when executing a reasoning task.

[0070] Modality Correlation: A metric that measures the consistency and correlation of information between different modalities (such as images and text), and is used to judge the effectiveness of information fusion between modalities.

[0071] Reward Weight: Used to quantify the weight of a reasoning step in the entire reasoning process. Reasoning steps of different complexity tasks will have different reward weights.

[0072] Reasoning Depth: refers to the level or complexity of reasoning steps involved in a model when processing reasoning tasks. Tasks with higher depth usually require more reasoning steps and abstraction capabilities.

[0073] Counterfactual Confidence: Measures the sensitivity of a reasoning step to the final result. The higher the counterfactual confidence, the greater the impact of that step on the final result.

[0074] Total Reward: A comprehensive reward calculated by combining the final result reward with other reward mechanisms (such as back reasoning rewards, hierarchical rewards, and cross-modal collaboration rewards), used to guide the model to optimize the reasoning process.

[0075] Figure 2 This is a flowchart illustrating a model processing method provided in an exemplary embodiment of this application. Figure 1 For example, the entity executing this method can be... Figure 1 The server in the example. This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many steps and does not represent the only execution order. In actual system or server products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown in the examples... Figure 2 As shown, the method may include the following steps.

[0076] S210. Obtain sample data; the sample data includes sample tasks and the inference steps corresponding to the sample tasks.

[0077] In some embodiments, the sample task refers to a specific and explicit "task" or "goal" that a pre-defined model needs to complete, describing the capabilities that the pre-defined model should possess. Optionally, the sample task can be a task in various scenarios, without specific limitations. For example, question-answering tasks in multimodal question-answering systems, content generation tasks in automated content generation scenarios, etc. The multimodal question-answering system scenario refers to enhancing the model's reasoning ability in complex situations by combining image and text information, thereby improving the accuracy and relevance of question answers. The automated content generation scenario refers to improving the accuracy and richness of generated content in multimodal content generation tasks such as image generation and video content description.

[0078] Optionally, the reasoning step refers to each reasoning operation or logical judgment performed sequentially by the model when executing the reasoning task. This reasoning step can form a reasoning path.

[0079] For example, the sample task processing went through n steps, and the reasoning steps included: T1, T2, T3...Tn.

[0080] S220. Input the sample data into the preset model, process the sample task according to the inference steps based on the preset model, and obtain the first sample task processing result corresponding to the sample task; determine the task reward information based on the difference between the first sample task processing result and the real task processing result corresponding to the sample task.

[0081] In some embodiments, the server can input sample task processing into a preset model, which processes the sample task according to inference steps to obtain a first sample task processing result. Then, the server calculates the difference between the first sample task processing result and the corresponding real task processing result, calculates loss data based on this difference, and obtains the task reward information. For example, the smaller the loss data, the larger the task reward information; the larger the loss data, the smaller the task reward information. This approach guides the model to optimize its inference process, moving it closer to the true correct answer.

[0082] Optionally, the preset model can be any type of large language model, without specific limitations. For example, in a multimodal question-answering system scenario, it can be an image question-answering large language model, and in an automated content generation scenario, it can be a visual reasoning large language model.

[0083] Optionally, the result of the first sample task can be considered a traditional reward result. The result of the real task can be a real label that has been pre-labeled for the sample task.

[0084] S230. Adjust the reasoning steps to obtain the adjusted reasoning steps; determine the step reward information corresponding to the adjusted reasoning steps based on the degree of influence of the adjusted reasoning steps on the processing results of the first sample task.

[0085] In some embodiments, to avoid relying on a single traditional reward mechanism and a preset inference path, after determining that the first sample task processing result has been obtained, the server does not directly use the task reward information to adjust the inference strategy of the preset model, but further adjusts the inference steps to obtain adjusted inference steps, and determines the step reward information corresponding to the adjusted inference steps according to the degree of influence of the adjusted inference steps on the first sample task processing result.

[0086] Optionally, the "degree of influence of adjusting the inference steps on the processing result of the first sample task" can refer to: the changes that the change in the inference steps will have on the final processing result and the degree of such changes.

[0087] S240. Based on the step reward information and task reward information, generate the target reward information corresponding to the sample task.

[0088] S250. Adjust the inference strategy of the preset model based on the target reward information to obtain the target model.

[0089] In some embodiments, after the server determines that it has obtained the step reward information and the task reward information, it can combine the step reward information and the task reward information to generate the target reward information corresponding to the sample task. Finally, the inference strategy of the preset model is adjusted based on the target reward information to obtain the target model.

[0090] It should be noted that steps S210-S250 above constitute the training process of the target model. By combining step reward information and task reward information, the model not only considers the final result during training, but also whether the final answer will change after certain steps are adjusted. That is, it provides detailed rewards for the intermediate links in the reasoning process, avoiding the model from relying on incorrect reasoning steps. This effectively solves the problems that the model is prone to getting stuck in a fixed reasoning path during training, missing the training and optimization of the reasoning process, and lacking flexibility and diversity, thereby improving the model's reasoning accuracy and efficiency.

[0091] The following is an introduction to the reverse reasoning reward mechanism: The back-reasoning reward mechanism aims to optimize the reasoning process of a pre-defined model by evaluating the robustness and rationality of each step in the reasoning process. In traditional reinforcement learning and reward mechanisms, models typically focus only on whether the final answer is correct, ignoring potential flaws or suboptimal paths in the reasoning process. The innovation of the back-reasoning reward mechanism lies in its focus not only on the accuracy of the final answer but also in encouraging the model to consider whether the result would change if certain assumptions or steps in the reasoning process were altered. In this way, the model can improve its choice of reasoning paths, avoiding deviations in the final answer caused by a single erroneous step.

[0092] Principle Overview: The basic idea of ​​the reverse reasoning reward mechanism is that at each step of the reasoning process, the model should reflect on the rationality of that step and assess whether a change in the conditions of that reasoning step would lead to a different final answer. In other words, the model needs to speculate whether the result of the reasoning would be different if a certain reasoning path or assumption were not true. Through this method, the model in this application's embodiments can identify and avoid reasoning steps that would lead to erroneous results.

[0093] Optionally, a back-reasoning confidence score can be calculated to measure the robustness of each reasoning step. Specifically, the back-reasoning confidence score represents "the magnitude of change in the final result if the assumptions at a certain step change." If a reasoning step is robust, the change in the final result will be small even if the assumptions change; however, if a step is fragile, the final result may change significantly if the assumptions change.

[0094] Figure 3 This is a flowchart illustrating a model processing method provided in an exemplary embodiment of this application. Figure 2 ,like Figure 3 As shown, in step S230 above, the number of reasoning steps is at least two. Adjusting the reasoning steps to obtain adjusted reasoning steps may include: S230-1-1. When adjusting the reasoning strategy of the preset model, and in the case of adjusting the reasoning path of the preset model, each reasoning step is modified to obtain the modified reasoning step corresponding to each reasoning step.

[0095] S230-1-2. Based on the preset model, the sample task is processed according to the modified inference steps corresponding to each inference step to obtain the second sample task processing result corresponding to the sample task.

[0096] S230-1-3. If the difference between the processing result of the first sample task and the processing result of the second sample task is greater than or equal to the first threshold, each inference step is adjusted to the corresponding modified inference step, and the adjusted inference step corresponding to each inference step is obtained.

[0097] In some embodiments, when the reasoning strategy of the preset model is adjusted, that is, the reasoning path of the preset model is adjusted, it indicates that a reverse reasoning reward mechanism can be executed at this time. Then the server can modify each reasoning step in turn to obtain the modified reasoning step corresponding to each reasoning step.

[0098] Next, the preset model processes the sample task according to the modified inference steps corresponding to each inference step, obtaining the second sample task processing result. This second sample task processing result is the result obtained by the preset model processing the sample task processing result according to the modified inference steps after the step changes.

[0099] Finally, the server determines the difference between the processing results of the first sample task and the processing results of the second sample task. If the difference is greater than or equal to the first threshold, it indicates that the modification of the inference step has a significant impact on the final task processing result. In this case, each inference step is adjusted to the corresponding modified inference step, thus obtaining the adjusted inference step corresponding to each inference step.

[0100] It's important to note that only one inference step is modified at a time; the evaluation process of the back-reasoning reward mechanism is step-by-step. Only after one inference step has been evaluated is the next inference step evaluated. The impact of each inference step needs to be evaluated individually, especially the impact of erroneous steps on the result. If multiple inference steps are evaluated simultaneously, the model may not be able to accurately determine the specific impact of each step on the final result. Step-by-step evaluation ensures the rationality and stability of each inference step, helping the model better identify and adjust for potential errors. Therefore, only one inference step is evaluated at a time. This step-by-step analysis ensures that the model optimizes each inference stage, avoiding incorrect inference paths that lead to erroneous final results.

[0101] In some embodiments, the above method may further include: if the difference between the second sample task processing result and the second sample task processing result is less than a first threshold, not adjusting each inference step, that is, keeping the inference step unchanged.

[0102] It should be noted that the embodiments of this application do not limit the first threshold, which can be set according to actual business needs.

[0103] Therefore, in the reverse reasoning reward mechanism, each reasoning step will only be adjusted to the corresponding modified reasoning step when the difference between the processing result of the first sample task and the processing result of the second sample task is greater than or equal to the first threshold, that is, when it is determined that the modification of the reasoning step has a significant impact on the final task processing result. This avoids the defect of high model training cost caused by modifying every step, improves model training efficiency, and reduces model training cost.

[0104] In some embodiments, continue as follows Figure 3 As shown, in step S230 above, determining the step reward information corresponding to the adjusted inference step based on the degree of influence of the adjusted inference step on the processing result of the first sample task may include: S230-1-4. Based on the degree of influence of the adjusted inference step corresponding to each inference step on the processing result of the first sample task, determine the reverse inference confidence of the adjusted inference step corresponding to each inference step.

[0105] In some embodiments, in step S2304 above, the server can determine the back-inference confidence of the adjusted inference step corresponding to each inference step based on the degree of influence of the adjusted inference step on the processing result of the first sample task. That is, the back-inference confidence measures the sensitivity of the inference step to the final result; the higher the back-inference confidence, the greater the impact of the step on the final result.

[0106] Optionally, determining the back-inference confidence of the adjusted inference step corresponding to each inference step based on the degree of influence of the adjusted inference step on the processing result of the first sample task can include: Determine the first partial derivative of the adjusted inference step corresponding to each inference step, and the second partial derivative corresponding to the processing result of the first sample task. Determine the ratio of the second partial derivative to the first partial derivative to obtain the back-inference confidence of the adjusted inference step corresponding to each inference step.

[0107] In this embodiment, let T represent the inference step and R represent the task processing result. The server can determine the first partial derivative of the inference step corresponding to each inference step. Ti, and the second partial derivative corresponding to the processing result of the first sample task R is calculated, and the ratio of the second partial derivative to the first partial derivative is used to obtain the back-inference confidence of the adjusted inference step corresponding to each inference step. The calculation formula can be as follows: Ci=| R / Ti|; Where Ci represents the back-reasoning confidence of the adjustment reasoning step corresponding to the i-th reasoning step, that is, the degree to which a small change in step Ti affects the final result R. If Ti has a large impact on the result, then Ci will be large; conversely, if Ci has a small impact on the result, then this value will be small.

[0108] Therefore, through R / Ti can accurately represent "the impact of changes in reasoning step Ti on the final result R", that is, accurately represent how changes in the i-th reasoning step (e.g., incorrect assumptions or different reasoning paths) will affect the final reasoning result R, improve the determination accuracy of the confidence of backward reasoning, and thus improve the determination accuracy of step reward information.

[0109] S230-1-5. Weight the confidence of the reverse reasoning to obtain the step reward information.

[0110] In some embodiments, after the server determines the back-inference confidence of the adjusted inference step corresponding to each inference step, the back-inference confidence of the adjusted inference steps corresponding to all inference steps can be weighted to obtain the step reward information corresponding to the adjusted inference step.

[0111] Optionally, during the weighting process, the server can first determine the reward weight of the adjusted inference step corresponding to each inference step, and then multiply the reward weight of the adjusted inference step corresponding to each inference step with the product of the adjusted inference step itself, obtaining the product for each inference step. Finally, the server determines the sum of the products for each inference step to obtain the step reward information. It should be noted that the reward weight of the adjusted inference step can be adaptively adjusted according to the task complexity or the model's objective. For example, in simple inference tasks, the model may assign a lower reward weight, relying more on the final result; while in complex tasks, the reward weight of back-reasoning is more important.

[0112] It should be noted that the reverse reasoning reward mechanism is used here, so the step reward information here can be considered as a reverse reasoning reward.

[0113] For example, the formula for calculating the reward information in this step can be as follows: ; in, This refers to step-by-step reward information, i.e., the reward for backward reasoning. This refers to the reward weight for adjusting the reasoning step corresponding to the i-th reasoning step. This refers to the reverse reasoning confidence of the adjusted reasoning step corresponding to the i-th reasoning step.

[0114] Back-reasoning confidence measures the robustness of each reasoning step. Specifically, it represents "the magnitude of change in the final result if the assumption at a certain reasoning step changes." If a reasoning step is robust, the change in the final result will be small even if the assumption changes; conversely, if a step is fragile, the final result may change significantly if the assumption changes. Therefore, by weighting the back-reasoning confidence of each reasoning step, the model can reflect on the rationality of each step and assess whether a change in the conditions of that step would lead to a change in the final answer. This improves the accuracy of determining step reward information and, consequently, enhances the model's training accuracy.

[0115] In some embodiments, continue as follows Figure 3 As shown, in step S240 above, generating the target reward information corresponding to the sample task based on the step reward information and task reward information may include: S240-1-1. Determine the first reward weight; the first reward weight is used to balance the step reward information and the task reward information.

[0116] S240-1-2. Based on the first reward weight, step reward information and task reward information, generate the first target reward information corresponding to the sample task.

[0117] In some embodiments, the final first target reward information should include not only the reverse reasoning reward but also the task reward information. In this case, the server can predetermine a first reward weight λ. The first reward weight λ is used to balance the step reward information and the task reward information. λ can be dynamically adjusted to ensure that the model can reasonably balance the two reward signals during the reasoning process.

[0118] Next, based on the first reward weight, step reward information, and task reward information, the server generates the first target reward information corresponding to the sample task. The specific calculation formula can be as follows: Rtotal=λ Routcome+(1 λ) Rcounterfactual; Where Rtotal refers to the first objective reward information, λ refers to the first reward weight, Routcome refers to the task reward information, and Rcounterfactual refers to the step reward information.

[0119] Therefore, by combining step reward information and task reward information, the model not only looks at the final result during training, but also considers whether the final answer will change after certain steps are adjusted. This effectively solves the problems that the model is prone to getting stuck in a fixed reasoning path and lacks flexibility and diversity during training, thereby improving the model's reasoning accuracy and efficiency. In addition, the first reward weight can balance the step reward information and task reward information, further ensuring the accuracy and stability of the generated first target reward information.

[0120] The following is a general explanation of the above-mentioned reverse reasoning reward process: Figure 4 This is a flowchart illustrating the model processing method provided in an exemplary embodiment of the application. Figure 3 ,like Figure 4 As shown, the model processing method includes: 1) Define the reverse reasoning process: For each sample task, the model generates a reasoning path, which consists of at least two reasoning steps. Assume that the model goes through n steps in the reasoning process, with each step having reasoning steps T1, T2, ..., Tn, and the final result of the first sample task is R.

[0121] 2) Modify each inference step TI to obtain the modified inference step corresponding to each inference step. Based on the preset model, process the sample task according to the modified inference step corresponding to each inference step to obtain the second sample task processing result R' corresponding to the sample task.

[0122] 3) Calculate the difference between the processing results of the first sample task and the processing results of the second sample task = R - R'.

[0123] 4) If the difference is greater than or equal to the first threshold, adjust each inference step to the corresponding modified inference step, thus obtaining the adjusted inference step for each inference step. For example, in the reverse inference process, if step T2 is modified to T2', and the difference is greater than or equal to the threshold, the inference step T2 needs to be adjusted, which means adjusting T2 in the initial inference step to T2', i.e., the inference path becomes T1, T2', ..., Tn.

[0124] If the difference is less than the first threshold, no adjustment is made to each reasoning step; that is, the reasoning step is kept unchanged.

[0125] 5) Calculate the confidence score for backward reasoning. The formula is as follows: Ci=| R / Ti|.

[0126] 6) Step reward information calculation (i.e., reverse reasoning reward), the calculation formula can be as follows: .

[0127] 7) The calculation formula for combining rewards and results can be as follows: Rtotal=λ Routcome+(1 λ) Rcounterfactual.

[0128] The following example illustrates the process of rewarding reverse reasoning: Suppose we have a simple mathematical reasoning task: given a rectangle with length L=5 and width W=3, find the area of ​​the rectangle.

[0129] Reasoning steps: The first step is to calculate the area using the formula A=L×W.

[0130] The second step involves model calculation to obtain an area result of A = 5 × 3 = 15.

[0131] Reverse reasoning: Suppose that during the reasoning process, the model initially used an incorrect formula, mistakenly applying A=L+W. In the second step, the model would calculate A=5+3=8, which is incorrect.

[0132] Through backward reasoning, the model traces its own reasoning steps back, analyzing that if the formula A=L+W is incorrect, then the final result R=8 will also be incorrect. The model calculates the confidence score of backward reasoning: ; In this case, the high confidence level C1 of the backward reasoning indicates that the impact of the reasoning error is very large, and the model should give a low reward (or penalty) to this step.

[0133] It's important to note that back reasoning confidence measures the degree to which a change in a reasoning step affects the final result. If an error in a step (e.g., using an incorrect formula) significantly alters the final result, the back reasoning confidence for that step will be very high, close to 1. Conversely, if an error in a step has a minor impact (e.g., small numerical errors don't affect the final answer), the back reasoning confidence will be low. Therefore, a back reasoning confidence of 1 means that an error in this step has a significant impact on the result; any change will lead to a large deviation in the reasoning outcome.

[0134] Reward Calculation: In this example, if the model uses an incorrect formula, the confidence level of the back-reasoning will be relatively high, and the final back-reasoning reward will be correspondingly lower. The model will recognize that this incorrect reasoning method needs improvement. The final reward will be adjusted based on the difference between the incorrect reasoning process and the correct result.

[0135] Final Reward: Assuming the final answer is correct (A=15), and the final reward (i.e., task reward information) Routcome is 1, the model will adjust based on the back-reasoning reward Rcountfactual. If the back-reasoning confidence is high, the model will be penalized (the higher the back-reasoning confidence, the larger λ, and the more severe the penalty, thus reducing the positive impact of incorrect reasoning steps on the final reward), and the final reward will be balanced and corrected.

[0136] Through the reverse reasoning reward mechanism, the model can reflect on and adjust its reasoning process, avoiding reliance on incorrect assumptions or paths. This mechanism not only makes the model more stable but also enhances its sensitivity to reasoning steps, especially when faced with complex multi-step reasoning tasks.

[0137] In summary, the reverse reasoning reward mechanism optimizes the reasoning path by guiding the model to reflect on each step of the reasoning process and assessing the impact of changes in certain assumptions on the final result. Through reverse reasoning confidence and reward calculation, the embodiments of this application can effectively improve the robustness and flexibility of large language models in complex reasoning tasks, avoiding errors in the results caused by incorrect steps in the reasoning process.

[0138] The following is an introduction to the hierarchical thinking reward mechanism.

[0139] The hierarchical thinking reward mechanism dynamically adjusts the depth and level of reasoning based on the complexity of the task, ensuring that the model can adaptively adjust its reasoning strategy when handling different types of tasks. Simple tasks rely on basic logical reasoning and fast computation, while complex tasks require the model to perform multi-level reasoning and higher-level abstraction.

[0140] By using hierarchical rewards, the model can adjust the reward distribution method during the reasoning process according to the complexity of the task, the number of steps, and the required depth of reasoning, thereby exhibiting the most suitable reasoning strategy in tasks of different complexities.

[0141] Principle Overview: The basic idea behind hierarchical reasoning reward mechanisms is that tasks of different complexities require different levels of reasoning strategies. Simple tasks should focus on basic reasoning steps, while complex tasks should include more abstract reasoning and steps. By assigning different reward weights to different levels of reasoning processes, the model can be guided to quickly reach conclusions in simple tasks and engage in deeper reasoning in complex tasks.

[0142] To implement hierarchical reward for thinking, the complexity of the task must first be evaluated. This application embodiment can evaluate task complexity in the following ways: Number of steps: The number of reasoning steps in the task. The more steps, the higher the task complexity.

[0143] Depth of reasoning: The level of abstraction in each step of reasoning. The more layers of reasoning, the higher the complexity of the task.

[0144] Background knowledge requirements: Does the task require additional background knowledge or reasoning assumptions? The more a task relies on external information, the higher its complexity.

[0145] Figure 5 This is a flowchart illustrating the model processing method provided in an exemplary embodiment of the application. Figure 4 ,like Figure 5 As shown, in step S230 above, the sample task also includes reasoning depth and background knowledge. Therefore, adjusting the reasoning steps to obtain adjusted reasoning steps can include: S230-2-1. When adjusting the inference strategy of the preset model to adjust the inference depth of the preset model, the complexity of the sample task is evaluated based on the number of inference steps, the inference depth, and the background knowledge to obtain the task complexity corresponding to the sample task.

[0146] In some embodiments, when the reasoning strategy of the preset model is adjusted, or the reasoning depth of the preset model is adjusted, it indicates that a hierarchical thinking reward mechanism is being implemented. In this case, the server can evaluate the complexity of the sample task based on the number of reasoning steps, the reasoning depth, and the background knowledge, and obtain the task complexity corresponding to the sample task.

[0147] In one approach, the server performs a complexity evaluation on the sample task based on the number of inference steps, the depth of inference, and background knowledge to obtain the task complexity corresponding to the sample task, which may include: The server determines the contribution of the number of inference steps, the depth of inference, and background knowledge to the task complexity. Based on this contribution, it assigns a weight to each of these factors, calculates the product of each factor, and sums the results to obtain the task complexity. For example, the task complexity calculation process includes: Ctask=α Steps+β Depth+γ Knowledge; Here, Ctask refers to task complexity, Steps refers to the number of reasoning steps, Depth refers to reasoning depth, Knowledge refers to background knowledge, and α, β, and γ refer to the weights of the number of reasoning steps, reasoning depth, and background knowledge, respectively.

[0148] Since the number of reasoning steps, the depth of reasoning, and background knowledge can all reflect the complexity of a task to some extent, the accuracy of determining the task complexity can be improved by considering the contribution of each of these three factors to the task complexity, thereby improving the accuracy of determining hierarchical thinking rewards.

[0149] S230-2-2. When the task complexity is greater than or equal to the second threshold, add new reasoning steps in at least two reasoning steps to obtain the adjusted reasoning steps.

[0150] S230-2-3. When the task complexity is less than the second threshold, reduce the reasoning steps from at least two reasoning steps to obtain the adjusted reasoning steps.

[0151] In one approach, if the task complexity is greater than or equal to a second threshold, it indicates that the task complexity is high and requires multi-level reasoning and higher-level abstraction. Therefore, new reasoning steps are added in at least two reasoning steps to obtain adjusted reasoning steps, enabling the model to perform multi-level reasoning and higher-level abstraction, thereby exhibiting the most suitable reasoning strategy in tasks of different complexities.

[0152] In another approach, if the task complexity is less than the second threshold, it indicates that the task complexity is low. Simple tasks rely on basic logical reasoning and fast computation. Therefore, by reducing the number of reasoning steps in at least two reasoning steps, the reasoning steps are adjusted so that the model relies on basic logical reasoning and fast computation to perform simple task processing, thereby exhibiting the most suitable reasoning strategy in tasks of different complexities.

[0153] Since the number of reasoning steps, the depth of reasoning, and background knowledge can all reflect the complexity of a task to some extent, determining the complexity based on these three factors can improve the accuracy of task complexity determination, thereby improving the accuracy of determining hierarchical thinking rewards. Furthermore, when the task complexity is greater than or equal to the second threshold, it indicates a high level of task complexity, requiring multi-level reasoning and higher-level abstraction. Therefore, adding new reasoning steps in at least two reasoning steps allows the model to perform multi-level reasoning and higher-level abstraction. Conversely, when the task complexity is less than the second threshold, it indicates a low level of task complexity. Simple tasks rely on basic logical reasoning and fast computation. Therefore, reducing the number of reasoning steps in at least two reasoning steps allows the model to rely on basic logical reasoning and fast computation for simple task processing, thus enabling the model to exhibit the most suitable reasoning strategy for tasks of different complexities.

[0154] In some embodiments, continue as follows Figure 5 As shown, in step S230 above, determining the step reward information corresponding to the adjusted inference step based on the degree of influence of the adjusted inference step on the processing result of the first sample task may include: S230-2-4. Determine the reasoning complexity of each adjusted reasoning step and the degree of impact of each adjusted reasoning step on the processing results of the first sample task.

[0155] S230-2-5. Based on the reasoning complexity of each adjustment reasoning step and the degree of influence of each adjustment reasoning step on the processing result of the first sample task, determine the reward information corresponding to each adjustment reasoning step.

[0156] In some embodiments, the reward information Wi corresponding to the adjusted inference step is determined based on the complexity of the adjusted inference step and its importance to the final result.

[0157] Simple tasks: When the task complexity is low, the model will give a higher reward for each step because the task requires fewer inference steps, and each inference step is more important, that is, each inference step has a greater impact on the processing result of the first sample task.

[0158] Complex tasks: When task complexity is high, there are more reasoning steps, resulting in more dispersed rewards for each step and relatively lower reward weights. If a particular adjustment reasoning step has high reasoning complexity and a significant impact on the outcome of the first sample task, the reward information for each adjustment reasoning step will be larger. Conversely, if a particular adjustment reasoning step has low reasoning complexity and a small impact on the outcome of the first sample task, the reward information for each adjustment reasoning step will be smaller. If the reasoning complexity of an adjustment reasoning step and its impact on the outcome of the first sample task are not directly proportional (e.g., high reasoning complexity, low impact, or low reasoning complexity, high impact), then weights can be assigned to reasoning complexity and impact, and the products of reasoning complexity and impact with their respective weights can be calculated. Finally, the sum of these two products yields the reward information for each adjustment reasoning step.

[0159] S230-2-6. Determine the sum of reward information corresponding to each adjustment reasoning step to obtain the step reward information.

[0160] Optionally, after the server determines that it has obtained the reward information corresponding to each adjustment reasoning step, it can calculate the sum of the reward information corresponding to each adjustment reasoning step to obtain the step reward information.

[0161] Since the reasoning complexity of each adjustment reasoning step and the degree of influence of each adjustment reasoning step on the processing result of the first sample task can reflect the reward level of that step in the entire reasoning process, determining the reward information corresponding to each adjustment reasoning step by using the reasoning complexity of each adjustment reasoning step and the degree of influence of each adjustment reasoning step on the processing result of the first sample task can improve the accuracy of determining the reward information corresponding to each adjustment reasoning step, thereby improving the accuracy of determining the step reward information of the hierarchical reward mechanism.

[0162] In some embodiments, in step S240 above, generating target reward information corresponding to the sample task based on step reward information and task reward information may include: Determine the second reward weight; the second reward weight is used to balance the step reward information and the task reward information.

[0163] Based on the second reward weight, step reward information, and task reward information, the second target reward information corresponding to the sample task is generated.

[0164] In some embodiments, the final second objective reward information should include not only hierarchical thinking rewards but also task reward information. In this case, the server can predetermine a second reward weight. The second reward weight is used to balance the step reward information (i.e., hierarchical thinking rewards) and the task reward information. The second reward weight can be dynamically adjusted to ensure that the model can reasonably balance the two reward signals during the reasoning process.

[0165] Next, based on the second reward weight, step reward information, and task reward information, the server generates the first target reward information corresponding to the sample task. The specific calculation formula can be as follows: Rtotal = λRoutcome + (1 λ)Rhierarchical; Where Rtotal refers to the second objective reward information, λ refers to the second reward weight, Routcome refers to the task reward information, and Rhierarchical refers to the step reward information.

[0166] Therefore, in the hierarchical reward mechanism, by combining step reward information and task reward information, the inference strategy can be adaptively adjusted when handling different types of tasks. Simple tasks rely on basic logical reasoning and fast computation, while complex tasks require the model to perform multi-level reasoning and higher-level abstraction. Through hierarchical rewards, the model can adjust the reward allocation method in its inference process according to the complexity of the task, the number of steps, and the required inference depth, thereby exhibiting the most suitable inference strategy in tasks of different complexities. This effectively solves the problems of models easily getting stuck in fixed inference paths and lacking flexibility and diversity during training, improving the model's inference accuracy and efficiency. In addition, the second reward weight can balance step reward information and task reward information, further ensuring the accuracy and stability of the generated second target reward information.

[0167] It's important to note that hierarchical reward information modifies only one inference step at a time; that is, the evaluation process for hierarchical reward information is performed step-by-step. Only after one inference step has been evaluated will the next inference step be evaluated. This step-by-step evaluation ensures the rationality and stability of each inference step, helping the model better identify and adjust for potential errors. Therefore, only one inference step can be evaluated at a time. This step-by-step analysis ensures that the model optimizes each inference stage, avoiding incorrect inference paths that could lead to an incorrect final result.

[0168] The following is a general explanation of the above-mentioned tiered reward process: Figure 6 This is a flowchart illustrating the model processing method provided in an exemplary embodiment of the application. Figure 5 ,like Figure 6As shown, the model processing method includes: 1) Obtain sample data; the sample data includes sample tasks and the corresponding reasoning steps, reasoning depth and background knowledge.

[0169] 2) Based on the number of reasoning steps, the depth of reasoning, and the background knowledge, the complexity of the sample task is evaluated to obtain the task complexity corresponding to the sample task.

[0170] The process of calculating the complexity of this task includes: Ctask=α Steps+β Depth+γ Knowledge.

[0171] 3) If the task complexity is greater than or equal to the second threshold, add a new reasoning step in at least two reasoning steps to obtain an adjusted reasoning step; if the task complexity is less than the second threshold, reduce the reasoning steps from at least two reasoning steps to obtain an adjusted reasoning step.

[0172] 4) Determine the reasoning complexity of each adjustment reasoning step and the degree of impact of each adjustment reasoning step on the processing result of the first sample task; based on the reasoning complexity of each adjustment reasoning step and the degree of impact of each adjustment reasoning step on the processing result of the first sample task, determine the reward information Wi corresponding to each adjustment reasoning step. Sum the reward information corresponding to each adjustment reasoning step to obtain the step reward information.

[0173] 5) Determine the second reward weight; the second reward weight is used to balance the step reward information and the task reward information; based on the second reward weight, the step reward information, and the task reward information, generate the second target reward information corresponding to the sample task. The calculation formula can be as follows: Rtotal = λRoutcome + (1 λ)Rhierarchical.

[0174] The following example illustrates the tiered reward process described above: Suppose there is a mathematical reasoning task: given a triangle, and knowing the measures of two of its angles, find the measure of the third angle.

[0175] Reasoning steps: The first step is for the model to identify that this is a math problem involving finding an angle, and then apply the basic triangle angle formula: 180° (Angle 1 + Angle 2).

[0176] The second step involves the model calculating the third angle: if angle 1 is 50° and angle 2 is 60°, then the third angle is 180°. (50° + 60°) = 70°.

[0177] Complexity Assessment: The reasoning process for this problem is very simple, involving only two steps, resulting in low task complexity. The model only needs to apply simple formulas for reasoning, thus requiring only two steps. The reasoning depth is shallow, the background knowledge requirement is low, and the overall complexity of Ctask is low.

[0178] Hierarchical rewards: Because this is a simple task, each inference step of the model (e.g., angle calculation) receives a high reward weight. For example: W1=W2=1 (higher reward weight because the steps are simpler).

[0179] It's important to note that simple tasks typically have higher reward weights for each reasoning step because they require less reasoning depth and abstraction. Steps in simple tasks, such as basic computations, usually receive similar or nearly identical high reward weights, for example, W1=W2=1. Complex tasks, however, have different reward weights, adjusted according to the complexity of each step. For instance, reasoning steps in complex tasks involve more abstraction and higher-level reasoning, thus receiving lower reward weights.

[0180] In summary, if the task is simple, the reward Wi for each step may be 1; while for complex tasks, the reward for each step will vary depending on the complexity.

[0181] In summary, the final hierarchical reward (Rhierarchical) will be higher, helping the model to quickly obtain the answer.

[0182] Complex Tasks: Assume the task becomes more complex, such as solving a geometric reasoning problem involving multiple steps and abstract geometric relationships. The model has many reasoning steps, and each step involves higher-level reasoning and abstraction, such as reasoning about the properties of triangles and angular relationships. In this case, the model's hierarchical reward will automatically adjust according to the complexity of each step, reducing the reward weight for complex steps and encouraging the model to perform deeper reasoning.

[0183] For example: W1=0.8, W2=0.6, W3=0.4 (lower reward weights; more complex reasoning steps will result in fewer rewards) Final reward: Combining the results and hierarchical rewards, the model will adaptively adjust each step of the reasoning process according to the complexity of the task, thereby quickly drawing conclusions in simple tasks and performing multi-level reasoning in complex tasks to obtain accurate answers.

[0184] Summarize: The hierarchical reasoning reward mechanism adaptively adjusts the reward strategy based on task complexity, enabling the model to employ the most appropriate reasoning depth and strategy across different tasks. Simple tasks focus on basic reasoning, while complex tasks guide the model to engage in multi-level reasoning and abstraction. The core advantage of this model lies in its ability to optimize the reasoning process and improve the reasoning quality across tasks of varying complexity.

[0185] In some embodiments, the above method may further include: First comprehensive reward information is generated based on first target reward information and second target reward information; the first target reward information is determined based on task reward information and step reward information determined by back reasoning confidence, the back reasoning confidence is the back reasoning confidence of the adjustment reasoning step corresponding to each reasoning step; the adjustment reasoning step corresponding to each reasoning step is to adjust each reasoning step to the corresponding modified reasoning step, etc., and the modified reasoning step corresponding to each reasoning step is obtained by modifying each reasoning step.

[0186] Accordingly, the inference strategy of the preset model is adjusted based on the target reward information to obtain the target model, including: The reasoning strategy of the preset model is adjusted based on the first comprehensive reward information to obtain the target model.

[0187] In some embodiments, the aforementioned reverse reasoning reward and hierarchical thinking reward can be used individually or in combination. When used individually, for the reverse reasoning reward, the server can use the first target reward information to adjust the inference strategy of the preset model to obtain the target model. For the hierarchical thinking reward, the server can use the second target reward information to adjust the inference strategy of the preset model to obtain the target model.

[0188] Optionally, when the reverse reasoning reward and the aforementioned hierarchical thinking reward are used together, the server can generate a first comprehensive reward information based on the first target reward information and the second target reward information, and adjust the reasoning strategy of the preset model based on the first comprehensive reward information to obtain the target model.

[0189] In one approach, "generating first comprehensive reward information based on first target reward information and second target reward information" may include: calculating the sum of the first target reward information and the second target reward information to obtain the first comprehensive reward information. In another approach, the influence of each of the first and second target reward information on the task processing result may be evaluated, and corresponding weights may be assigned to the first and second target reward information according to their respective influence levels. The products of each target reward information and its corresponding weights may be calculated, and finally, the two products may be added together to obtain the first comprehensive reward information.

[0190] Because the back-reasoning reward mechanism guides the model to reflect on each step of the reasoning process and assess the impact of changes in certain assumptions on the final result, it optimizes the reasoning path. Through back-reasoning confidence and reward calculation, it effectively improves the robustness and flexibility of large language models in complex reasoning tasks, avoiding errors in the result due to incorrect steps in the reasoning process. The hierarchical thinking reward mechanism adaptively adjusts the reward strategy according to the complexity of the task, enabling the model to adopt the most appropriate reasoning depth and strategy in different tasks. Simple tasks focus on basic reasoning, while complex tasks guide the model to perform multi-level reasoning and abstraction. The core advantage of this model lies in its ability to optimize the reasoning process and improve the reasoning quality of the model in tasks of various complexities. The combination of these two mechanisms effectively solves the problems that models easily get stuck in fixed reasoning paths and lack flexibility and diversity during training, thereby improving the model's reasoning accuracy and efficiency.

[0191] The following explains the cross-modal mutual assistance thinking reward mechanism.

[0192] Cross-modal collaborative thinking reward mechanisms aim to improve the reasoning ability of Large Language Models (LLMs) when processing multimodal information. In many practical tasks, information is not confined to a single modality; it is often necessary to extract information from multiple modalities (such as text, images, and videos) and combine these modalities for reasoning. For example, given an image and a text describing that image, the model needs to simultaneously understand the visual information in the image and the linguistic information in the text, and fuse these two to arrive at the correct answer.

[0193] Traditional large language models typically process data from different modalities separately when handling multimodal tasks, lacking effective intermodal collaboration, resulting in less than expected inference performance. To address this issue, this invention proposes a cross-modal collaborative thinking reward mechanism. This mechanism rewards the model for its collaboration across different modalities, encouraging it to more effectively integrate information from different modalities, thereby improving the inference quality for multimodal tasks.

[0194] Principle Overview: The basic idea behind the cross-modal collaborative thinking reward mechanism is that when a task involves multiple modalities (such as images and text), the model not only needs to understand the information from each modality, but also needs to be able to effectively combine the knowledge and information from these modalities. By rewarding the model's synergy between modalities, the model is encouraged to fully utilize the complementary information between different modalities during the reasoning process. This collaboration and mutual assistance between modalities will enable the model to better understand the task and ultimately obtain more accurate reasoning results.

[0195] In this application embodiment, the cross-modal collaboration reward is based on the following key points: Modal information integration: The model needs to understand and combine information from different modalities, rather than just processing the independent content of each modality.

[0196] Modal complementarity: Information from different modalities can often complement each other, and models enhance reasoning by understanding the relationships between modalities.

[0197] Modal collaboration reward: By rewarding steps that effectively integrate information from multiple modalities for reasoning, the model is guided to improve its ability to reason across modalities.

[0198] Figure 7 This is a flowchart illustrating the model processing method provided in an exemplary embodiment of the application. Figure 6 ,like Figure 7 As shown, in some embodiments, the above method may further include: S310. When the sample task includes at least two modal information, determine the modal correlation between at least two modalities.

[0199] In some embodiments, when the sample task includes at least two modalities, the server may first determine the modal correlation between the at least two modalities. Modal correlation refers to an indicator that measures the consistency and similarity of information between different modalities (such as images and text), and is used to determine the effectiveness of information fusion between modalities.

[0200] Optionally, the at least two modal information may include, but is not limited to, at least two of the following modalities: image, text, audio, video, etc.

[0201] Alternatively, modal correlation can be represented by similarity. For example, the server can first extract modal features corresponding to at least two modal informations using models such as convolutional neural networks (CNNs), and then calculate the modal correlation between the modal features corresponding to the at least two modal informations.

[0202] Taking at least two modalities, including images and text, as an example, the calculation process of modal correlation is explained: Assuming the input received by the pre-defined model is multimodal, including image I and text T, the model needs to perform inference based on information from both modalities. First, features are extracted from image I using a model such as a Convolutional Neural Network (CNN), and text T is processed using a language model (such as Transformer). In this way, the model can obtain feature representations of both the image and the text. Ifeat=CNN(I),Tfeat=Transformer(T); Where Ifeat and Tfeat represent the feature vectors of the image and text, respectively.

[0203] To enable the pre-defined model to effectively combine information from different modalities, the server can first evaluate the modal relevance between images and text. Modal relevance reflects the semantic overlap between images and text, and is typically obtained by calculating the similarity between image and text features. ; Here, Sim(Ifeat, Tfeat) represents the cosine similarity between image and text features. A higher similarity means that the image and text content are strongly correlated, and the model needs to better integrate these two types of information.

[0204] S320. Generate cross-modal reward information based on modal correlation.

[0205] Next, the server can generate cross-modal reward information based on modal correlation.

[0206] In one approach, a model capable of generating cross-modal reward information based on modal relevance can be pre-trained, and this model is used to generate cross-modal reward information.

[0207] In another approach, the generation of cross-modal reward information based on modal correlation can include: If the modal correlation is greater than or equal to the third threshold, determine the degree of fusion of the preset model with at least two modal information; generate a third reward weight based on the degree of fusion; and generate cross-modal reward information based on the third reward weight and modal correlation.

[0208] In this embodiment, when the modal correlation is greater than or equal to the third threshold, it indicates a high correlation between different modal information. At this point, the degree of fusion of the preset model with at least two modal information is further determined based on whether the preset model can effectively combine the two for inference. If the preset model can effectively combine the two for inference, it is determined that the preset model has a high degree of fusion with at least two modal information; if the preset model cannot effectively combine the two for inference, it is determined that the preset model has a low degree of fusion with at least two modal information. For example, in the case where multimodal information includes images and text, the formula for calculating cross-modal reward information can be as follows: Rcross-modal=Wmodality Sim(Ifeat,Tfeat); Rcross-modal represents cross-modal reward information, and Wmodality represents the third reward weight.

[0209] Next, the server generates a third reward weight based on the degree of fusion. Optionally, if the modal correlation is greater than or equal to the third threshold, the model receives a higher cross-modal reward if the preset model can effectively combine the two modalities for reasoning. Conversely, if the modal correlation is greater than or equal to the third threshold, the model receives a lower cross-modal reward if the preset model cannot effectively combine the two modalities for reasoning. For example, in an image question-answering task, the question might involve an object in an image, the description of which can be found in the text. In this case, if the model can combine information from both the image and the text to arrive at the correct answer, it receives a higher cross-modal collaborative reward; if the model cannot combine information from both the image and the text to arrive at the correct answer, it receives a lower cross-modal collaborative reward. This ensures that the determination of the modal collaborative reward is closely related to the degree of fusion of at least two modalities by the preset model, improving the accuracy of cross-modal reward information determination.

[0210] In other embodiments, when the modal correlation is less than a third threshold, it indicates that the correlation between different modal information is low, and thus lower cross-modal reward information can be obtained.

[0211] S330. Generate third-objective reward information based on cross-modal reward information and task reward information; the third-objective reward information is used to adjust the inference strategy of the preset model.

[0212] In some embodiments, the final third objective reward information includes not only cross-modal reward information but also task reward information. In this case, the server can predetermine a reward weight. The reward weight is used to balance the cross-modal reward information and the task reward information. The reward weight can be dynamically adjusted to ensure that the model can reasonably balance the two reward signals during the inference process.

[0213] Next, based on the reward weight, cross-modal reward information, and task reward information, the server generates the first-order target reward information corresponding to the sample task. The specific calculation formula can be as follows: Rtotal = λRoutcome + (1 λ)Rcross-modal; Where Rtotal refers to the third objective reward information, λ refers to the reward weight, Routcome refers to the task reward information, and Rcross-modal refers to the cross-modal reward information.

[0214] Therefore, in the cross-modal reward mechanism, by combining cross-modal reward information and task reward information, the model can not only understand the information of each modality, but also effectively combine the knowledge and information of these modalities. By rewarding the model's synergistic effect between modalities, the model is encouraged to fully utilize the complementary information between different modalities during the reasoning process. This collaboration and mutual assistance between modalities will enable the model to better understand the task and ultimately obtain more accurate reasoning results.

[0215] The following section uses the example of cross-modal reward information including text and images to provide an overall explanation of the cross-modal reward process: Figure 8 This is a flowchart illustrating the model processing method provided in an exemplary embodiment of the application. Figure 7 ,like Figure 8 As shown, the model processing method includes: 1) Input images and text.

[0216] 2) Extract image features using CNN and extract text features using Transformer.

[0217] 3) Calculate the modal correlation between images and text.

[0218] 4) When the modal correlation is greater than or equal to the third threshold, determine the degree of fusion of at least two modal information by the preset model; generate a third reward weight based on the degree of fusion; generate cross-modal reward information based on the third reward weight and modal correlation. When the modal correlation is less than the third threshold, assign lower cross-modal reward information.

[0219] 5) Generate third-objective reward information based on cross-modal reward information and task reward information.

[0220] The following example, using text and images as examples of cross-modal reward information, illustrates the process of cross-modal reward: Suppose there is an image question answering task, the question is: "How many cats are in the image?", and the following image and text description are provided: Image: A picture containing two cats.

[0221] Text: Description of the image content: "There are two cats in the image." The pre-defined model first extracts features (Ifat) from the image using a CNN and features (Tfeat) from the text using a Transformer. Then, it calculates the similarity between the image and text features to obtain modal correlation. Sim(Ifeat,Tfeat)=0.95; Because the content of the image and text is highly correlated, the model can clearly combine the information from the image and text, resulting in a high degree of fusion, which can be set to 1, i.e., Wmodality is 1. During the inference process, the preset model identifies two cats in the image and combines this with the text description to derive the answer.

[0222] Cross-modal reward information is calculated as follows: Rcross-modal=Wmodality Sim(Ifeat,Tfeat)=1.0×0.95=0.95.

[0223] Assuming the final answer is correct ("two cats"), let the task reward information Routcome obtained by the preset model be 1. The final third objective reward information is: Rtotal=λ1+(1 λ) 0.95.

[0224] It is evident that the pre-defined model can effectively combine image and text information to obtain higher cross-modal collaborative rewards, enabling the model to make fuller use of the relationship between images and text in other similar tasks.

[0225] In summary, the cross-modal collaborative thinking reward mechanism guides the model to better integrate information from different modalities for reasoning by evaluating the collaboration between multiple modalities such as images and text. The core of this model lies in improving the model's reasoning ability in multimodal tasks by calculating modal relevance and cross-modal collaborative rewards during the reasoning process. This is particularly true in complex tasks such as image question answering and visual reasoning, where the model can fully leverage the complementarity of information between modalities, thereby improving the accuracy of the reasoning results.

[0226] In some embodiments, generating target reward information corresponding to a sample task based on step reward information and task reward information may include: Based on step reward information, task reward information and third objective reward information, generate second comprehensive reward information corresponding to the sample task; Accordingly, the inference strategy of the preset model is adjusted based on the target reward information to obtain the target model, including: The reasoning strategy of the preset model is adjusted based on the second comprehensive reward information to obtain the target model.

[0227] In some embodiments, the aforementioned cross-modal rewards, reverse reasoning rewards, and hierarchical thinking rewards can be used individually or in combination. When used individually, for reverse reasoning rewards, the server can use the first target reward information to adjust the inference strategy of the preset model to obtain the target model. For hierarchical thinking rewards, the server can use the second target reward information to adjust the inference strategy of the preset model to obtain the target model. For cross-modal rewards, the server can use the third target reward information to adjust the inference strategy of the preset model to obtain the target model.

[0228] Optionally, when cross-modal rewards are used in conjunction with other reward information, the server can generate a second comprehensive reward information based on third target reward information, step reward information (backward reasoning reward and / or hierarchical thinking reward), and task reward information. Based on this second comprehensive reward information, the inference strategy of the preset model is adjusted to obtain the target model. In one approach, the sum of step reward information (backward reasoning reward) and task reward information can be calculated to obtain the first target reward information; the sum of step reward information (hierarchical thinking reward) and task reward information can be calculated to obtain the second target reward information; finally, the second comprehensive reward information is obtained based on the first target reward information, the second target reward information, and the third target reward information. In another approach, the weight of each reward information can be evaluated, the product of each reward information and its corresponding weight can be calculated, and then all products can be summed to obtain the second comprehensive reward information.

[0229] Because the back-reasoning reward mechanism guides the model to reflect on each step of the reasoning process and assess the impact of changes in certain assumptions on the final result, it optimizes the reasoning path. Through back-reasoning confidence and reward calculation, it effectively improves the robustness and flexibility of large language models in complex reasoning tasks, avoiding errors in the results due to incorrect steps in the reasoning process. The hierarchical thinking reward mechanism adaptively adjusts the reward strategy according to the complexity of the task, enabling the model to adopt the most appropriate reasoning depth and strategy in different tasks, improving the reasoning quality of the model in tasks of varying complexity. The cross-modal collaborative thinking reward mechanism guides the model to better combine information from different modalities for reasoning through multimodal collaboration, helping the model improve its reasoning ability in multimodal tasks, especially in complex tasks such as image question answering and visual reasoning, fully leveraging the complementarity of information between modalities to improve the accuracy of reasoning results. Therefore, through the combined effect of back-reasoning rewards, hierarchical thinking rewards, and cross-modal collaborative thinking rewards, it is possible to effectively solve problems such as models easily getting stuck in fixed reasoning paths and lacking flexibility and diversity during training, thereby improving the model's reasoning accuracy and efficiency.

[0230] This application also provides a task processing method, the method comprising: Obtain the target task.

[0231] The target task is input into the target model for processing to obtain the task processing result corresponding to the target task; wherein, the target model is trained based on the model processing method of any of the above embodiments.

[0232] In this embodiment, after the target model is trained through the above training process, the target task can be directly input into the target model. The target task is then processed based on the inference strategy in the target model to obtain the corresponding task processing result. Since the target model is trained by combining step reward information and task reward information, the model can consider not only the final result during training but also whether the final answer will change after certain steps are adjusted. Therefore, it can improve the processing efficiency and accuracy of the task processing result.

[0233] The multimodal reward mechanism (including a reverse reasoning reward mechanism, a hierarchical thinking reward mechanism, and a cross-modal collaborative thinking reward mechanism) provided in this application can be applied to various large language model (LLM) products, especially in tasks requiring complex or multimodal reasoning, where it can effectively improve performance. The following are examples of how this technology is applied in actual products and the effects it brings: Product method: Integration into Existing Language Models: By embedding at least one of the following reward mechanisms—backward reasoning rewards, hierarchical rewards, and cross-modal collaboration rewards—into existing multimodal language models, and combining this with a standard training framework, embodiments of this application can guide the model to perform more rational thinking and reasoning in different reasoning tasks. It should be noted that these reward mechanisms are calculated and adjusted synchronously with the model's reasoning process during each training iteration. When the model performs reasoning, the reward mechanism provides real-time reward feedback based on the current reasoning step, path, and modal collaboration. Then, based on this feedback, the model adjusts its reasoning strategy, gradually improving its reasoning ability. Therefore, the entire optimization process is dynamic and real-time, rather than a static offline adjustment. Automated inference path adjustment: In multimodal inference tasks (such as visual question answering, image description generation, etc.), the system automatically analyzes the relationship between images and text, dynamically adjusts the complexity of the inference path, and ensures the accuracy and consistency of the task.

[0234] Reward Mechanism Optimization: Through the reverse reasoning reward mechanism, the product can optimize the credibility and stability of each reasoning step, ensuring the reliability of the reasoning results.

[0235] The training method provided in this application can achieve the following beneficial effects: Improve the accuracy and stability of reasoning: By introducing a reverse reasoning reward mechanism, the model can identify potential errors in the reasoning steps and avoid relying on erroneous reasoning paths. Experiments show that in tasks involving multiple reasoning steps, the model reduces the reasoning error rate by 30%. Especially when dealing with complex reasoning tasks, the model is better able to correct erroneous paths and ensure the accuracy of the final result.

[0236] Optimize the collaboration of multimodal tasks: In multimodal tasks, the cross-modal collaborative thinking reward mechanism significantly improved the model's performance in image-text fusion reasoning tasks. By calculating the modal correlation between images and text, the model can more effectively combine image and text information, improving the accuracy and robustness of reasoning. Experimental data show that in visual question answering (VQA) tasks, the model's accuracy improved by 15%, and in multimodal reasoning tasks combining images and text, the cross-modal collaborative reward improved the model's accuracy by 12% compared to traditional methods.

[0237] Improve the generalization ability of cross-modal reasoning tasks: In multimodal tasks, the model not only accurately infers the current task but also generalizes well without direct training data. By combining back-reasoning rewards and hierarchical rewards, the model can quickly adapt to new types of reasoning tasks. Experimental results show that this method improves the model's generalization ability by 10%-15% in reasoning tasks across different domains, especially in cross-modal tasks.

[0238] Figure 9 This is a block diagram of a model processing apparatus provided in an exemplary embodiment of this application, such as... Figure 9 As shown, the model processing device includes: The sample data acquisition module 410 is used to acquire sample data; the sample data includes the sample task and the inference steps corresponding to the sample task. The model prediction module 420 is used to input the sample data into a preset model, process the sample task according to the inference steps based on the preset model, and obtain a first sample task processing result corresponding to the sample task; and determine task reward information based on the difference between the first sample task processing result and the actual task processing result corresponding to the sample task. The adjustment module 430 is used to adjust the inference steps to obtain adjusted inference steps; and to determine the step reward information corresponding to the adjusted inference steps based on the degree of influence of the adjusted inference steps on the processing result of the first sample task. The target reward information generation module 440 is used to generate target reward information corresponding to the sample task based on the step reward information and the task reward information; The strategy adjustment module 450 is used to adjust the inference strategy of the preset model based on the target reward information to obtain the target model.

[0239] In an optional embodiment, the number of inference steps is at least two, and the adjustment module is further configured to: When the reasoning strategy of the preset model is adjusted, that is, the reasoning path of the preset model is adjusted, each reasoning step is modified to obtain the modified reasoning step corresponding to each reasoning step; Based on the preset model, the sample task is processed according to the modified inference steps corresponding to each inference step to obtain the second sample task processing result corresponding to the sample task. If the difference between the processing result of the first sample task and the processing result of the second sample task is greater than or equal to a first threshold, each inference step is adjusted to a corresponding modified inference step to obtain the adjusted inference step corresponding to each inference step.

[0240] In an optional embodiment, the adjustment module is further configured to: Based on the degree of influence of the adjusted inference step corresponding to each inference step on the processing result of the first sample task, the reverse inference confidence of the adjusted inference step corresponding to each inference step is determined. The confidence scores of the reverse reasoning are weighted to obtain the reward information for the step.

[0241] In an optional embodiment, the adjustment module is further configured to: Determine the first partial derivative of the adjusted inference step corresponding to each inference step, and the second partial derivative corresponding to the processing result of the first sample task; The ratio of the second partial derivative to the first partial derivative is determined to obtain the reverse reasoning confidence of the adjusted reasoning step corresponding to each reasoning step.

[0242] In an optional embodiment, the target reward information generation module is further configured to: Determine a first reward weight; the first reward weight is used to balance the step reward information and the task reward information. Based on the first reward weight, the step reward information, and the task reward information, the first target reward information corresponding to the sample task is generated.

[0243] In an optional embodiment, the sample data further includes inference depth and background knowledge, wherein the inference depth characterizes the inference complexity of the inference step, and the background knowledge characterizes the background knowledge and / or inference assumptions required to perform the sample task; The adjustment module is also used for: When the inference strategy of the preset model is adjusted to adjust the inference depth of the preset model, the complexity of the sample task is evaluated based on the number of inference steps, the inference depth, and the background knowledge to obtain the task complexity corresponding to the sample task. If the task complexity is greater than or equal to the second threshold, a new reasoning step is added to at least two of the reasoning steps to obtain the adjusted reasoning step; If the task complexity is less than the second threshold, the inference steps are reduced from at least two inference steps to obtain the adjusted inference steps.

[0244] In an optional embodiment, the number of adjustment inference steps is at least two, and the adjustment module is further configured to: Determine the reasoning complexity of each of the adjusted reasoning steps, and the degree of impact of each of the adjusted reasoning steps on the processing result of the first sample task; Based on the reasoning complexity of each of the adjusted reasoning steps and the degree of influence of each of the adjusted reasoning steps on the processing result of the first sample task, the reward information corresponding to each of the adjusted reasoning steps is determined. The sum of the reward information corresponding to each of the aforementioned adjustment reasoning steps is determined to obtain the step reward information.

[0245] In an optional embodiment, the target reward information generation module is further configured to: Determine a second reward weight; the second reward weight is used to balance the step reward information and the task reward information. Based on the second reward weight, the step reward information, and the task reward information, the second target reward information corresponding to the sample task is generated.

[0246] In an optional embodiment, the apparatus further includes: A first comprehensive reward information generation device is used to generate first comprehensive reward information based on first target reward information and second target reward information; the first target reward information is determined based on the task reward information and step reward information determined by back-inference confidence, wherein the back-inference confidence is the back-inference confidence of the adjustment inference step corresponding to each inference step; the adjustment inference step corresponding to each inference step is to adjust each inference step to the corresponding modified inference step, etc., and the modified inference step corresponding to each inference step is obtained by modifying each inference step. Accordingly, the strategy adjustment module is also used to: adjust the reasoning strategy of the preset model based on the first comprehensive reward information to obtain the target model.

[0247] In an optional embodiment, the above-described apparatus further includes: A modal correlation determination module is used to determine the modal correlation between at least two modalities when the sample task includes at least two modal information. A cross-modal reward information generation module is used to generate cross-modal reward information based on the modal correlation; The third objective reward information generation module is used to generate third objective reward information based on the cross-modal reward information and the task reward information; the third objective reward information is used to adjust the inference strategy of the preset model.

[0248] In an optional embodiment, the cross-modal reward information generation module is further configured to: If the modal correlation is greater than or equal to a third threshold, determine the degree of fusion of the preset model with the at least two modal information; A third reward weight is generated based on the degree of fusion. The cross-modal reward information is generated based on the third reward weight and the modal correlation.

[0249] In an optional embodiment, the target reward information generation module is further configured to: Based on the step reward information, the task reward information, and the third target reward information, generate the second comprehensive reward information corresponding to the sample task; Accordingly, the aforementioned strategy adjustment module is also used for: The reasoning strategy of the preset model is adjusted based on the second comprehensive reward information to obtain the target model.

[0250] Figure 10 An exemplary embodiment of the application provides a block diagram of a task processing apparatus, such as Figure 10 The task processing device includes: The target task acquisition module 510 is used to acquire target tasks. Task processing module 520 is used to input the target task into the target model for processing and obtain the task processing result corresponding to the target task; The target model is trained based on the model processing method described in any of the above embodiments.

[0251] It should be noted that the device embodiments provided in this application are based on the same inventive concept as the method embodiments described above.

[0252] This application also provides an electronic device, which includes a processor and a memory. The memory stores a computer program, which is loaded and executed by the processor to implement the model processing method or task processing method provided in any of the above embodiments.

[0253] Embodiments of this application also provide a computer-readable storage medium storing a computer program that is loaded and executed by a processor to implement the model processing method or task processing method provided in any of the above embodiments.

[0254] Optionally, in the embodiments of this specification, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0255] The memory described in this specification can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for functions, etc.; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0256] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the model processing method or task processing method provided in the above-described method embodiments.

[0257] The model processing method or task processing method embodiments provided in this application can be executed in a terminal, computer terminal, server, or similar computing device. Taking running on a server as an example, Figure 11 This is a hardware structure block diagram of a server according to an exemplary embodiment. For example... Figure 11As shown, the server 600 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 610 (CPUs 610 may include, but are not limited to, microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 630 for storing data, and one or more storage media 620 (e.g., one or more mass storage devices) for storing application programs 623 or data 622. The memory 630 and storage media 620 may be temporary or persistent storage. The program stored in the storage media 620 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 610 may be configured to communicate with the storage media 620 and execute the series of instruction operations in the storage media 620 on the server 600. Server 600 may also include one or more power supplies 660, one or more wired or wireless network interfaces 650, one or more input / output interfaces 640, and / or one or more operating systems 621, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0258] The input / output interface 640 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 600. In one example, input / output interface 640 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, input / output interface 640 may be a radio frequency (RF) module used for wireless communication with the Internet.

[0259] Those skilled in the art will understand that Figure 11 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 600 may also include... Figure 11 The more or fewer components shown, or having the same Figure 11 The different configurations shown.

[0260] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0261] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and server embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0262] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0263] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A model processing method characterized by comprising: The method comprises: acquiring sample data; the sample data comprises a sample task and an inference step corresponding to the sample task; inputting the sample data into a preset model, processing the sample task according to the inference step based on the preset model to obtain a first sample task processing result corresponding to the sample task; determining task reward information based on a difference between the first sample task processing result and a real task processing result corresponding to the sample task; adjusting the inference step to obtain an adjusted inference step; determining step reward information corresponding to the adjusted inference step according to an influence degree of the adjusted inference step on the first sample task processing result; generating target reward information corresponding to the sample task based on the step reward information and the task reward information; adjusting an inference strategy of the preset model based on the target reward information to obtain a target model.

2. The model processing method of claim 1, wherein, The number of inference steps is at least two, and the adjusting of the inference step to obtain an adjusted inference step comprises: in the case of adjusting the inference strategy of the preset model, modifying each inference step to obtain a modified inference step corresponding to each inference step; processing the sample task according to the modified inference step corresponding to each inference step based on the preset model to obtain a second sample task processing result corresponding to the sample task; in the case that the difference between the first sample task processing result and the second sample task processing result is greater than or equal to a first threshold, adjusting each inference step to the corresponding modified inference step to obtain an adjusted inference step corresponding to each inference step.

3. The model processing method according to claim 2, characterized by, The determining of the step reward information corresponding to the adjusted inference step according to the influence degree of the adjusted inference step on the first sample task processing result comprises: determining a reverse inference confidence of the adjusted inference step corresponding to each inference step according to the influence degree of the adjusted inference step corresponding to each inference step on the first sample task processing result; weighting the reverse inference confidence to obtain the step reward information.

4. The model processing method according to claim 3, characterized by, The determining of the reverse inference confidence of the adjusted inference step corresponding to each inference step according to the influence degree of the adjusted inference step corresponding to each inference step on the first sample task processing result comprises: determining a first partial derivative of the adjusted inference step corresponding to each inference step and a second partial derivative corresponding to the first sample task processing result; determining a ratio of the second partial derivative and the first partial derivative to obtain the reverse inference confidence of the adjusted inference step corresponding to each inference step.

5. The model processing method according to claim 2, wherein The generating of the target reward information corresponding to the sample task based on the step reward information and the task reward information comprises: determining a first reward weight; the first reward weight is used to balance the step reward information and the task reward information. generate first target reward information corresponding to the sample task based on the first reward weight, the step reward information and the task reward information.

6. The model processing method of claim 1, wherein The sample data further comprises reasoning depth and background knowledge, wherein the reasoning depth represents reasoning complexity of the reasoning steps, and the background knowledge represents background knowledge and / or reasoning assumption required for performing the sample task. The adjusting the reasoning steps to obtain adjusted reasoning steps comprises: In a case that the adjusting the reasoning strategy of the preset model comprises adjusting reasoning depth of the preset model, performing complexity evaluation processing on the sample task according to the number of the reasoning steps, the reasoning depth and the background knowledge to obtain task complexity corresponding to the sample task; In a case that the task complexity is greater than or equal to a second threshold, adding a new reasoning step in at least two of the reasoning steps to obtain the adjusted reasoning steps; In a case that the task complexity is less than the second threshold, reducing a reasoning step from at least two of the reasoning steps to obtain the adjusted reasoning steps.

7. The model processing method according to claim 6, characterized by, The number of the adjusted reasoning steps is at least two, and the determining step reward information corresponding to the adjusted reasoning steps according to the influence degree of the adjusted reasoning steps on the first sample task processing result comprises: determining reasoning complexity of each of the adjusted reasoning steps and influence degree of each of the adjusted reasoning steps on the first sample task processing result; determining reward information corresponding to each of the adjusted reasoning steps according to the reasoning complexity of each of the adjusted reasoning steps and the influence degree of each of the adjusted reasoning steps on the first sample task processing result; determining a sum of the reward information corresponding to each of the adjusted reasoning steps to obtain the step reward information.

8. The model processing method according to claim 6, characterized by, The generating target reward information corresponding to the sample task based on the step reward information and the task reward information comprises: determining a second reward weight; the second reward weight is used to balance the step reward information and the task reward information; generating second target reward information corresponding to the sample task based on the second reward weight, the step reward information and the task reward information. 9.The model processing method of claim 8, wherein, The method further comprises: generating first comprehensive reward information based on the first target reward information and the second target reward information; the first target reward information is determined based on the task reward information and step reward information determined based on reverse reasoning confidence; the reverse reasoning confidence is reverse reasoning confidence of an adjusted reasoning step corresponding to each of the reasoning steps; the adjusted reasoning step corresponding to each of the reasoning steps is obtained by adjusting each of the reasoning steps to a corresponding modified reasoning step; and the modified reasoning step corresponding to each of the reasoning steps is obtained by modifying each of the reasoning steps. The adjusting the reasoning strategy of the preset model based on the target reward information to obtain a target model comprises: adjusting the reasoning strategy of the preset model based on the first comprehensive reward information to obtain a target model.

10. The model processing method of any one of claims 1 to 9, characterized in that, The method further comprises: In a case where the sample task includes at least two modal information, a modal correlation between the at least two modal information is determined; cross-modal reward information is generated according to the modal correlation; third target reward information is generated based on the cross-modal reward information and the task reward information; the third target reward information is used to adjust an inference strategy of the preset model.

11. The model processing method of claim 10, wherein, The cross-modal reward information is generated according to the modal correlation, including: In a case where the modal correlation is greater than or equal to a third threshold, a fusion degree of the preset model to the at least two modal information is determined; a third reward weight is generated according to the fusion degree; the cross-modal reward information is generated according to the third reward weight and the modal correlation.

12. The model processing method of claim 10, wherein, The target reward information corresponding to the sample task is generated based on the step reward information and the task reward information, including: second comprehensive reward information corresponding to the sample task is generated based on the step reward information, the task reward information and the third target reward information; The inference strategy of the preset model is adjusted based on the target reward information to obtain a target model, including: The inference strategy of the preset model is adjusted based on the second comprehensive reward information to obtain a target model.

13. A model processing apparatus characterized by comprising: The device includes: a sample data acquisition module configured to acquire sample data; the sample data includes a sample task and an inference step corresponding to the sample task; a model prediction module configured to input the sample data into a preset model, process the sample task according to the inference step based on the preset model, and obtain a first sample task processing result corresponding to the sample task; and determine task reward information based on a difference between the first sample task processing result and a real task processing result corresponding to the sample task; an adjustment module configured to adjust the inference step to obtain an adjusted inference step; and determine step reward information corresponding to the adjusted inference step according to an influence degree of the adjusted inference step on the first sample task processing result; a target reward information generation module configured to generate target reward information corresponding to the sample task based on the step reward information and the task reward information; a strategy adjustment module configured to adjust an inference strategy of the preset model based on the target reward information to obtain a target model.

14. An electronic device, comprising: The electronic device includes a processor and a memory, the memory stores a computer program, the computer program is loaded and executed by the processor to implement the model processing method in any one of claims 1-12.

Images