Model parameter determination method and device, storage medium and program product

Abstract

The embodiment of the invention provides a model parameter determination method and device, a storage medium and a program product, and the method comprises the steps: processing input data through employing a specified model, so as to generate at least one piece of thinking process information and corresponding answer information, and at least part of the thinking process information corresponds to a plurality of pieces of answer information; performing evaluation processing on the input data based on at least part of the answer information to determine corresponding optimization feedback information; updating at least one parameter of the specified model according to the optimization feedback information; wherein the specified model is a visual language model or a large language model, and the input data comprises at least one of text data, image data and point cloud data. Through multi-answer sampling, optimization feedback of more answer levels can be obtained under the condition that the generation number in the thinking process is reduced or is kept unchanged, so that the training time and the calculation overhead are remarkably reduced, the training efficiency is improved, and the thinking and summarization ability of the model can be improved under limited calculation resources through a training scheme after reinforcement learning.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and large model technology, and in particular to methods and apparatus for determining model parameters, storage media, and program products. Background Technology

[0002] With the rapid development of Large Language Models (LLMs) and Vision Language Models (VLMs), the application capabilities of artificial intelligence in natural language processing, computer vision, and multimodal tasks have been significantly enhanced. These models typically acquire perception and generation capabilities through large-scale data training, enabling them to perform tasks such as question answering, reasoning, image understanding, and cross-modal information processing. However, in practical applications, the training efficiency of these models is relatively low.

[0003] Based on this, embodiments of this application provide a method and apparatus for determining model parameters, a storage medium, and a program product to improve related technologies. Summary of the Invention

[0004] The purpose of this application is to provide a method and apparatus for determining model parameters, a storage medium, and a program product, thereby reducing training time and computational overhead and improving training efficiency.

[0005] The objective of this application embodiment is achieved using the following technical solutions:

[0006] In a first aspect, embodiments of this application provide a method for determining model parameters. The method includes: processing input data using a specified model to generate at least one thought process information and its corresponding answer information, wherein at least some of the thought process information corresponds to multiple answer information; evaluating the input data based on at least some of the answer information to determine corresponding optimization feedback information; and updating at least one parameter of the specified model according to the optimization feedback information; wherein the specified model is a visual language model or a large language model, and the input data includes at least one of text data, image data, and point cloud data.

[0007] In some embodiments, the optimization feedback information includes reward values ​​corresponding to at least some of the answer information. The step of evaluating the input data based on at least some of the answer information to determine the corresponding optimization feedback information includes: performing reward evaluation on at least some of the answer information based on reference answer information and preset reward evaluation rules to obtain the corresponding reward values.

[0008] In some embodiments, the optimized feedback information further includes a thinking advantage value corresponding to at least some of the thinking process information. The step of evaluating the input data based on at least some of the answer information to determine the corresponding optimized feedback information further includes: evaluating the value of at least some of the thinking process information based on the reward value of its corresponding partial or all answer information to obtain the corresponding value; and performing advantage analysis on at least some of the thinking process information according to the at least some value to determine the corresponding thinking advantage value.

[0009] In some embodiments, the value assessment process includes: substituting the reward value of some or all of the answer information corresponding to the thinking process information into a preset value calculation formula to calculate the corresponding value; the advantage analysis process includes: performing numerical statistical analysis based on at least some of the value to obtain value statistics; substituting the value corresponding to the thinking process information and the value statistics into a preset thinking advantage calculation formula to calculate the corresponding thinking advantage value.

[0010] In some embodiments, the optimization feedback information further includes answer advantage values ​​corresponding to at least some answer information. The step of evaluating the input data based on at least some answer information to determine the corresponding optimization feedback information further includes: performing numerical statistical analysis based on at least some reward values ​​to obtain reward value statistics; and substituting the reward values ​​corresponding to the answer information and the reward value statistics into a preset answer advantage calculation formula to calculate the corresponding answer advantage value.

[0011] In some embodiments, the designated model is further configured to output the confidence distribution information of at least a portion of the thinking process information and the confidence distribution information of at least a portion of the answer information, wherein the optimization feedback information includes the thinking advantage value corresponding to at least a portion of the thinking process information and the answer advantage value corresponding to at least a portion of the answer information; updating at least one parameter of the designated model based on the optimization feedback information includes: performing gradient calculation processing based on the thinking advantage value and the confidence distribution information corresponding to at least a portion of the thinking process information, and the answer advantage value and the answer confidence distribution information corresponding to at least a portion of the answer information, to obtain target gradient information for parameter updating; and updating at least one parameter of the designated model based on the target gradient information.

[0012] In some embodiments, the step of performing gradient calculation based on the thinking advantage value and thinking confidence distribution information corresponding to at least some thinking process information, and the answer advantage value and answer confidence distribution information corresponding to at least some answer information, to obtain target gradient information for parameter updating includes: performing a combined operation on the thinking advantage value and thinking confidence distribution information corresponding to at least some thinking process information to obtain a thinking gradient term related to the thinking process information; performing a combined operation on the answer advantage value and answer confidence distribution information corresponding to at least some answer information to obtain an answer gradient term related to the answer information; and performing constraint processing on the thinking gradient term and the answer gradient term according to a preset gradient constraint rule to form the target gradient information.

[0013] Secondly, embodiments of this application provide a model parameter determination device, the device comprising: a thinking and answer generation module, used to process input data using a specified model to generate at least one thinking process information and its corresponding answer information, wherein at least some of the thinking process information corresponds to multiple answer information; an optimization feedback generation module, used to evaluate the input data based on at least some of the answer information to determine corresponding optimization feedback information; and a parameter update module, used to update at least one parameter of the specified model according to the optimization feedback information; wherein the specified model is a visual language model or a large language model, and the input data includes at least one of text data, image data, and point cloud data.

[0014] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0015] Fourthly, embodiments of this application provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the steps of any of the above methods.

[0016] This application provides a method and apparatus for determining model parameters, a storage medium, and a program product. The method includes: processing input data using a specified model to generate at least one thought process information and its corresponding answer information, wherein at least some of the thought process information corresponds to multiple answer information; evaluating the input data based on at least some of the answer information to determine corresponding optimization feedback information; and updating at least one parameter of the specified model according to the optimization feedback information. The specified model is a visual language model or a large language model, and the input data includes at least one of text data, image data, and point cloud data. When processing the input data, the specified model first generates at least one thought process information (Think), and generates corresponding answer information (Answer) based on each thought process information, wherein at least some of the thought process information corresponds to multiple answer information to achieve multi-answer sampling. Subsequently, evaluation processing is performed based on these answer information to form optimization feedback information. Finally, one or more parameters of the model are updated using the optimization feedback information. Multiple answer sampling can obtain more optimization feedback at the answer level while reducing or maintaining the number of generated answers during the thinking process. This significantly reduces training time and computational overhead, improves training efficiency, and provides a general, stable, and efficient reinforcement learning post-training scheme for various models such as visual language models and large language models. It can enhance the model's thinking and summarizing ability under limited computing resources. Attached Figure Description

[0017] The embodiments of this application are further described below with reference to the accompanying drawings and specific implementation details.

[0018] Figure 1 This is a flowchart illustrating a method for determining model parameters provided in an embodiment of this application.

[0019] Figure 2 This is a structural block diagram of a model parameter determination device provided in an embodiment of this application.

[0020] Figure 3 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the embodiments of this application.

[0022] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0023] With the Deepseek-R1 model revealing that deep thinking capabilities can be naturally derived from basic LLMs through reinforcement learning (more precisely, Group Relative Policy Optimization, GRPO), increasing research has focused on utilizing reinforcement learning as a post-training method to enhance the capabilities of large language models. Inspired by the development of Reasoning Large Language Models (RLLMs) from LLMs through reinforcement learning, researchers have also begun to extend this type of post-training technique from single-language modalities to multimodal tasks to train VLMs.

[0024] However, current research on multimodal tasks largely focuses on finding specific reward functions for particular tasks and designing suitable training schemes accordingly, lacking a more general multimodal task optimization training mechanism. Meanwhile, compared to pure text tasks, multimodal tasks rely more heavily on the model's perception and understanding of image modalities, and on this basis, reasoning and summarizing to arrive at answers. Although relevant VLMs (Visual Models) already possess strong image perception and understanding capabilities, they still lack systematic thinking and summarizing abilities. Applying the GRPO algorithm directly to VLMs can, to some extent, guide the model to possess reasoning and summarizing abilities, but due to the large action space and sparse reward signals of multimodal tasks, it often leads to unstable gradient calculations and slow convergence during training, thus affecting the overall performance improvement of the model.

[0025] To address the aforementioned issues, this application provides a reinforcement learning post-training method for large thinking models based on answer multi-sampling. This method offers a general training scheme for various models, including multimodal models. By introducing a multi-answer sampling mechanism during the generation process, it effectively alleviates the reward sparsity problem and obtains a more stable update signal during gradient calculation, thereby enabling the model to converge faster and exhibit better performance in reasoning and answer generation tasks.

[0026] See Figure 1 , Figure 1 This is a flowchart illustrating a method for determining model parameters provided in an embodiment of this application.

[0027] As mentioned above, when transferring reinforcement learning post-training techniques (such as GRPO) suitable for large-scale pure text models to multimodal models such as visual language models (VLM), the sampling pattern of "one thought process corresponds to one answer" has the following drawbacks due to the characteristics of multimodal tasks, such as a huge answer space and sparse reward signals. First, the evaluation of the thought process is too singular and arbitrary. A high-quality thought may be underestimated due to the bias of a single answer, while a poor thought may be overestimated due to the accidental generation of a correct answer. This high-variance evaluation makes the model gradient calculation extremely unstable. Second, in order to obtain a large number of samples required for stable training, a massive number of thought processes must be generated. However, the tokens (which serve as unit examples for the model to process text, and can be understood as words, characters, or sub-words) for generating thought processes are themselves long, resulting in extremely low model training efficiency, slow convergence, and huge consumption of computational resources.

[0028] In order to improve the relevant technology, this application provides a method for determining model parameters, the method including steps S101 to S103.

[0029] Step S101: Process the input data using a specified model to generate at least one thought process information and its corresponding answer information, wherein at least some of the thought process information corresponds to multiple answer information.

[0030] Step S102: Evaluate the input data based on at least some of the answer information to determine the corresponding optimization feedback information.

[0031] Step S103: Update at least one parameter of the specified model based on the optimization feedback information.

[0032] The specified model is a visual language model or a large language model, and the input data includes at least one of text data, image data, and point cloud data.

[0033] In some embodiments, the specified model is a visual language model, and the input data includes at least two of text data, image data, and point cloud data. In other embodiments, the specified model is a large language model, and the input data includes at least text data.

[0034] The designated model refers to the large model used in this application embodiment to perform thought generation and answer generation, which can be a large language model (LLM) or a visual language model (VLM).

[0035] Input data refers to the information received and processed by the specified model, and can be single-modal or multimodal data. For example, input data may include one or more of text data, image data, and point cloud data. Text data may include, for example, natural language text. Image data may include two-dimensional color images (RGB images) containing only color information, or color-depth images (RGB-D images) containing both color and depth information. Image data is used to provide appearance information such as color, texture, and shape of the scene, and, when a depth channel is available, to provide partial three-dimensional geometric information of objects and the environment. RGB stands for Red, Green, and Blue. D stands for Depth. Point cloud data may include three-dimensional spatial coordinate information for spatial reasoning or three-dimensional perception.

[0036] In the embodied agent scenario, text data in the input data can be obtained, for example, through user input, while image data and point cloud data can be obtained, for example, through the embodied agent's perception of the environment. In this scenario, text data refers, for example, to natural language text used to control the embodied agent to perform tasks, and can be task instructions, prompts, or task scene descriptions. As an example, text data could be: "Grab the red cup on the table and put it on the shelf." Text data can be used to guide the embodied agent in understanding the task objective and operational intent. Image data refers, for example, to visual information acquired by a visual acquisition device carried by the embodied agent, which can include a monocular camera, a binocular camera, a panoramic camera, or an RGB-D camera, etc. Point cloud data refers, for example, to scene spatial geometric information acquired by 3D sensors (such as LiDAR, 3D scanners, binocular cameras, millimeter-wave radar, ultrasonic sensors, etc.). Point cloud data can include the 3D coordinates of multiple spatial points, accurately representing the spatial position, shape, and structure of objects and the environment.

[0037] Think information refers to the intermediate reasoning processes or reasoning chains that a model develops before generating a final answer. This think information can be output as an explicit or implicit sequence. As an example, think information can be wrapped in labels via prompts. <think> and< / think> The information in the thought process is not the direct answer, but rather the logical deduction process from the input data. Through reinforcement learning training, the model is guided to generate text sequences that conform to logical reasoning chains, which assist in answer generation. This improves the model's interpretability and reasoning depth, making the generated answers not only more accurate but also more in line with human thinking habits.

[0038] Answer information, for example, specifies the concrete output derived by the model based on information about the thought process. As an example, prompts can be used to request that the answer information be wrapped in a label.<answer> and< / answer> In between, one thought process can correspond to one answer. Alternatively, one thought process can correspond to multiple answers (answer multi-sampling mechanism). The number of answers corresponding to different thought process information can be the same or different.

[0039] Generating at least one thought process and its corresponding answer means generating one or more thought process pieces of information, and for each thought process, generating its corresponding answer. At least some thought process information corresponds to multiple answer pieces of information means that some or all of the thought process information corresponds to multiple answer pieces of information. Therefore, the number of answer pieces of information is greater than the number of thought process pieces of information.

[0040] As one example, some thought process information corresponds to multiple answer information, while other thought process information corresponds to one answer information. As another example, each thought process information corresponds to multiple answer information. Generating multiple answer information for the same thought process information can increase the richness of the optimized feedback information. For example, a specified model can sample the output multiple times under the same thinking conditions (corresponding to the same thought process information) to obtain different answer information, which can be used for reward evaluation and statistical stability.

[0041] In the relevant algorithm, for each prompt, assuming the model is required to generate K thought process messages (Think) and one answer message (Answer) for each thought process message, then K thought process messages and K answer messages are generated based on each prompt. The Think and Answer generated for each Prompt are in one-to-one correspondence. Here, K is a positive integer.

[0042] Unlike related algorithms, in this application's embodiments, at least some of the thought process information corresponds to multiple answer information (answer multi-sampling mechanism). For example, for each input data, assuming the specified model is required to generate K thought process information pieces, and for each thought process information piece, generate M answer information pieces, then based on each input data, K thought process information pieces and KM answer information pieces are generated. Here, K and M are positive integers, and M is greater than 1.

[0043] Evaluation processing refers to the process of scoring or calculating rewards for multiple answer information. The resulting optimization feedback information may include, for example, the reward value corresponding to the answer information, the value and thinking advantage value corresponding to the thinking process information, and the answer advantage value corresponding to the answer information, etc. The above embodiments are not limited in this regard, and the specific processing process will be illustrated below. Optimization feedback information can be used to update the parameters of a specified model, for example, by using optimization feedback information to correct one or more parameters of the model. For example, model parameter updating may include updating one or more of the following parameters of the specified model: neural network weight parameters, output distribution parameters, policy function parameters, and value function parameters. The above embodiments are not limited in this regard.

[0044] A high-quality thought process should have the potential to derive multiple high-quality answers. Therefore, the above embodiments, by generating multiple answers for part or all of the thought process and using them for evaluation, can obtain more answer information than the amount of thought process information, thereby improving model training efficiency by optimizing the sampling structure. Specifically, through a heterogeneous pattern of multiple-answer sampling (e.g., M>1), i.e., the number of thought processes differs from the number of answers, each thought process is provided with multiple opportunities to generate answers, thus obtaining more answer information than the amount of thought process information. Next, the input data is evaluated based on at least some of the answer information to obtain optimized feedback information for updating model parameters. This significantly improves model training efficiency. While achieving the same performance, the total number of tokens generated can be reduced by generating more answers rather than more thought processes, thus saving considerable computation time and cost. Furthermore, this scheme has broad applicability, not only suitable for training large language models but also providing a general, stable, and efficient training framework for fine-tuning reinforcement learning in multimodal models. Compared with related GRPO methods, it can achieve better model performance under the same resource conditions, while shortening the training time, improving the model's reasoning and answer generation capabilities in complex multimodal tasks, and thus improving the overall intelligence level.

[0045] When fine-tuning large models (especially visual-language models) based on reinforcement learning, a core step is to accurately evaluate the rewards of the answers generated by the model, using this as a signal to drive model optimization. To efficiently, consistently, and automatically evaluate the quality of a rapidly increasing number of answers (e.g., K*M), in some embodiments, the optimization feedback information may include reward values ​​corresponding to at least some of the answer information. The evaluation processing of the input data based on at least some answer information to determine the corresponding optimization feedback information may include: performing reward evaluation on at least some of the answer information based on reference answer information and preset reward evaluation rules to obtain corresponding reward values.

[0046] The reward value can be a scalar numerical value used to directly measure the quality of a single answer generated by a specified model. The reward value of an answer is determined jointly by a preset reward evaluation rule and the reference answer information. As an example, in a visual question-answering task, a preset reward evaluation rule can be specified as follows: Assume the input image data contains a "table" and a "cup," and the reference answer information is "table and cup." When the answer generated by the specified model is completely consistent with the reference answer information (i.e., both "table" and "cup" are recognized), the corresponding reward value is +1; when only one target is recognized (e.g., only "table" or only "cup"), the corresponding reward value is +0.5; when the recognition result is inconsistent with the reference answer information or completely wrong (e.g., the answer is "chair" or "book"), the corresponding reward value is 0. In practical applications, the reward value can be used as raw data input for calculating subsequent indicators (such as thinking value, thinking advantage value, answer advantage value, etc.).

[0047] As mentioned above, reference answer information can refer to a preset standard answer or a set of answer features used as an evaluation benchmark. For example, reference answer information can be a single accurate answer (such as a closed-ended question), a set of acceptable answers, or the key elements an answer should contain (such as an open-ended question). Reference answer information provides an objective standard for reward evaluation and is crucial for achieving automated evaluation and reducing human subjective bias. In practical applications, its form can depend on the specific task. For example, for point cloud data recognition tasks, reference answer information can include standard object category labels. Reference answer information can serve as the standard upon which preset reward evaluation rules are based. The answer information generated by the specified model is compared with the reference answer information according to the rules to output a reward value.

[0048] Preset reward evaluation rules are, for example, a predefined, programmable automated logic or function used to compare answer information with reference answer information and output a reward value. Preset reward evaluation rules define how rewards are quantified; their implementation can be simple (e.g., +1 reward for a complete string match, 0 otherwise) or complex (e.g., using another neural network to calculate semantic similarity as the reward value). Preset reward evaluation rules transform subjective quality judgments into objective, computable indicators. As an example, preset reward evaluation rules can be represented in the form of rule functions. This method can call the rule function, passing in a single answer and reference answer, and receiving the returned reward value.

[0049] Performing reward evaluation separately refers to the process of independently determining the reward for at least a portion of the answer information (e.g., some or all of KM answer information). This evaluation can be performed in parallel or sequentially. The evaluation of each answer information is independent of the results of other answer information, allowing for highly parallel computation and significantly improving processing efficiency to accommodate the evaluation needs of a large number of answers (e.g., K*M). In practical applications, this method can iterate through each answer information in a loop or in a vectorized manner, calling the same preset reward evaluation rule to generate a set of reward values.

[0050] The above embodiments design an efficient and objective reward evaluation mechanism that can be integrated into automated training processes. Based on the task's objectives, it can quickly and in batches assess the quality of a large amount of answer information, thus providing stable and reliable raw data input for subsequent advantage value calculations. By standardizing and automating the reward evaluation process through preset reward evaluation rules, consistency and objectivity of the evaluation are ensured, reducing human bias. Reference answer information provides an objective benchmark or standard for evaluation, making automated evaluation possible. Therefore, automated and batch processing of reward evaluation is achieved, greatly improving training efficiency and enabling the multi-answer sampling framework to operate efficiently. Furthermore, evaluation through preset, unified rules ensures the consistency of reward signals (e.g., reward values), thereby providing the model with stable and reliable learning signals and improving the reliability of training effects and final model performance.

[0051] In the relevant GRPO algorithm, the advantage value of the thinking process is directly determined by the reward of a single answer. This "one-to-one" mapping relationship is extremely unstable in multimodal tasks due to the sparsity and high variance of the reward, resulting in huge noise in the evaluation of the thinking process and making it difficult for the model training to converge.

[0052] To reduce evaluation noise in the thinking process and facilitate model training convergence, in some embodiments, the optimization feedback information may further include thinking advantage values ​​corresponding to at least some of the thinking process information. The step of evaluating the input data based on at least some of the answer information to determine the corresponding optimization feedback information may further include: evaluating the value of at least some of the thinking process information based on the reward values ​​of its corresponding partial or complete answer information to obtain the corresponding value; and performing advantage analysis on at least some of the thinking process information according to the at least some value to determine the corresponding thinking advantage value.

[0053] The thinking advantage value refers to the advantage value corresponding to the thinking process information (Think). The thinking advantage value can be a quantifiable value used to measure whether a specific thinking process is "good" or "bad" relative to the average performance of all thinking processes in this sample, and the degree of good or bad. For example, the calculation method of the thinking advantage value can follow the concept of the advantage function in reinforcement learning. As an example, in gradient calculation, the log probability of the thinking tokens can be multiplied by the thinking advantage value; a positive thinking advantage value increases the probability of generating these thinking tokens, and vice versa.

[0054] Value assessment, for example, is the step of assigning a representative quality score (as the value of the thinking process) to the thinking process, representing a robust estimate of the quality of that thinking process. The above embodiment does not employ the unstable assessment method of "one thought, one answer" in GRPO, but instead uses statistical analysis. Specifically, the value of a thinking process can be determined by the reward value of some or all of the answer information it generates. For example, if a thinking process corresponds to M answer information, its value can be equal to the arithmetic mean of the reward values ​​of the M answers.

[0055] Strengths analysis can be a data processing procedure that transforms the absolute scores of thinking value into relative scores (i.e., thinking strength values). As an example, this method can call relevant statistical functions to calculate the mean and standard deviation of the values ​​of all thinking processes, and then iterate through each value to perform relative strengths analysis.

[0056] The above embodiment replaces the method in related approaches that determines the value of thinking with a single answer through a value assessment step. It innovatively uses value assessment based on multiple answer reward values ​​to calculate the value corresponding to the thinking process information, thereby significantly reducing the volatility of value estimation from the source. Subsequently, through a dominance analysis step, the absolute value score is transformed into a relative dominance score in the entire batch of samples, ensuring the stability and comparability of the thinking dominance value. Finally, the resulting robust thinking dominance value is used to guide the update of model parameters, enabling the specified model to learn how to generate high-quality thinking processes more stably and efficiently, improving training efficiency and achieving better model performance.

[0057] In order to efficiently, objectively and reproducibly aggregate the reward values ​​scattered across multiple answer information to form a single value indicator that can represent the overall quality of the thinking process, in some embodiments, the value assessment process may include: substituting the reward values ​​of some or all of the answer information corresponding to the thinking process information into a preset value calculation formula to calculate the corresponding value.

[0058] As one example, the reward value of a portion of the answer information corresponding to the thought process information is substituted into a preset value calculation formula to calculate the corresponding value. As another example, the reward value of all the answer information corresponding to the thought process information is substituted into the preset value calculation formula to calculate the corresponding value. In practical applications, the reward value of all answer information can be used to maximize the stability effect. However, in some embodiments, obviously abnormal reward values ​​(such as extremely large or small values) can be excluded before calculation; that is, the reward value of a portion of the answer information is used for calculation. This design should also fall within the protection scope of the embodiments of this application.

[0059] A predefined value calculation formula is, for example, a pre-defined, deterministic mathematical function used to aggregate a set of reward values ​​into a scalar representing the overall level of its parent (thinking process). This predefined value calculation formula can be predetermined, determined before training begins, ensuring the consistency and reproducibility of the evaluation process. As an example, the predefined value calculation formula could be an arithmetic mean formula. The arithmetic mean formula is an excellent estimate of expected reward in the sense of minimizing squared error. Its working principle is to smooth out the random noise of individual answer information by averaging (e.g., a bad thought might accidentally get a good answer, or a good thought might be penalized due to a minor error), thereby significantly reducing the variance of the value estimate (e.g., using the arithmetic mean formula, the variance can theoretically be reduced to 1 / M of the original), objectively reflecting the true quality trend of the thinking process.

[0060] The above embodiments standardize and automate the reward value aggregation process by introducing a pre-defined and definite mathematical calculation formula. This design achieves complete automation and standardization of value assessment, ensuring the efficiency, objectivity, and reproducibility of the assessment process, which aligns with the needs of reinforcement learning training processes. Secondly, by effectively filtering noise through statistical principles, the variance of value estimation is reduced, providing a stable and reliable data foundation and improving the problem of unstable training gradients, thus guaranteeing the model's rapid and stable convergence.

[0061] If there is no unified benchmark for value scores across different training batches, they are difficult to use directly to guide model optimization. A value score alone is not very informative, and if the absolute value of the value is used directly to calculate the gradient, the model may be misled, potentially only learning to distinguish between easy and difficult problems (scoring generally higher on easy problems) instead of learning to generate relatively better thought processes for similar problems.

[0062] To establish dynamic, batch-to-batch relative evaluation criteria, in some embodiments, the advantage analysis process may include: performing numerical statistical analysis based on at least a portion of the value to obtain value statistics; and substituting the value corresponding to the thinking process information and the value statistics into a preset thinking advantage calculation formula to calculate the corresponding thinking advantage value.

[0063] In the above embodiments, advantage analysis is, for example, a data processing procedure that transforms the absolute value of thinking process information into a standardized value that characterizes its relative superiority or inferiority within the current batch. The aim is to reduce the differences in numerical distribution between different training batches and generate stable, comparable training signals. This process is a step in the gradient update of reinforcement learning policies, transforming an absolute score (value) without clear physical meaning into a statistically significant relative indicator (thinking advantage value) that guides the direction of model parameter updates. As an example, relevant statistical functions can be called to perform numerical statistical analysis on all values ​​within a batch. Then, each thinking value is iterated over, and it, along with the value statistics obtained in the previous step, is substituted into a preset formula (i.e., a preset thinking advantage calculation formula) for calculation. The entire process can be automated and batch-processed.

[0064] Numerical statistical analysis, for example, refers to the process of statistically calculating the value set corresponding to all thinking process information in the current batch. The purpose is to characterize the overall distribution of the value in the current batch and establish a standard for subsequent relative evaluation. As an example, basic descriptive statistical methods can be used to calculate the sample mean and sample standard deviation of this value set as value statistics. The mean represents the average performance level of the thinking value in this batch, while the standard deviation represents the dispersion of these values ​​around the mean.

[0065] Value statistics can be the output of numerical statistical analysis steps. These data are dynamic and batch-based, meaning that these statistics are recalculated for different input data or different training epochs of the same data, thus ensuring the real-time nature of the evaluation criteria. As an example, value statistics can include the mean and standard deviation.

[0066] The predefined formula for calculating thinking advantage is, for example, a predefined, definite mathematical function. As an example, the predefined formula for calculating thinking advantage can take the value corresponding to a thinking process and the aforementioned value statistics as input, and output its standardized thinking advantage value. Because this formula can be pre-set, the consistency and reproducibility of the calculation are guaranteed. As an example, the predefined formula for calculating thinking advantage can be constructed based on the Z-score (also known as the standard score) in statistics, and the predefined formula for calculating thinking advantage can be expressed as follows.

[0067] AdvTi =[TV] i ―Mean(TV)] / STD(TV)

[0068] Among them, T i This represents the i-th thought process information, where i is a positive integer. AdvT i For T i The corresponding thinking advantage value, TV i For T i The corresponding value. Value statistics include the value mean (Mean, TV) and the value standard deviation (STD, TV). "TV" i "Mean(TV)" represents the centralization process, and calculates the information T of this thinking process. i The value of AdvT is the absolute difference between the think process and the batch average. A positive value indicates better than average, and a negative value indicates worse than average. " / STD(TV)" represents the scaling process, dividing the absolute difference by the batch standard deviation, thus expressing the think advantage value as "how many standard deviations it deviates from the average," making the data from different batches comparable. The resulting think advantage value can be a dimensionless relative score, clearly indicating the relative ranking and quality of the corresponding think process among all candidate thinks. This value can be directly used for subsequent gradient calculations. For example, when calculating the policy gradient, after generating the log probabilities of all tokens involved in the think process, it can be multiplied by this think advantage value. A positive AdvT... i This will increase the probability of generating these tokens, a negative AdvT i This will reduce its probability.

[0069] The above embodiments dynamically obtain the evaluation benchmark (e.g., mean and standard deviation) for the current batch through numerical statistical analysis, and then transform the absolute value of each thought into a relative thought advantage value through a pre-defined thinking advantage calculation formula (e.g., Z-score standardization formula). This design transforms the value score from an absolute indicator affected by the batch to a relative indicator comparable across batches, stabilizing the estimation of the policy gradient because the update signal no longer depends on the absolute magnitude of the value, but only on its relative ranking advantage within the batch. This effectively improves the model's ability to avoid being misled by fluctuations in problem difficulty, ensuring that the training process converges stably and efficiently towards the goal of "always generating thoughts that are better than average," thereby improving the instability of training results. In practical applications, a positive thought advantage value can explicitly tell the model to increase the probability of generating that thought process, while a negative value explicitly indicates that the process should be suppressed. This relative evaluation based on intra-batch competition can more efficiently guide the model to eliminate inferior thinking patterns and retain and strengthen superior thinking patterns. Furthermore, the above embodiments abstract the calculation of the advantage value into an automated process based on statistical analysis and preset formulas, reducing manual intervention and meeting the needs of large-scale reinforcement learning training.

[0070] In multimodal tasks, the answer space is vast and the reward signal is sparse. A high-quality thought might be underestimated due to a single, accidental error in the answer, while a poor thought might be overestimated due to an accidental correct answer. This introduces extremely high variance, causing unstable gradient calculations, slow convergence, and requiring the model to generate numerous lengthy thought processes during training, further increasing computational overhead. Therefore, this application's embodiments are no longer limited to a single answer but enrich the reward signal through a "multi-answer sampling" mechanism and combine statistical methods to aggregate and standardize the reward results, thereby obtaining more stable optimization feedback. In some embodiments, as described above, the overall value and relative advantage of the thought process can be calculated by aggregating the reward values ​​of multiple answers, thereby reducing the high variance evaluation at the thought level. In other embodiments, the answer layer can be directly targeted by statistical analysis and standardization of the reward values ​​of multiple answers to obtain the relative quality advantage value of each answer, providing more refined guidance for model parameter updates. In still other embodiments, the advantage analysis results of the thought layer and the answer layer can be combined to provide a more comprehensive guide for model parameter updates.

[0071] In some embodiments, the optimization feedback information may further include answer advantage values ​​corresponding to at least some of the answer information. The step of evaluating the input data based on at least some of the answer information to determine the corresponding optimization feedback information may further include: performing numerical statistical analysis based on at least some of the reward values ​​to obtain reward value statistics; and substituting the reward values ​​corresponding to the answer information and the reward value statistics into a preset answer advantage calculation formula to calculate the corresponding answer advantage value.

[0072] In the above embodiments, the answer advantage value is used to characterize the relative superiority or inferiority of a single answer compared to other answers in the same batch. That is, the answer advantage value is not a single reward score, but a normalized relative indicator used to distinguish between "good answers" and "bad answers," guiding the model to tend to generate answers with high advantage values ​​during updates. By processing the reward value of a single answer with statistical data such as batch mean and variance, and calculating its relative advantage (e.g., based on the Z-score normalization formula), the absolute value bias caused by differences in task difficulty across different batches can be reduced, ensuring comparability across batches.

[0073] Reward value statistics refer to data obtained by performing a comprehensive statistical analysis on the reward values ​​of all answers (including reward values ​​corresponding to answer information with different thought processes) within a training batch (e.g., corresponding to the same input data). As an example, reward value statistics may include the mean and standard deviation of the reward value set, or other statistical characteristics. Reward value statistics are used to establish a dynamic benchmark, transforming different answer reward values ​​into relative indicators.

[0074] The preset answer advantage calculation formula is, for example, a standardized formula used to calculate the answer advantage value. As an example, the preset answer advantage calculation formula can be a pre-defined function to ensure consistency and reproducibility of the calculation process. For instance, the preset answer advantage calculation formula could receive the reward value and reward value statistics for a single answer, and output a relative advantage value as the answer advantage value. The preset answer advantage calculation formula can be constructed based on the Z-score standardized formula, which will not be elaborated further here.

[0075] In the above embodiment, statistical analysis is first performed based on the answer reward value to obtain reward value statistics. Then, the reward value of a single answer and the reward value statistics are substituted into a preset answer advantage calculation formula to calculate the answer advantage value corresponding to that answer. This design allows for optimization based on the relative performance of the answer in the current batch during model parameter updates, rather than relying solely on absolute reward scores. This results in a relatively stable and comparable optimization signal during model training, significantly reducing the impact of reward sparsity and high variance, ensuring the stability of gradient updates and the convergence speed of training, and reducing the volatility of training gradients. Furthermore, since the advantage analysis at the answer level is based on relative ranking within a batch rather than absolute values, the model can obtain a consistent optimization direction across tasks of varying difficulty, thereby improving convergence speed and training efficiency. Ultimately, this leads to more stable performance improvements across various types of tasks, including multimodal tasks, enabling the model to generate more reliable thought processes and higher-quality answer outputs.

[0076] In reinforcement learning training of multimodal large models, existing methods struggle to fully utilize the confidence distribution information output by the model itself, resulting in insufficiently refined and stable gradient update signals, which in turn affects convergence speed and final performance. Therefore, embodiments of this application can introduce confidence distribution information into the optimization feedback, enabling gradient updates to rely not only on external evaluation signals (reward value, advantage value) but also on the confidence generated internally by the model, thus forming a dual-constraint optimization signal.

[0077] In some embodiments, the specified model can also be used to output the confidence distribution information of at least some thought process information and the confidence distribution information of at least some answer information. The optimization feedback information may include the thinking advantage value corresponding to at least some thought process information and the answer advantage value corresponding to at least some answer information. Updating at least one parameter of the specified model based on the optimization feedback information may include: performing gradient calculation based on the thinking advantage value and confidence distribution information corresponding to at least some thought process information, and the answer advantage value and confidence distribution information corresponding to at least some answer information, to obtain target gradient information for parameter updating; and updating at least one parameter of the specified model based on the target gradient information.

[0078] In the above embodiments, the specified model outputs not only thought process information and answer information, but also thought confidence distribution information and answer confidence distribution information. Optimization feedback information may include thought advantage values ​​and answer advantage values. During parameter updates, gradient calculation is performed based on the thought advantage value and thought confidence distribution information, and the answer advantage value and answer confidence distribution information, to obtain target gradient information, and the model parameters are updated accordingly. Therefore, the above embodiments achieve dual constraints from external reward signals and internal confidence signals, making the direction of parameter updates more accurate and the gradient estimation more stable. This enhances the model's ability to generate high-quality thought processes and answers in multimodal tasks, thereby achieving better performance with the same computing resources.

[0079] During reinforcement learning training of a multimodal large model, even with the introduction of more reward signals and reduction of variance through multi-answer sampling, the model is still susceptible to instability driven by a single advantage value when actually updating parameters. This is because the advantage value only reflects the relative merit of a single thought or answer in the current batch, failing to incorporate the confidence distribution information of the model when generating that thought or answer. If the gradient update is entirely dependent on the advantage value, the model may over-amplify low-confidence, accidentally correct results, or over-penalize high-confidence but slightly biased results, leading to a shift in optimization direction, training oscillations, or even slow convergence. Therefore, in the gradient calculation stage of this application, the advantage value of the answer can be combined with the confidence distribution of the answer only at the answer layer to form the answer gradient term. Alternatively, the advantage value of the thought can be combined with the confidence distribution of the thought only at the thought layer to form the thought gradient term. Alternatively, the advantage value and confidence distribution can be combined simultaneously at both the thought layer and the answer layer, and a preset gradient constraint rule can be introduced to unify the two into a target gradient for overall model parameter updates.

[0080] In some embodiments, the step of performing gradient calculation based on the thinking advantage value and thinking confidence distribution information corresponding to at least some thinking process information, and the answer advantage value and answer confidence distribution information corresponding to at least some answer information, to obtain target gradient information for parameter updating may include: combining the thinking advantage value and thinking confidence distribution information corresponding to at least some thinking process information to obtain a thinking gradient term related to the thinking process information; combining the answer advantage value and answer confidence distribution information corresponding to at least some answer information to obtain an answer gradient term related to the answer information; and constraining the thinking gradient term and the answer gradient term according to a preset gradient constraint rule to form the target gradient information.

[0081] The confidence distribution information of thinking processes refers to the probability distribution information output by the model when generating information about the thinking process. This probability distribution information reflects the model's confidence in the current generated result and can be used as a weighting factor in gradient calculation, determining the model's reliance on its high-confidence outputs. Similarly, the confidence distribution information of answers refers to the probability distribution information output by the model when generating answer information.

[0082] The thinking gradient term refers to the gradient component calculated by combining the thinking advantage value and the thinking confidence distribution information. When the thinking advantage value is positive, the generation probability of the corresponding token is increased; conversely, when the thinking advantage value is negative, its generation probability is decreased. As an example, the thinking gradient term is formed by multiplying the thinking advantage value with its corresponding log probability, serving as the policy gradient signal, and is used to correct the model parameters.

[0083] The answer gradient term refers to the gradient component calculated by combining the answer advantage value and the answer confidence distribution information. When the answer advantage value is positive, the generation probability of the corresponding token is increased; conversely, when the answer advantage value is negative, its generation probability is decreased. As an example, the answer gradient term is formed by multiplying the answer advantage value with its corresponding log probability, serving as the policy gradient signal, and is used to adjust the model parameters.

[0084] Preset gradient constraint rules refer to rules that constrain the magnitude and direction of gradients. For example, they can include the pruning mechanism in the Partial Reinforcement Optimizer (PPO) algorithm, which can improve the problem of excessive updates and ensure the stability of the training process. As an example, preset gradient constraint rules can set a range for gradient ratio changes, such as limiting it to a specified numerical range. By minimizing the difference in the objective function before and after pruning, it ensures that parameter updates do not deviate from a reasonable range.

[0085] Target gradient information refers to the overall update signal obtained by combining the thought gradient term and the answer gradient term and processing them through preset gradient constraint rules. Target gradient information can be used as input to the optimizer to ensure that the parameter update direction both conforms to the dominant value guidance and reduces over-adjustment.

[0086] In the above embodiments, gradient calculation not only incorporates the advantage value derived from external rewards but also introduces the confidence distribution information within the model. By constructing thought gradient terms and answer gradient terms and applying preset gradient constraint rules, the target gradient information is obtained. This design makes the model training signal more stable and the update amplitude controllable, significantly reducing the training instability problem caused by high gradient variance, accelerating the model convergence speed, and improving the inference and generation quality of the model in multimodal tasks. Specifically, through this design, model parameter updates no longer rely on a single signal but on the dual constraints of advantage value and confidence, ensuring a more reasonable update direction and more controllable amplitude. By introducing a joint gradient calculation method of advantage value and confidence, gradient fluctuations during training are significantly reduced, erroneous updates caused by individual abnormal rewards are reduced, and the reasonable weight of high-confidence results in training is guaranteed. Compared with related methods, this scheme not only improves the stability and convergence speed of training but also enables the model to generate more reliable thought processes and answers in complex multimodal tasks, thereby comprehensively improving the model's inference quality and application performance.

[0087] In a specific application scenario, this application embodiment also provides a method for determining model parameters, including the following steps.

[0088] S1: Thinking and Answer Generation. For example, the data loader retrieves the question and actual answer information (as an example of reference answer information), then edits the question into prompts, and inputs the prompts and image data into the VLM (as an example of a specified model). The VLM first generates K thought process messages (generation stops after this point, controlled by the Stop_word parameter); then, for each thought process message, it generates M answer messages (generation stops after this point, controlled by the Stop_word parameter). Finally, K thought process messages are obtained, denoted as T1, T2...T K ; and KM answer information, denoted as A 1,1 A 1,2 A 1,M A 2,1 A 2,2 A 2,M …A K,M A i,1 A i,2 …A i,M Information T for the thinking process i The corresponding M answer information generated.

[0089] S2: Reward Evaluation and Advantage Calculation. For each answer, a preset reward evaluation rule can be used to evaluate it, resulting in KM reward values, denoted as R. 1,1 ,R 1,2 ..R1,M ,R 2,1 …R 2,M …R K,M Then, calculate the value corresponding to each piece of information in the thought process. This can be done using a pre-defined value calculation formula, for example, let's say... TV i Information T for the thinking process i The value of information can be, for example, equal to the average reward value of the M corresponding answer choices for that thought process information. Then, a preset thinking advantage calculation formula is used to calculate the thinking advantage value (AdvT) for each piece of information in the thought process. i =[TV] i ―Mean(TV)] / STD(TV), where AdvT i Information T for the thinking process i The corresponding thinking advantage values, Mean(TV) and STD(TV), represent the mean and standard deviation of the value corresponding to the K thinking process information, respectively, and can be obtained through statistical calculations. Finally, the answer advantage value, AdvA, is calculated for each answer information. i,j =[R i,j ―Mean(R)] / STD(R), where AdvA i,j For answer information R i,j The corresponding answer advantage values, Mean(R) and STD(R), represent the average and standard deviation of the reward values ​​corresponding to KM answer information, respectively, and can be obtained through statistical calculations.

[0090] S3: Gradient calculation and backpropagation. Next, we will consider the advantage value AdvT. {i} And the answer advantage value AdvA {i,j} The target gradient information is obtained by multiplying the log probabilities of the thought process information and the answer information by the log probability ratios corresponding to those information. This differs from GRPO, where the log probabilities of the thought process information and the answer information are multiplied by a uniform advantage value (in the GRPO algorithm, the thought process information and the answer information are in a one-to-one correspondence, and their corresponding advantage values ​​are the same). Finally, this target gradient information is pruned using the PPO method, as shown in the following formula.

[0091]

[0092] in, To optimize the objective function. This is the expectation operator. q~P(Q) refers to sampling a problem q from the distribution P(Q) of a problem dataset Q. π represents the specified model, and θ represents the parameters of the specified model. θ For the updated model strategy, This represents the model strategy before the update. `thi` represents the thought process information. This refers to the model strategy before the update. The middle sample generates a thought process information thi, where, It is a conditional probability distribution, representing the probability distribution of a model based on a policy given a problem q. The probability distribution for generating thought process information. The symbol (|) is the condition symbol, indicating that the generation of thought process information depends on the question q. || is the length operator. clip(...,1-ε,1+ε) is the clipping function, used to restrict the value of the first parameter within the parentheses to the interval [1-ε,1+ε], where ε is a constant, for example, 0.2.

[0093] The above algorithm is called the GRPO-MA algorithm to distinguish it from related GRPO algorithms. Here, M represents the number of answer options corresponding to each thought process, and A represents the Answer. When M=1, GRPO-MA can be considered a degenerate into GRPO. In policy gradient methods, the advantage value determines whether the probability of generating a corresponding token (e.g., a token) increases or decreases, and by what magnitude. In GRPO, the advantage value corresponding to a thought process is determined by only one answer derived from that thought process. The probabilistic nature of LLM means that this single determinant leads to a very large variance in the advantage value. In GRPO-MA, however, the advantage value corresponding to a thought process is determined based on the reward values ​​corresponding to multiple answer options, reducing the variance of the advantage value. Furthermore, GRPO-MA's multi-sampled answers enrich the optimization feedback information, and the calculation of the answer advantage value is more robust.

[0094] Therefore, GRPO-MA has several advantages over GRPO. First, when generating the same amount of thought process information, GRPO-MA generates more answer information, achieving better performance with fewer samples or the same number of samples, and because the word length of the answer information is shorter, it adds less to the training time. Second, when generating the same number of answer information, GRPO-MA does not compromise the model's performance; in fact, it is more stable than GRPO and generates fewer thought processes, thus significantly reducing training time. Furthermore, like GRPO, GRPO-MA is suitable for various tasks with verifiable rewards, especially visual tasks requiring perception, and therefore can be applied to multimodal tasks.

[0095] In a specific application scenario, the following table shows a comparison of training time between GRPO-MA and GRPO. The test card is, for example, a 4-card H100, but this embodiment of the application does not limit this.

[0096]

[0097] As can be seen, GRPO-MA introduces a heterogeneous sampling training process with inconsistent numbers of thought processes and answers. This heterogeneous sampling improves the stability of training gradients and model performance. For example, when GRPO-MA and GRPO each generate 4 thought process pieces, GRPO generates 4 answer pieces and takes 1 day to train; GRPO-MA (assuming M is 4) generates 16 answer pieces and takes 1.2 days to train. Although the training time of GRPO-MA is longer than that of GRPO, the increase is not significant, while the number of answers obtained is greatly increased. For another example, when GRPO-MA and GRPO each generate 16 answer pieces, GRPO generates 16 thought process pieces and takes 4 days to train; while GRPO-MA, generating 4 thought process pieces with 4 answer pieces per thought process, takes 1.2 days to train; and generating 8 thought process pieces with 2 answer pieces per thought process, takes 2.1 days to train. Clearly, when generating the same number of answer pieces, GRPO-MA requires significantly less training time.

[0098] In addition to the aforementioned beneficial effects, GRPO-MA can also reduce the variance of thinking advantage value and answer advantage value, which will be explained separately below.

[0099] First, we analyze the variance changes of the dominance value after sampling from multiple answers.

[0100] Because {A i,j} 1≤j≤M Sampled from the same distribution π(·|T) i (Prompt), assuming the thought process information T i The reward value for the answer information {R} i,j} 1≤j≤M Follows a mean Standard deviation is The distribution of [the value of thought]. Assume the value function of thought (as an example of a presupposed value calculation formula) is [formula / method]. Therefore, the corresponding variance is: The average value is: At this point, TV can be considered as i Follows a mean Standard deviation is The distribution of i, i = 1, 2, ..., k.

[0101] Based on the pre-defined formula for calculating thinking advantage, a new random variable can be designed as the thinking advantage value, i.e. From K random variables {TV i}1≤i≤K It can form K new random variables {AdvT} i} 1≤i≤K What we need to analyze are these K random variables {AdvT} i} 1≤i≤K The variance of the thinking advantage value is the variance of the thinking advantage value.

[0102] Let AdvT i =f i (TV1,TV2,..TV K ), AdvT i Considered The function, and in The first-order Taylor expansion is as follows.

[0103]

[0104] In the above formula, the symbol This is a vector differential operator used to perform partial derivative operations on a function f, the result of which is the gradient vector. The symbol Σ represents a vector of random variables. The covariance matrix is ​​defined as follows:

[0105] Next, we calculate the partial derivatives. For convenience, let δ... ik It represents an index function that is 1 when i = k and 0 otherwise.

[0106] Design a random variable as the average value of the information corresponding to the thinking process. Design a random variable as the standard deviation of the value corresponding to the information in the thinking process. The following expression can be obtained.

[0107]

[0108]

[0109] This leads to the following expression.

[0110]

[0111] Next, in The gradient is calculated as follows.

[0112]

[0113] If TV is considered i If the values ​​are independent of each other, then the covariance matrix is ​​a diagonal matrix, and the following expression can be derived for the variance of the advantage value.

[0114]

[0115] It can be easily observed that M can be considered as part of the denominator, and the variance of the advantage value Var[AdvT] can be considered. k It decreases rapidly as M increases.

[0116] Next, we will analyze and calculate the answer advantage value (AdvA) for each individual answer. i,j The variance. To maintain consistency with previous notation, R0 is used. i,j Representing answer information A i,j The reward value (i.e., Reward), AdvR i,j It can be considered equivalent to AdvA i,j Assume the total number of samples is N = K × M.

[0117] Assume the answer advantage value is calculated using the following pre-defined answer advantage calculation formula:

[0118] AdvR i,j Consider all reward values ​​R k,m The function, denoted as g i,j (R), where the reward matrix R is a matrix containing all R. i,j The K×M matrix.

[0119] Using the multivariate Delta method (a method in statistics used to estimate the form of a function), by measuring the mean at μ... R For function g i,j We approximate its variance using a first-order Taylor expansion:

[0120] Where: μ R It is the expected value matrix of the reward matrix R, where all elements in the k-th row are equal to 1. It is the gradient vector of the function at the mean point, and ∑ is the gradient vector of all reward values ​​R. k,m The covariance matrix of the vectors formed.

[0121] To calculate the gradient First, clarify the function g i,j The specific form. Order as well as but

[0122] Next, it is necessary to calculate its value for any variable R. k,m partial derivatives

[0123] As an example, the derivative rule for the quotient is shown below.

[0124] molecular The partial derivatives are: (where δ) ik δ jm It is a double Kronecker function, δ only when k = i and m = j ik δ jm (1).

[0125] denominator The partial derivatives are:

[0126] Based on the above results, the (k,m)th component of the gradient is obtained:

[0127] The above expression can be rewritten as:

[0128] To reach the mean point μ R Evaluate the gradient at the Taylor expansion point R = μ R At each R k,m by its expected value Replacement.

[0129] At this point, the mean of all elements It becomes: Variance of all elements Transform into the mean of thinking Its own variance: Advantage Value (AdvR) i,j The value at that point is:

[0130] Substituting the above parameters into the partial derivative expression, we obtain the gradient components evaluated at the mean point:

[0131] When determining the covariance matrix ∑, the reward value {R} for all answer information is determined according to the problem setting. k,m All variables are independent random variables. Therefore, their covariance matrix ∑ is a diagonal matrix, where the elements on the diagonal are the variances of the individual random variables. All off-diagonal elements are 0.

[0132] Now, substitute the gradient and covariance matrix into the variance approximation formula. In the case that ∑ is a diagonal matrix, the quadratic form calculation simplifies to the sum of squares of each term, i.e.:

[0133] The variance of the answer's advantage value is calculated as follows:

[0134] It can be observed that increasing K and M contributes equally to the variance of the answer advantage value. Therefore, in practical applications, we can choose to increase the value of M, that is, generate more answer information for each thought process.

[0135] The above analysis examined the impact of increasing the value of M on the variance of the thinking advantage value and the answer advantage value. The conclusion is that increasing the value of M can inversely decrease the variance of the thinking advantage value and the answer advantage value.

[0136] See Figure 2 , Figure 2 This is a structural block diagram of a model parameter determination device provided in an embodiment of this application.

[0137] This application embodiment also provides a model parameter determination device, which includes a thinking and answer generation module, an optimization feedback generation module, and a parameter update module. The thinking and answer generation module processes input data using a specified model to generate at least one thinking process information and its corresponding answer information, wherein at least some of the thinking process information corresponds to multiple answer information. The optimization feedback generation module evaluates the input data based on at least some of the answer information to determine corresponding optimization feedback information. The parameter update module updates at least one parameter of the specified model according to the optimization feedback information. The specified model is a visual language model or a large language model, and the input data includes at least one of text data, image data, and point cloud data.

[0138] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0139] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0140] The computer program product may be a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the computer program product of the embodiments of this application is not limited thereto, and the computer program product may be any combination of one or more computer-readable media.

[0141] See Figure 3 , Figure 3 This is a structural block diagram of a computer device provided in an embodiment of this application.

[0142] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0143] The embodiments of this application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0144] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected through internal connection paths.

[0145] The memory 110 is used to store computer programs, which in some implementations may include code for implementing the methods of the embodiments of this application.

[0146] The processor 120 executes the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information, and output operation results and other data. In some implementations, when the solutions of the embodiments of this application are implemented by software or firmware, the computer program used to implement the solutions of the embodiments of this application can be stored in the processor 120 and executed by the processor 120.

[0147] The memory 110 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes random access memory (RAM), cache memory, and read-only memory (ROM). The memory 110 stores a computer program that can be executed by processor 120, causing processor 120 to implement the steps of any of the methods described above.

[0148] The processor 120 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 120 can be any conventional processor.

[0149] In implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 120 or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor 120. The software modules can be located in mature storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0150] In some implementations, in addition to the hardware units described above, computer devices may also include software modules, such as operating systems, basic input / output systems (BIOS), and application software.

[0151] An operating system is used to manage the hardware and / or software resources of a computer device; it is the kernel and foundation of the computer. The operating system handles fundamental tasks such as managing and configuring memory, determining the priority of system resource allocation, controlling input and output devices, operating the network, and managing the file system. To facilitate user operation, most operating systems provide a user interface for interaction with the system.

[0152] The BIOS is used to perform hardware initialization during the power-on boot phase and to provide runtime services for the operating system and applications. In some implementations, the BIOS can also monitor and display processor temperature and execute temperature protection strategies.

[0153] Application software, also known as an application program, can be understood as software written for a specific user application purpose, and is one of the main categories of computer software. For example, application software can be a program used to achieve purposes such as power control and temperature management.

[0154] It is understood that the specific examples in this application are only intended to help those skilled in the art better understand the implementation of this application, and are not intended to limit the scope of protection of this application.

[0155] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0156] It is understood that the various implementation methods described in this application can be implemented individually or in combination, and this application does not limit them.

[0157] Unless otherwise stated, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0158] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0159] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the embodiments described above can be referred to the corresponding processes in other embodiments, and will not be repeated here.

[0160] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0161] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the technical solution in this application, depending on actual needs.

[0162] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0163] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to related technologies, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for determining model parameters, characterized in that, The method includes: The input data is processed using a specified model to generate at least one thought process information and its corresponding answer information, wherein at least some of the thought process information corresponds to multiple answer information. The input data is evaluated and processed based on at least some of the answer information to determine the corresponding optimization feedback information; Based on the optimization feedback information, update at least one parameter of the specified model; The specified model is a visual language model or a large language model, and the input data includes at least one of text data, image data, and point cloud data.

2. The model parameter determination method according to claim 1, characterized in that, The optimization feedback information includes reward values ​​corresponding to at least some of the answer information. The evaluation processing of the input data based on at least some of the answer information to determine the corresponding optimization feedback information includes: Based on the reference answer information and the preset reward evaluation rules, reward evaluation is performed on at least some of the answer information to obtain the corresponding reward value.

3. The model parameter determination method according to claim 2, characterized in that, The optimized feedback information also includes a thinking advantage value corresponding to at least a portion of the thinking process information. The step of evaluating the input data based on at least a portion of the answer information to determine the corresponding optimized feedback information further includes: For at least some of the thought process information, a value assessment is performed based on the reward value of the corresponding partial or complete answer information to obtain the corresponding value; Based on at least some of the value, advantage analysis is performed on at least some of the thinking process information to determine the corresponding thinking advantage value.

4. The model parameter determination method according to claim 3, characterized in that, The value assessment process includes: substituting the reward value of some or all of the answer information corresponding to the thinking process information into a preset value calculation formula to calculate the corresponding value; The advantage analysis process includes: performing numerical statistical analysis based on at least some of the value to obtain value statistics; and substituting the value corresponding to the thinking process information and the value statistics into a preset thinking advantage calculation formula to calculate the corresponding thinking advantage value.

5. The model parameter determination method according to claim 2, characterized in that, The optimization feedback information also includes answer advantage values ​​corresponding to at least some of the answer information. The step of evaluating the input data based on at least some of the answer information to determine the corresponding optimization feedback information further includes: Numerical statistical analysis is performed on at least a portion of the reward values ​​to derive reward value statistics. The reward value corresponding to the answer information and the statistical data of the reward value are substituted into the preset answer advantage calculation formula to calculate the corresponding answer advantage value.

6. The model parameter determination method according to claim 1, characterized in that, The specified model is also used to output the confidence distribution information of at least some of the thinking process information and the answer confidence distribution information of at least some of the answer information. The optimization feedback information includes the thinking advantage value corresponding to at least some of the thinking process information and the answer advantage value corresponding to at least some of the answer information. Updating at least one parameter of the specified model based on the optimization feedback information includes: Based on the thinking advantage value and thinking confidence distribution information corresponding to at least some of the thinking process information, and the answer advantage value and answer confidence distribution information corresponding to at least some of the answer information, gradient calculation processing is performed to obtain the target gradient information for parameter updating; Based on the target gradient information, at least one parameter of the specified model is updated.

7. The model parameter determination method according to claim 6, characterized in that, The gradient calculation based on the thinking advantage value and thinking confidence distribution information corresponding to at least some thinking process information, and the answer advantage value and answer confidence distribution information corresponding to at least some answer information, to obtain target gradient information for parameter updating, includes: The thinking advantage value and thinking confidence distribution information corresponding to at least some of the thinking process information are combined and calculated to obtain the thinking gradient term related to the thinking process information; The answer advantage value corresponding to at least some of the answer information is combined with the answer confidence distribution information to obtain the answer gradient term related to the answer information; According to the preset gradient constraint rules, the thinking gradient term and the answer gradient term are constrained to form the target gradient information.

8. A model parameter determination device, characterized in that, The device includes: The thinking and answer generation module is used to process input data using a specified model to generate at least one thinking process information and its corresponding answer information, wherein at least some of the thinking process information corresponds to multiple answer information. An optimization feedback generation module is used to evaluate and process the input data based on at least some answer information to determine the corresponding optimization feedback information; The parameter update module is used to update at least one parameter of the specified model based on the optimization feedback information. The specified model is a visual language model or a large language model, and the input data includes at least one of text data, image data, and point cloud data.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images