Multimodal large model task processing method, device and equipment based on reinforcement learning

Abstract

The invention provides a multi-modal large model task processing method, device and equipment based on reinforcement learning, and the method comprises the steps: employing a multi-modal large model, generating G groups of responses for multi-modal visual task data, carrying out the scoring of each group of responses, obtaining a reward value, carrying out the standardization of the reward value, obtaining a dominance score, building a strategy updating gradient through the dominance score, and carrying out the calculation of the strategy updating gradient. A final result is obtained after expectation reward maximization, model parameter adjustment and multi-round iteration, and an expectation reward objective function comprises an expectation calculation item and KL divergence constraint, so that a current strategy can be updated by comparing the performance of candidate strategies through group relative strategy optimization, and local optimum can be helped to be jumped out; the strategy updating amplitude is limited by the KL divergence constraint, and the model is prevented from violently changing in the optimization process, so that the training stability is improved; the dynamic strategy iteration allows the model to adjust the balance between exploration and utilization according to the learning progress on the basis of keeping the stability, thereby further ensuring the effectiveness and stability of strategy optimization.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a multimodal large model task processing method, device and equipment based on reinforcement learning. Background Art

[0002] In recent years, with the innovation of the Transformer architecture and breakthroughs in large-scale pre-training technology, artificial intelligence models have made significant progress in single-modal tasks (such as natural language processing and computer vision). Large-scale language models (LLMs) and visual models (VLMs) represented by GPT-4 and CLIP have demonstrated powerful feature extraction capabilities, contextual reasoning capabilities, and zero-shot generalization capabilities. However, complex tasks in real-world scenarios often require models to understand both visual and semantic information and perform cross-modal reasoning and decision-making, which has driven a wave of research on large multimodal models (LVLMs). By combining the advantages of visual encoders and language models, large multimodal models can handle tasks such as image description, visual question answering, and object detection, but their training and optimization still face many challenges.

[0003] Traditional multimodal models are often trained using end-to-end joint optimization strategies, such as aligning image and text features through contrastive learning (e.g., CLIP) or enabling cross-modal interaction based on generative objectives (e.g., Flamingo). However, as model size grows exponentially (e.g., models with hundreds of billions of parameters), the computational cost and data requirements of end-to-end training rise dramatically. To alleviate this problem, researchers have proposed modular training schemes. For example, BLIP-2 freezes the pre-trained visual encoder and language model and introduces a lightweight query transformer (Q-Former) to achieve cross-modal feature alignment. Q-Former is trained in two stages: the first stage completes the visual-to-text feature mapping, and the second stage uses instruction data to fine-tune the generation capabilities. Although these approaches reduce training overhead, their core modules (e.g., Q-Former) still contain millions of parameters, rely on massive amounts of labeled data for convergence, and perform poorly in few-shot scenarios. This results in insufficient learning and generalization when data is scarce.

[0004] The mainstream optimization method for multimodal models is still supervised fine-tuning (SFT), which directly fits the task objectives through a large amount of labeled data. However, SFT still has the following technical drawbacks:

[0005] 1. In visual tasks such as fine-grained classification and open vocabulary detection, the scarcity of labeled data severely constrains model performance. Furthermore, SFT tends to mechanically memorize labeled answers rather than explore potential solutions through reasoning, resulting in insufficient generalization in complex tasks such as object detection that requires spatial reasoning. For example, in the task of few-shot object detection, the recognition accuracy of traditional SFT models for new or rare categories drops significantly, making them difficult to adapt to the demands of dynamic scenarios.

[0006] 2. Reinforcement Learning (RL) uses reward mechanisms to guide models to autonomously explore optimal strategies, providing new insights for efficient data learning. In the field of language modeling, policy gradient-based methods (such as PPO and GRPO) have been successfully applied to tasks such as mathematical reasoning and code generation. Their core lies in designing verifiable reward functions (such as code execution correctness and mathematical answer matching). However, when migrating RL to multimodal scenarios, the reward function for visual tasks cannot simultaneously measure spatial accuracy and semantic consistency, and traditional binary rewards cannot capture fine-grained performance differences.

[0007] 3. The complexity of multimodal output (such as text-visual joint generation) leads to poor stability of strategy optimization, and the model is prone to falling into local optimality or training divergence.

[0008] Although there have been attempts to introduce reinforcement learning into multimodal alignment (such as reducing hallucinations and improving human preferences), its application is still limited to behavioral correction in the output layer and fails to deeply integrate the core indicators of visual tasks (such as detection accuracy and classification confidence). For example, reinforcement learning based on human feedback (RLHF) relies on manually labeled preference data and requires multiple models with comparable parameter sizes, which is costly and difficult to scale; while most open source solutions (such as DeepSeek-R1) are only designed for plain text tasks and lack support for joint optimization of visual semantics. In addition, existing methods perform poorly in complex scenarios such as multi-round reasoning and cross-modal causal modeling, limiting their practicality in tasks such as open vocabulary detection and fine-grained reasoning.

[0009] In summary, existing large multimodal models have problems such as limited reasoning ability, poor generalization ability of large multimodal models, and traditional supervised fine-tuning requiring a large amount of data. Summary of the Invention

[0010] This application proposes a multimodal large model task processing method, device and equipment based on reinforcement learning, which can solve one of the problems existing in the background technology.

[0011] To achieve the above objectives, this application adopts the following technical solutions:

[0012] In a first aspect, a multimodal large model task processing method based on reinforcement learning is provided, the task processing method comprising:

[0013] Obtaining visual task data, wherein the visual task data includes: visual images and text instructions;

[0014] Generate G groups of responses to the visual task data based on a strategy model, wherein the strategy model adopts a multimodal large model, and the responses include: a structured reasoning process and an answer;

[0015] Using a predefined verifiable reward function and format constraint rules, each group of responses is comprehensively scored to obtain a reward value;

[0016] Normalizing the reward value to obtain a dominance score reflecting the dominance of each group of responses; and

[0017] The advantage score is used to construct a policy gradient update direction. The parameters of the policy model are adjusted by maximizing an expected reward objective function, and the final result is obtained after multiple rounds of iteration. The expected reward objective function includes: an expected calculation term for the distribution of response and visual task data constructed by importance sampling, and a KL divergence constraint.

[0018] Based on the above technical solution, a multimodal large model is used to generate G groups of responses for multimodal visual task data, and each group of responses is scored to obtain a reward value. The advantage score is obtained by standardizing the reward value, and the policy update gradient is constructed with the advantage score. By maximizing the expected reward, the model parameters are adjusted, and the final result is obtained after multiple rounds of iteration. Among them, the expected reward objective function contains the expected calculation term and the KL divergence constraint. In this way, by adopting group relative policy optimization, the current policy can be updated by comparing the performance of candidate strategies, which helps to escape the local optimum; the KL divergence constraint limits the policy update amplitude, prevents the model from drastic changes during the optimization process, and thus improves the stability of training; dynamic policy iteration allows the model to adjust the balance between exploration and utilization according to the learning progress while maintaining stability, further ensuring the effectiveness and stability of policy optimization while retaining the ability to explore unknown solutions.

[0019] In addition, the reinforcement learning method based on policy gradient optimization enables the model to learn through interaction with the environment and sparse reward signals, reducing the dependence on large-scale labeled data.

[0020] In a possible design manner of the first aspect, the visual task is: a detection task or a classification task.

[0021] In a possible design manner of the first aspect, when the visual task is a detection task, the verifiable reward function includes: an intersection-over-union reward item, a confidence reward item and a format constraint item, the intersection-over-union reward item is defined by the average intersection-over-union value of all prediction boxes, and the confidence reward item is defined by the average confidence value of all prediction boxes; when the visual task is a classification task, the verifiable reward function includes: an accuracy reward item and a format constraint item, the accuracy reward item and the format reward item are defined by binary numerical values ​​reflecting the degree of matching, and the format constraint item is used to reward or punish those that meet the format requirements.

[0022] Based on the above technical solution, setting intersection-over-union reward items, confidence reward items, format constraint items, and accuracy reward items, which intersect with traditional binary rewards, can better capture fine-grained performance differences. These reward functions can provide more detailed and precise feedback, thereby more effectively guiding the model to optimize the joint performance of vision and semantics.

[0023] In a possible design of the first aspect, the advantage score is constructed by the following formula:

[0024]

[0025] Among them, A i is the advantage score of the response of group i, R i is the reward value, μ R is the mean reward value, σ R is the standard deviation of the reward value, and ε=1e-6 is used to avoid division by zero errors.

[0026] In a possible design manner of the first aspect, the advantage score is clipped using an extreme value of a range of advantages scores.

[0027] In a possible design manner of the first aspect, the expected reward objective function is constructed by the following formula:

[0028]

[0029] Among them, θ is the parameter of the policy model, L(θ) is the policy optimization target, o is the response, q is the visual task data input to the model, π is the policy model, π old , π θ , π ref is the policy model of the past, current, and initial training, is the expected calculation term, is the importance sampling, and β is the KL divergence penalty coefficient.

[0030] In a possible design manner of the first aspect, the task processing method further includes:

[0031] After each round of iteration, the KL divergence mean KL of the new and old strategies is calculated by the following formula: avg :

[0032]

[0033] Among them, D KL for divergence calculations; and

[0034] Determine whether the KL divergence mean meets the preset conditions. If so, the dynamic adjustment of the KL divergence penalty coefficient is triggered, as shown in the following formula:

[0035]

[0036] Among them, β new is the adjusted KL divergence penalty coefficient, β old is the KL divergence penalty coefficient before adjustment, τ KL is the threshold.

[0037] In a possible design manner of the first aspect, the visual task processing method further includes:

[0038] Using the TOP-K strategy, save the strategy π every K rounds of training θ as π for the next K rounds of training old .

[0039] In a second aspect, a task processing device is provided, the task processing device comprising:

[0040] An input unit, configured to obtain visual task data, wherein the visual task data includes: visual images and text instructions; and

[0041] A processing unit is configured to generate G groups of responses to the visual task data based on a policy model, wherein the policy model adopts a multimodal large model, and the responses include: a structured reasoning process and an answer; a predefined verifiable reward function and format constraint rules are used to comprehensively score each group of responses to obtain a reward value; the reward value is standardized to obtain an advantage score reflecting the advantage of each group of responses; and the advantage score is used to construct a policy gradient update direction, and the parameters of the policy model are adjusted by maximizing an expected reward objective function, and a final result is obtained after multiple rounds of iterations. The expected reward objective function includes: an expected calculation term for the distribution of responses and visual task data constructed by importance sampling, and a KL divergence constraint.

[0042] In a third aspect, an electronic device is provided, comprising: a processor, and a memory coupled to the processor, the memory being used to store a computer program; and the processor being used to execute the computer program stored in the memory, so that the electronic device performs the task processing method as any possible implementation method in the first aspect.

[0043] In a fourth aspect, a computer-readable storage medium is provided, comprising a computer program or instructions, which, when executed on a computer, causes the computer to execute the task processing method of any possible implementation of the first aspect.

[0044] In a fifth aspect, a computer program product is provided, comprising: a computer program or instructions, which, when the computer program or instructions are run on a computer, enables the computer to execute the visual task processing method of any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 This is a flow chart of a multimodal large model efficient and stable pre-training solution based on an improved lightweight query transformer based on instruction intelligence provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0048] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0050] like Figure 1 As shown, this embodiment provides a method for designing verifiable rule-based rewards for multimodal face vision-text tasks, as well as an efficient reinforcement learning method for large multimodal models based on policy gradient optimization. The following method content can be used to illustrate model training and visual task processing methods using trained models.

[0051] This embodiment provides a method for designing verifiable rule rewards for a face vision-text multimodal task, the method comprising the following steps:

[0052] S110 builds a multimodal input dataset and generates multiple sets of responses including reasoning process and final answers through a large visual language model.

[0053] This step specifically includes:

[0054] S111, for target visual tasks (including but not limited to object detection, image classification and open vocabulary recognition), collects raw image data and performs screening and deduplication; eliminates blur, noise and low-resolution samples through image sharpening algorithms to ensure input data quality.

[0055] The specific process involves using specialized equipment to capture images of the target scene and organizing the collected data based on actual needs. Images that do not meet the requirements, such as those with blurred scenes, weak features, or excessive repetition, are discarded. For less-than-clear images, deep learning denoising techniques can be used to improve their quality, thereby optimizing the overall performance of the dataset.

[0056] Secondly, annotation is performed for different tasks using large multimodal models or manual methods. For object detection tasks, the coordinates and categories of objects in the image need to be annotated. For classification tasks, the categories of images need to be annotated. For open vocabulary detection tasks, natural language descriptions of objects in the image can be constructed using caption generation models or manual methods.

[0057] S112, designs standardized prompt templates for different tasks, constructs visual-text multimodal datasets, and uses language as an interface to convert visual tasks into text generation tasks based on visual understanding capabilities and natural language understanding capabilities.

[0058] Specifically, based on the characteristics of the target visual task, a structured prompt template is constructed to force the model to output intermediate steps of the reasoning process and a structured answer. The specific template example is as follows (the characters in {} are replaced with actual images or categories):

[0059] Object detection task: {input_image}, detect all objects belonging to category {category} in the image, and only output the coordinates and confidence scores. If there is no target object, return No Object.

[0060] Image classification task: {input_image}, identify the plant species in the image and output the plant species.

[0061] Open vocabulary recognition task: Given {input_image}, identify objects in the image that match the semantics of {description} and output the object's coordinates. If no semantically matching object is found, return No Object.

[0062] S113: Input the preprocessed image into the large multimodal model (LVLM) and generate multiple sets of candidate responses in combination with the task template.

[0063] Specifically, a large multimodal model is called to perform multiple rounds of independent generation on the same input image, providing support for subsequent group relative policy gradient optimization.

[0064] S114, establish a mapping relationship between the image data and the corresponding multiple groups of responses, and store them in a JSON file according to the "image ID-response group" structure; at the same time, build a metadata index table to record the image size, task type and annotation information required for reward calculation (such as the real bounding box coordinates, category label).

[0065] Assign a unique identifier (image ID) to each image, define a structured storage format based on the task type and dataset source, and associate the original image path, image resolution, task type (such as detection, classification), annotation truth value (such as the true category, bounding box coordinates), and multiple sets of model-generated response data (including the inference process and answers) by image ID in the JSON file. At the same time, create a global metadata index table to independently store the mapping relationship between basic image information and task annotations, and implement data alignment verification and dynamic updates through automated scripts to ensure the consistency of response data and annotation truth values, providing structured input support for subsequent reward calculation and model optimization.

[0066] S120, design verifiable regularized reward functions for different visual-textual multimodal tasks, including intersection-over-union (IoU) rewards for object detection, accuracy rewards for classification tasks, and format constraint rewards.

[0067] Specifically,

[0068] S121, design an Intersection over Union (IoU) reward mechanism for the object detection task: based on the bounding box coordinates and confidence output by the model, match them with the real bounding box in the annotated data, calculate the Intersection over Union (IoU) value of each predicted box, and use the average IoU value as the reward.

[0069] Specifically, the bounding boxes output by the model are sorted in descending order of confidence to generate a prediction box sequence {b1,b2,...,b n}; Compare each predicted box to the unmatched true box Calculate the Intersection over Union (IoU) one by one. If IoU ≥ a preset threshold (such as 0.5), it is considered a valid match. For unmatched prediction boxes (IoU < threshold) and redundant prediction boxes (such as repeatedly detecting the same target and only taking the prediction box with the largest IoU), mark their IoU as 0. Calculate the average IoU value RIoU of all prediction boxes as a reward. The formula is:

[0070]

[0071] in, Indicates that after IoU comparison with the predicted box b i The matching true value box; IoU is an indicator that measures the degree of overlap between the two bounding boxes (BoundingBox) of the predicted box and the true value box, and is widely used in target detection tasks to evaluate the matching accuracy between the predicted box and the true box. Its formula is defined as:

[0072]

[0073] Among them, A represents the predicted box, B represents the real box, the numerator represents the overlapping area of ​​​​A and B, and the denominator represents the total area covered by A and B; IoU∈[0,1], the larger the value, the higher the overlap, the value of 1 represents complete overlap, and the value of 0 represents no overlap between A and B.

[0074] In addition, a confidence reward is given according to the confidence level, and the formula is as follows:

[0075]

[0076] The overall confidence reward is: Among them, τ is the preset threshold.

[0077] S122, design an accuracy reward mechanism for classification tasks: strictly match the category labels output by the model with the true categories of the labeled data. If they are consistent, the accuracy reward is 1, otherwise it is 0.

[0078] Specifically, the predicted category is extracted from the model output and the string is matched exactly with the marked true category (not case-sensitive and not synonymous). If the predicted category is exactly the same as the true category, an accuracy reward R is given. acc =1, otherwise R acc =0.

[0079] For the open vocabulary classification task, a semantic similarity calculation module (such as text embedding cosine similarity based on CLIP) is introduced. If the similarity is ≥ 0.8, it is judged as correct and assigned R acc =1, otherwise R acc =0.

[0080] S123 uses a predefined structured prompt template and an automated verification module based on regular expressions to perform real-time verification of the label integrity, numerical range (such as the bounding box coordinates must be integers between 0 and 1000), and confidence accuracy (retain two decimal places) of the output text. Violations will result in direct deduction of the corresponding reward points.

[0081] Specifically, first check the label integrity and force matching through regular expressions (such as <\s*think\s*>.*?<\s* / \s*think\s*>.*?<\s*answer\s*>.*?<\s* / \s*answer\s*) <think>and <answer>The existence and nesting order of the tags. If the tags are complete, the format reward is R format =1, otherwise R format =0.

[0082] For the detection task, set a numerical range check to verify whether the bounding box coordinates are integers between 0 and 1000 (regular expression: \d{1,3}|\d{4}), and whether the confidence level is 0.00-1.00 with two decimal places (regular expression: 0\.\d{2}|1\.00); if there are coordinates or confidence levels that exceed the limit, and the label integrity of the first step is met, the format reward R will be deducted. format 20%; otherwise, R format Set to 0. The total reward for the target detection task is defined as follows:

[0083] R d =αR IoU +βR conf +γR format

[0084] Among them, α, β and γ are weight values.

[0085] For classification tasks, set the classification task format test, verify <answer>Is it the name of a single category? It is prohibited to contain additional symbols or values. If it contains additional symbols or values ​​and meets the label integrity, then R format Set to 0. Define the total reward of the classification task as R cls =R acc +R format .

[0086] This embodiment provides a multimodal large model efficient reinforcement learning method based on policy gradient optimization, which includes the following steps:

[0087] S130, through structured prompt templates, standardize the reasoning steps and answer format of the model output, and enhance the model's self-learning ability.

[0088] This step specifically includes:

[0089] S131, design a structured prompt template based on XML tags, define <think>Reasoning process labels and <answer>The answer labels are formatted in a standardized way, and prompts are written to guide the model to think and reason before completing the task. The detection task prompts explicitly require the model to output a JSON array containing coordinate values. The classification task prompts force the species name to be output in a standardized format to ensure that the category name is strictly aligned with the predefined vocabulary.

[0090] Specifically, a unified XML tag system is defined, including <think>(reasoning process container), <answer>(Answer container) and task-specific sub-tags (such as the detection task <bbox>), and specifies the label hierarchy and nesting rules, as shown in Table 1.

[0091] Table 1 Prompt template example

[0092]

[0093] For classification tasks, you can define a vocabulary based on the dataset and embed vocabulary verification instructions in the template, for example: "Answers must use predefined nouns; common names or abbreviations are prohibited."

[0094] In addition, it supports dynamic injection of task parameters through placeholders (such as {category}, {resolution}) to generate scenario-customized prompts.

[0095] S132 uses regular expressions to parse model output content in real time and imposes format penalty rewards on responses that do not follow template specifications; establishes a dynamic feedback mechanism to backpropagate format verification results as reinforcement learning reward signals, and iteratively optimizes the model's structured output capabilities.

[0096] Specifically, using the regular expression:

[0097] <\s*think\s*>(.*?)<\s* / \s*think\s*>.*? <\s*answer\s*>(.*?)<\s* / \s*answer\s* enforces matching label structures. The reward score for this model response is calculated based on the verifiable reward rules defined in S123. Penalties are also imposed for format errors, including incorrect labels, value limits, and format errors, providing support for subsequent reinforcement learning algorithm applications.

[0098] S140 uses a reinforcement learning algorithm based on group relative policy optimization (GRPO) and combines it with KL divergence constraints to achieve stable updates of the policy model and guide the model to explore the optimal solution.

[0099] This step specifically includes:

[0100] S141, based on the current strategy model, generates G groups of independent responses to the input multimodal data (including visual images and text instructions), each group of responses must include a structured reasoning process and a final answer.

[0101] Specifically, in the model inference phase, a strategy combining temperature coefficient adjustment (T∈[0.7,1.3]) and Top-k sampling (k=50) is adopted to perform G=5 independent generation on the same input; in the decoding phase, the model is guided to generate <think>The inference text within the label is then generated <answer>Structured answers within.

[0102] S142, for each visual task feature, call a predefined verifiable reward function (such as the IoU reward for the object detection task and the accuracy reward for the classification task), and perform a comprehensive score on each group of responses in combination with the format constraint rules.

[0103] Specifically, based on the total reward of the target detection task defined in S123, the R of each group of responses is calculated. d =αR IoU +βR conf +γR format (Default α=0.6,β=0.3,γ=0.1); For classification tasks, calculate R based on the accuracy reward defined in S123 cls =R acc +R format .

[0104] S143, calculating the average reward value and standard deviation of the responses of group G, standardizing the reward value of each group's responses, and generating an advantage score reflecting the relative advantages and disadvantages within the group.

[0105] Specifically, the reward value of group G responses is calculated as the mean μ R and standard deviation σ R The formula for the standardized advantage score is:

[0106]

[0107] Among them, R i is the reward value of the response of group i, and ε=1e-6 is used to avoid division by zero error. Set the advantage score range A i ∈[-A clip ,A clip ](Default A clip =2.0), when A i When it is greater than Aclip, directly change A i The value is clipped to A clip ; If less than -A clip , then directly cut to -A clip , thereby avoiding extreme values ​​interfering with strategy updates.

[0108] S144 uses the advantage score to construct the policy gradient update direction and adjusts the policy model parameters by maximizing the expected reward objective function; at the same time, the KL divergence constraint is introduced in the reward objective function to ensure that the policy update amplitude is controlled to maintain training stability.

[0109] Specifically, the strategy optimization goal is:

[0110]

[0111] Among them, θ represents the parameters of the multimodal large model, o represents the response of the model, there are G in total, q represents the input of the model, and π represents the strategy, that is, the multimodal large model, π θ , π old , π ref Represents the current, past, and initial multimodal model. Considering that the model π generates G groups of responses and rewards each time, and the dataset size may be large, if one round of answer generation is only used for a single update of the model parameters, the efficiency of the model parameter update will be low. Therefore, in order to reuse the generated G groups of rewards to update the model multiple times, importance sampling is used. The expectation is converted into π old Expected calculation of the generated response and input data distribution β is the KL divergence penalty coefficient, with an initial value of 0.01. KL divergence is used to measure the distribution difference between the current strategy and the strategy at the beginning of training, to prevent the strategy from being updated too much and to prevent the model from losing stability due to drastic changes. Its formula is defined as follows:

[0112]

[0113] In addition, during training, the AdamW optimizer is used, the learning rate is set to lr = 5e-6, and the gradient is clipped globally.

[0114] S145, after each round of optimization, compare the output distribution differences of the new and old strategy models. If the KL divergence exceeds the threshold, the weight coefficient is adaptively adjusted. Through multiple rounds of iteration, the model is guided to prioritize strengthening high-reward response modes while retaining the potential to generate unexplored solutions, achieving a balanced optimization of task objectives and generalization capabilities.

[0115] Specifically, after each round of training, the KL divergence between the new and old strategies is calculated:

[0116]

[0117] Where i represents the i-th response and G represents the number of response groups.

[0118] Set the threshold τ KL =1.0, if KL avg >τ KL , triggering the dynamic adjustment of the KL divergence coefficient β:

[0119]

[0120] If the average KL divergence exceeds the threshold τ KL , then increase β to strengthen the policy constraints; if the KL divergence is too low, then reduce β to allow greater exploration.

[0121] In the above formula L(θ) for the policy gradient optimization objective, there is a β, where the KL divergence is used to characterize π θ The similarity between the probability distribution of the generated answer and πref is high. The higher the KL divergence, the greater the difference between the two distributions. If the difference is too large, we increase β. In this way, according to the formula, the optimization target L(θ) will become smaller. The optimization target here is as large as possible. In this way, it will guide the model iteration to avoid a large difference with πref. If the KL divergence is too small, it means that π θ and π ref If the model is very similar, the generated answers will be similar, and the model will not be able to explore new and more appropriate answers. If β is large, then if the model becomes similar to π ref If the gap is large, the optimization target will become smaller, but the optimization target here is as large as possible, so we need to reduce β and weaken the limit of KL divergence to make π θ Update more freely.

[0122] In addition, every K=5 rounds of training, the strategy π is saved θ is π for the next K=5 rounds of training old .

[0123] The present application also provides a visual task processing device, comprising:

[0124] An input unit, configured to obtain visual task data, wherein the visual task data includes: visual images and text instructions; and

[0125] A processing unit is configured to generate G groups of responses to the visual task data based on a policy model, wherein the policy model adopts a multimodal large model, and the responses include: a structured reasoning process and an answer; a predefined verifiable reward function and format constraint rules are used to comprehensively score each group of responses to obtain a reward value; the reward value is standardized to obtain an advantage score reflecting the advantage of each group of responses; and the advantage score is used to construct a policy gradient update direction, and the parameters of the policy model are adjusted by maximizing an expected reward objective function, and a final result is obtained after multiple rounds of iterations. The expected reward objective function includes: an expected calculation term for the distribution of responses and visual task data constructed by importance sampling, and a KL divergence constraint.

[0126] An embodiment of the present application also provides an electronic device, comprising: a processor, and a memory coupled to the processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method described in any one of the above embodiments.

[0127] The electronic device may be a computing device such as a desktop computer, a notebook computer, a PDA, a cloud server, etc. The electronic device may include, but is not limited to, a processor and a memory.

[0128] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the electronic device, connecting various parts of the entire device using various interfaces and lines.

[0129] The memory may be used to store the computer program, and the processor implements various functions of the electronic device by running or executing the computer program stored in the memory and calling the data stored in the memory.

[0130] The memory may primarily include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function, and the like; the data storage area may store data generated based on the use of the mobile phone, and the like. Furthermore, the memory may include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0131] The embodiment of the present application also provides a storage medium, which is a computer-readable storage medium, and the computer program is stored in the computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0132] An embodiment of the present application further provides a computer program product, including: a computer program or instructions, which, when executed on a computer, causes the computer to execute any of the above-mentioned possible implementation methods.

[0133] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications are also considered to be within the scope of protection of the present application.< / answer> < / think> < / bbox> < / answer> < / think> < / answer> < / think> < / answer> < / answer> < / think>

Claims

1. A multimodal large model task processing method based on reinforcement learning, characterized in that: The task processing method includes: obtaining visual task data, wherein the visual task data includes: visual images and text instructions; Generate G groups of responses to the visual task data based on a strategy model, wherein the strategy model adopts a multimodal large model, and the responses include: a structured reasoning process and an answer; Using a predefined verifiable reward function and format constraint rules, each group of responses is comprehensively scored to obtain a reward value; Normalizing the reward value to obtain a dominance score reflecting the dominance of each group of responses; and The advantage score is used to construct a policy gradient update direction. The parameters of the policy model are adjusted by maximizing an expected reward objective function, and the final result is obtained after multiple rounds of iteration. The expected reward objective function includes: an expected calculation term for the distribution of response and visual task data constructed by importance sampling, and a KL divergence constraint.

2. The task processing method according to claim 1, wherein: Vision tasks are: detection tasks or classification tasks.

3. The task processing method according to claim 2, wherein: When the visual task is a detection task, the verifiable reward function includes: an intersection-over-union reward item, a confidence reward item and a format constraint item. The intersection-over-union reward item is defined by the average intersection-over-union value of all prediction boxes, and the confidence reward item is defined by the average confidence value of all prediction boxes. When the visual task is a classification task, the verifiable reward function includes: an accuracy reward item and a format constraint item. The accuracy reward item and the format reward item are defined by binary numerical values ​​reflecting the degree of matching. The format constraint item is used to reward or punish those that meet the format requirements.

4. The task processing method according to claim 1, wherein: The advantage score is constructed by the following formula: Among them, A i is the advantage score of the response of group i, R i is the reward value, μ R is the mean reward value, σ R is the standard deviation of the reward value, and ε=1e-6 is used to avoid division by zero errors.

5. The task processing method according to claim 4, wherein: The dominance scores are clipped using the extremes of the dominance score range.

6. The task processing method according to claim 1, wherein: The expected reward objective function is constructed as follows: Among them, θ is the parameter of the policy model, L(θ) is the policy optimization target, o is the response, q is the visual task data input to the model, π is the policy model, π old , π θ , π ref is the policy model of the past, current, and initial training, is the expected calculation term, is the importance sampling, and β is the KL divergence penalty coefficient.

7. The task processing method according to claim 5, wherein: The task processing method further includes: After each round of iteration, the KL divergence mean KL of the new and old strategies is calculated by the following formula: avg : Among them, D KL for divergence calculations; and Determine whether the KL divergence mean meets the preset conditions. If so, the dynamic adjustment of the KL divergence penalty coefficient is triggered, as shown in the following formula: Among them, β new is the adjusted KL divergence penalty coefficient, β old is the KL divergence penalty coefficient before adjustment, τ KL is the threshold.

8. The task processing method according to claim 5, wherein: The task processing method further includes: Using the TOP-K strategy, save the strategy π every K rounds of training θ as π for the next K rounds of training old .

9. A task processing device, characterized in that: The task processing device includes: An input unit, configured to obtain visual task data, wherein the visual task data includes: visual images and text instructions; and A processing unit is configured to generate G groups of responses to the visual task data based on a policy model, wherein the policy model adopts a multimodal large model, and the responses include: a structured reasoning process and an answer; a predefined verifiable reward function and format constraint rules are used to comprehensively score each group of responses to obtain a reward value; the reward value is standardized to obtain an advantage score reflecting the advantage of each group of responses; and the advantage score is used to construct a policy gradient update direction, and the parameters of the policy model are adjusted by maximizing an expected reward objective function, and a final result is obtained after multiple rounds of iterations. The expected reward objective function includes: an expected calculation term for the distribution of responses and visual task data constructed by importance sampling, and a KL divergence constraint.

10. An electronic device, characterized in that: The electronic device includes: a processor, and a memory coupled to the processor, The memory is used to store computer programs; and The processor is configured to execute the computer program stored in the memory, so that the electronic device executes the task processing method according to any one of claims 1 to 8.

Images