Multi-granularity visual reasoning model construction method and device based on reinforcement learning

Abstract

The invention discloses a multi-granularity visual reasoning model construction method and device based on reinforcement learning. The method comprises the following steps: constructing an'image-reasoning query-bounding box 'triple as a training data set; designing a composite reward function including positioning precision, target counting precision and format reward; training the multi-modal large language model by adopting a GRPO algorithm; the trained model can output a region-level bounding box, and a pixel-level mask is generated and a contour-level result is extracted in combination with the segmentation model. According to the method, the problems that in the prior art, a reasoning path depends on manual annotation, multi-granularity tasks cannot be expanded, and generalization is insufficient are solved, and the autonomous decision-making ability, task expansibility and generalization in a distribution offset scene of the model are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-modal artificial intelligence, and particularly relates to a multi-granularity visual reasoning model construction method and device based on reinforcement learning. BACKGROUND

[0002] Visual reasoning tasks need to deal with complex spatial relationships and ambiguous user intent (such as "find the object in the image that may cause the notebook computer to short circuit"). Current visual reasoning technology faces three major bottlenecks: first, mainstream methods such as LISA rely on the supervised fine-tuning paradigm, which requires manual annotation of complete reasoning paths as training data, resulting in a lack of autonomous exploration ability for the model and insufficient generalization in complex query scenarios. Second, existing technologies usually adopt task-specific architecture design, for example, PixelLM only supports pixel-level segmentation, and VLM-R1 is limited to region detection, resulting in system redundancy and high extension cost. More importantly, traditional methods cannot achieve cross-granularity task collaboration, such as deriving target region coordinates, semantic segmentation masks, and geometric contours from the same instruction.

[0003] These limitations are due to two fundamental technical defects: first, supervised training forces the model to fit the pre-set reasoning path, suppressing its ability to model implicit spatial relationships autonomously; second, fixed output head design causes strong coupling between model architecture and task granularity. Although recent research attempts to alleviate this problem through multi-task learning, it still requires the introduction of additional task decoders for each type of task, failing to achieve true unified reasoning.

[0004] The existing technology one provides SegEarth-R, SegEarth-R1 is a language-guided segmentation model for remote sensing images, aiming to solve the implicit query and spatial reasoning problems that traditional segmentation methods cannot handle. The model achieves the generation of corresponding target region masks from natural language implicit descriptions by introducing the "Geospatial Pixel Reasoning" task. The overall architecture consists of three core modules: hierarchical visual encoder, large language model (LLM), and spatially related mask generator. Among them, the visual encoder is based on Swin Transformer, which has multi-scale feature extraction capability, used to preserve small-scale object details and capture global semantic information; the text is parsed and encoded by the LLM, combined with the visual features for semantic understanding.

[0005] To adapt to the high resolution and redundancy characteristics of remote sensing images, SegEarth-R1 designs a visual token compression module, which significantly reduces the number of visual tokens processed by LLM through convolutional downsampling and normalization processing, reduces the computational cost, while preserving semantic information. In addition, to achieve efficient alignment of language description and spatial position, the model introduces a "description projector" (D-Projector) to project text encoding into a unified vector and interact with image multi-scale features through cross-attention mechanism, and then guide the Mask2Former decoder to perform mask prediction. Unlike the traditional method of using fixed query tokens, this scheme directly uses text embedding as mask query, improving the perception of implicit spatial relationships.

[0006] However, the prior art only supports segmentation, a single task, and cannot handle multi-granularity requirements such as region detection and contour extraction. At the same time, its training method relying on supervised fine-tuning leads to a single thinking process of the model, losing the original generalization ability of multi-modal large models.

[0007] The second prior art provides LISA, which is a language-guided segmentation framework for natural images. Its core idea is to use a large language model (LLM) to understand natural language instructions and combine image features for semantic segmentation. Its main construction strategy is to jointly embed language and image modalities and build a text-image bridge through a newly introduced [SEG] token, thereby achieving semantic alignment and mask prediction between language and image. Specifically, the image is processed by a pre-trained visual encoder such as CLIP to generate visual tokens, and the text is encoded by LLM (such as Vicuna) into semantic representations. Both are injected into a shared Transformer decoder through the [SEG] token to generate a segmentation mask.

[0008] The key innovation of LISA is its unified language instruction interface, which allows the model to understand the task type (such as segmentation, detection, etc.) through an instruction template, thereby enabling flexible multi-task reasoning. In addition, its mask generation uses a SAM (Segment Anything Model) style decoder structure, making the output have high-quality pixel-level segmentation capability.

[0009] However, the second prior art does not support fine-grained tasks such as contour extraction; the model cannot output the reasoning process, making it difficult to trace the decision logic, limiting its application in high-reliability scenarios.

[0010] In summary, the existing visual reasoning technology has defects in autonomous reasoning ability, multi-granularity task support, and cross-task collaboration, and a new model construction method is urgently needed to solve these problems. SUMMARY

[0011] To solve the above technical problems, the present application provides a multi-granularity visual reasoning model construction method and device based on reinforcement learning.

[0012] To achieve the above purpose, the technical scheme of the present application is as follows:

[0013] In the first aspect, the present application discloses a multi-granularity visual reasoning model construction method based on reinforcement learning, comprising:

[0014] Step S1: constructing a training data set, the training data set is a "image-reasoning query-boundary box" triple, by adding a preset prompt word in the reasoning sample, splicing the prompt word with the original query information, obtaining the reasoning information, and then combining the image and the corresponding boundary box label to form a triple;

[0015] Step S2: design the following composite reward function :

[0016] ;

[0017] Wherein: is the positioning accuracy reward, is the target counting accuracy reward, is the format reward;

[0018] Step S3: training a multi-modal large language model using the GRPO algorithm, and dynamically evaluating the output quality through the composite reward function;

[0019] The optimization objective function of the GRPO algorithm is:

[0020] ;

[0021] ;

[0022] Wherein:

[0023] is the parameter of the multi-modal large language model to be optimized;

[0024] is a set of N candidate outputs sampled from the old policy distribution, is the i-th candidate output;

[0025] is the old policy distribution;

[0026] is the input query information, i.e. "reasoning query" in the "image-reasoning query-boundary box" triple in step S1;

[0027] is the current model parameter the probability of generating the candidate output given the query ;

[0028] the probability of generating the candidate output given the query ;

[0029] the relative advantage value corresponding to the i-th candidate output, the reward value corresponding to the i-th candidate output, obtained by the compound reward function;

[0030] the mean of all reward values;

[0031] the standard deviation of all reward values;

[0032] the clipping parameter;

[0033] the clipping function;

[0034] the KL divergence regular term;

[0035] the coefficient of the KL divergence regular term;

[0036] Step S4: performing multi-granularity visual reasoning based on the trained multi-modal large language model, comprising:

[0037] The trained multi-modal large language model can directly output a region-level target bounding box;

[0038] The bounding box and its center point are input as a prompt into a segmentation model to generate a pixel-level segmentation mask;

[0039] The segmentation mask is analyzed to extract the outermost vertices of the target to obtain a contour-level result.

[0040] On the basis of the above technical solutions, the following improvements can be made:

[0041] As a preferred solution, in step S1, the preset prompt words include text understanding rules, output format requirements, coordinate rules, and output specifications.

[0042] As a preferred solution, in step S2, the positioning accuracy reward is used to measure the positioning accuracy, which is obtained by calculating the intersection over union of the positioning box after Hungarian matching, and the calculation formula is as follows:

[0043] ​​​ ;

[0044] Wherein: is a set of real boxes, is a predicted box, is a real box;

[0045] Target counting accuracy reward The target counting accuracy reward is used to measure the accuracy of the number of targets predicted by the model, and an exponential decay function is used to punish the deviation of the number of targets predicted, and the calculation formula is:

[0046] ;

[0047] Wherein: is the number of real boxes, is the number of predicted boxes;

[0048] Format reward The format reward is used to check whether the output conforms to the preset format, and returns the specified parameters according to the compliance.

[0049] As a preferred solution, in step S4, the morphological method or the deep learning model is used to analyze the segmentation mask, extract the outermost polygon vertices of the target, and obtain the contour level result.

[0050] In a second aspect, the application discloses a multi-granularity visual reasoning model construction device based on reinforcement learning, comprising:

[0051] A training data set construction module is used to construct a training data set, and the training data set is a "image-reasoning query-boundary box" triple. By adding a preset prompt word in the reasoning sample, the prompt word is spliced with the original query information to obtain reasoning information, and then the image and the corresponding boundary box label are combined to form a triple.

[0052] A composite reward function design module is used to design the following composite reward function :

[0053] ;

[0054] Wherein: is a positioning accuracy reward, is a target counting accuracy reward, is a format reward;

[0055] A training module is used to train a multi-modal large language model using a GRPO algorithm, and the output quality is dynamically evaluated by a composite reward function;

[0056] The optimization objective function of the GRPO algorithm is:

[0057] ;

[0058] ;

[0059] wherein:

[0060] is a parameter of the multi-modal large language model to be optimized;

[0061] is a set of N candidate outputs sampled from the old policy distribution, is the i-th candidate output;

[0062] is the old policy distribution;

[0063] is the input query information, i.e., the "reasoning query" in the "image-reasoning query-bounding box" triple in step S1;

[0064] is the probability of generating the candidate output under the current model parameter , given the query ;

[0065] is the probability of generating the candidate output under the flow model parameter , given the query ;

[0066] is the relative advantage value corresponding to the i-th candidate output, is the reward value corresponding to the i-th candidate output, obtained by the composite reward function;

[0067] is the mean of all reward values;

[0068] is the standard deviation of all reward values;

[0069] is the clipping parameter;

[0070] is the clipping function;

[0071] is the KL divergence regularization term;

[0072] is the coefficient of the KL divergence regularization term;

[0073] a reasoning module configured to perform multi-granularity visual reasoning based on the trained multi-modal large language model, comprising:

[0074] The trained multi-modal large language model can directly output a region-level target bounding box;

[0075] The bounding box and the center point are input into the segmentation model as a prompt to generate a pixel-level segmentation mask;

[0076] The segmentation mask is analyzed to extract the outermost polygon vertices of the target to obtain a contour-level result.

[0077] As a preferred scheme, in the training data set construction module, the preset prompt words include text understanding rules, output format requirements, coordinate rules and output specifications.

[0078] As a preferred scheme, in the composite reward function design module,

[0079] Positioning accuracy reward Used to measure the positioning accuracy, the intersection over union of the positioning box is calculated after the Hungarian matching, and the calculation formula is as follows:

[0080] ;

[0081] Wherein: is the real box set, is the predicted box, is the real box;

[0082] Target counting accuracy reward Used to measure the target number accuracy predicted by the model, an exponential decay function is used to punish the target number prediction deviation, and the calculation formula is:

[0083] ;

[0084] Wherein: is the number of real boxes, is the number of predicted boxes;

[0085] Format reward Used to check whether the output conforms to the preset format, and return specified parameters according to compliance.

[0086] As a preferred scheme, in the reasoning module, a morphological method or a deep learning model is used to analyze the segmentation mask, extract the outermost polygon vertices of the target, and obtain a contour-level result.

[0087] In a third aspect, the present application discloses a computing device, comprising:

[0088] One or more processors;

[0089] Memory;

[0090] and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions of any of the above-mentioned reinforcement learning-based multi-granularity visual reasoning model construction methods.

[0091] In a fourth aspect, the present application discloses a storage medium, which stores one or more computer-readable programs, and the one or more programs include instructions suitable for being loaded by a memory and executing any of the above-mentioned reinforcement learning-based multi-granularity visual reasoning model construction methods.

[0092] The reinforcement learning-based multi-granularity visual reasoning model construction method and device have the following beneficial effects:

[0093] Firstly, the present application can complete training data construction only by using boundary box labeling, without pixel-level or contour-level labels, thereby greatly reducing data labeling cost and workload and solving the problem that the prior art depends on multi-type labeling for multi-granularity tasks.

[0094] Secondly, the present application drives region-level, pixel-level and contour-level tasks through a single multi-modal large language model output, avoids redundant design of task-specific decoders, reduces system expansion cost, and realizes efficient collaboration of cross-granularity tasks.

[0095] Thirdly, the present application improves generalization capability through general strategy learning of reinforcement learning, and improves generalization performance of the model in a distribution offset scene. BRIEF DESCRIPTION OF DRAWINGS

[0096] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0097] Figure 1 The flowchart of the multi-granularity visual reasoning model construction method provided by the embodiments of the present application.

[0098] Figure 2 The framework schematic diagram of the multi-granularity visual reasoning model provided by the embodiments of the present application. DETAILED DESCRIPTION

[0099] The preferred embodiments of the present application will be described in detail below with reference to the drawings.

[0100] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0101] The expression "comprising" is an "open" expression that merely means that the corresponding parts or steps are present, but should not be interpreted as excluding additional parts or steps.

[0102] To achieve the object of the present application, some embodiments of the multi-granularity visual reasoning model construction method based on reinforcement learning include, as shown in Figures 1-2 The method comprises the following steps:

[0103] Step S101: Construct a training data set, which is a "image-reasoning query-boundary box" triple. By adding a preset prompt word in the reasoning sample, the prompt word is spliced with the original query information to obtain reasoning information, and then the image and the corresponding boundary box label are combined to form a triple.

[0104] Among them: the boundary box in the triple is the real label annotated by artificial (that is: the real box described later);

[0105] Step S102: design the following compound reward function :

[0106] ;

[0107] Among them: is the positioning accuracy reward, is the target counting accuracy reward, is the format reward;

[0108] Step S103: train the multi-modal large language model (MLLM) by using the group relative policy optimization (GRPO) algorithm, and dynamically evaluate the output quality by using the compound reward function;

[0109] The optimization objective function of the GRPO algorithm is:

[0110] ;

[0111] ;

[0112] Among them:

[0113] parameters of the multimodal large language model to be optimized, are constantly adjusted through the training process to improve the visual reasoning performance of the model;

[0114] N candidate output sets sampled from the old policy distribution, is the i-th candidate output, representing the visual reasoning result (such as the target bounding box and the corresponding thinking process) generated by the model for the input query;

[0115] is the old policy distribution, the model policy before updating in the training process, used for comparison with the updated current policy to constrain the updating range of the policy;

[0116] is the input query information, i.e. the "reasoning query" in the "image-reasoning query-bounding box" triple in step S1;

[0117] is the probability of generating the candidate output under the current model parameters given the query ;

[0118] is the probability of generating the candidate output under the flow model parameters given the query ;

[0119] is the relative advantage value corresponding to the i-th candidate output, used to measure the advantages and disadvantages of this output relative to other outputs, is the reward value corresponding to the i-th candidate output, obtained through the composite reward function;

[0120] is the mean of all reward values;

[0121] is the standard deviation of all reward values;

[0122] is the clipping parameter, used to limit the updating range of the policy to avoid unstable model training due to excessive updating;

[0123] is the clipping function;

[0124] is the KL divergence regular term, used to constrain the updating range of the policy;

[0125] is the coefficient of the KL divergence regular term;

[0126] Step S104: multi-granularity visual reasoning based on the trained multi-modal large language model, including:

[0127] The trained multi-modal large language model can directly output a region-level target bounding box (Bbox);

[0128] The bounding box and its center point are input into the segmentation model as prompts to generate a pixel-level segmentation mask (Mask);

[0129] The segmentation mask is analyzed using a morphological method to extract the outermost vertices of the target to obtain a contour-level result (Contour).

[0130] In the above step S103, the PPO algorithm can be used to replace GRPO, and the CriticModel is additionally trained to realize reinforcement learning.

[0131] Each of the above steps will be described in detail below.

[0132] In step S101, the preset prompt words can include but are not limited to text understanding rules, output format requirements, coordinate rules, and output specifications.

[0133] Further, the output format requirement can be:

[0134] If there is a single target, the output format is: [x min , y min , x max , y max ];

[0135] If there is no matching target, the output format is: an empty list [];

[0136] If there are multiple targets, the output format is: [[x min1 , y min1 , x max1 , y max1 ], [x min2 , y min2 , x max2 ,y max2 ],...].

[0137] The coordinate rule can be: the coordinate value is an integer representing the pixel position, and the bounding box is the tight enclosing box of the target.

[0138] The output specification can be: the thinking process is recorded in <think>In the tag, the positioning result is output to <answer>In the label, and the whole is in JSON format.

[0139] Example one: the prompt word can be as follows:

[0140]

[0141] The prompt word is spliced with the query information in the original data (such as directly spliced after "Text description:”), and the bounding box label and the picture constitute a "image-inference query-bounding box” triple.

[0142] Further, in step S102, the composite reward function R is used to optimize the positioning accuracy, target counting accuracy and output format of the model.

[0143] Positioning accuracy reward Used to measure the positioning accuracy, the Intersection-over-Union (IoU) of the positioning box is calculated after the Hungarian matching, and the calculation formula is as follows:

[0144] ;

[0145] Wherein: is the real box set, is the predicted box, is the real box.

[0146] Target counting accuracy reward Used to measure the target number accuracy predicted by the model, an exponential decay function is used to punish the target number prediction deviation, and the calculation formula is:

[0147] ;

[0148] Wherein: is the number of real boxes, is the number of predicted boxes.

[0149] Format reward Used to check whether the output conforms to the preset format, and returns 1 or 0 according to the compliance.

[0150] For example: the output content should be in JSON format, and the thinking process is recorded in the <think> ...< / think> label, and the positioning result is output in the <answer>[[x1, y1, x2, y2]...]< / answer> label.

[0151] Further, step S104 obtains the inference results of multi-granularity tasks (such as detection, segmentation and contour extraction tasks).

[0152] After training, the multi-modal large language model itself can generate the bounding box (bbox) representing the target area.

[0153] To further enhance the workflow and enable pixel-level and object-level inference tasks, the present application inputs the bbox and its center point as a prompt to a segment everything model (Segment Anything Model 2, SAM2) to generate a pixel-level mask.

[0154] Subsequently, a morphological method is applied to analyze the mask image to extract the outermost polygon vertices to obtain the target contour.

[0155] Notably, in another embodiment, a deep learning model can also be used to analyze the segmentation mask to obtain a contour-level result.

[0156] The existing technology has the following three problems: (1) the supervised fine-tuning paradigm relies on manually annotated inference paths, which inhibits the model's autonomous decision-making ability; (2) the task-specific decoder architecture prevents the system from being extended to multi-granularity tasks; (3) traditional training methods lack generalizability in distribution shift scenarios.

[0157] The present application solves these problems through an innovative reinforcement learning training mechanism and a multi-granularity inference engine, as follows:

[0158] (1) The existing technology relies on supervised fine-tuning, which requires manual annotation of complete inference paths, forcing the model to fit pre-set logic and resulting in a lack of autonomous exploration ability.

[0159] The present application breaks through this limitation through a reinforcement learning training paradigm. Instead of relying on "manually annotated inference paths" for supervised fine-tuning, the present application constructs "image-inference query-bounding box" triplets as training data, requiring only bounding box annotations (without the need for inference path annotations). A composite reward function is designed to dynamically guide model optimization through positioning accuracy, counting accuracy, and format compliance, rather than forcing the model to fit pre-set paths.

[0160] The GRPO reinforcement learning algorithm is used to train a multi-modal large language model (MLLM), allowing the model to autonomously explore optimal inference strategies in interactions with the environment (data), iteratively optimizing decision logic through reward feedback, and ultimately possessing autonomous modeling capabilities for implicit spatial relationships.

[0161] (2) The existing technology uses task-specific decoders (such as the mask head of SegEarth-R1), resulting in strong coupling between the model and the task granularity, which prevents the model from supporting unified inference for region-level, pixel-level, and contour-level tasks.

[0162] The application realizes unified expansion through a multi-granularity task conversion mechanism. The trained MLML directly outputs a region-level target bounding box (region-level task) without a dedicated detection decoder; a SAM2 model is called to generate a pixel-level segmentation mask (pixel-level task) by prompting with the bounding box and center point, and a general segmentation tool is reused instead of a dedicated mask head; a morphological method is used to analyze the segmentation mask to extract the outermost polygon vertices to obtain the contour-level result (contour-level task), avoiding the introduction of a contour-specific decoder.

[0163] Through the lightweight process of "single MLML output + general tool reuse", the task-specific decoder is completely eliminated, and unified support and flexible expansion of multi-granularity tasks are realized.

[0164] (3) Traditional supervised fine-tuning relies on the distribution of specific labeled data, and when encountering unseen targets or scenes (distribution shift), the model has poor generalization due to overfitting to the training data pattern.

[0165] The application improves generalization through general policy learning of reinforcement learning. In reinforcement learning training, the model learns a general goal of "positioning accuracy, counting correctness, and format compliance" through a composite reward function, rather than fitting the reasoning path pattern in specific data, forming an abstract modeling ability for spatial relationships; the GRPO algorithm constrains policy updates through relative advantage values (Q ) and KL divergence regularization terms, so that the model pays more attention to the relative difference in rewards rather than the absolute value during training, enhancing adaptability to distribution changes.

[0166] The actual effect shows that the accuracy of the method in the zero-shot visual positioning task reaches 49.85%, and the average accuracy of out-of-distribution target recognition reaches 36%, verifying its strong generalization in distribution shift scenarios.

[0167] Therefore, the application has the following beneficial effects:

[0168] First, the data labeling efficiency is significantly improved. Only the bounding box label is needed to complete the training data construction, without pixel-level or contour-level labels, greatly reducing the data labeling cost and workload, and solving the problem of multi-granularity task dependence on multiple types of labels in the prior art.

[0169] Second, the system performance is comprehensively optimized. The accuracy on multi-granularity tasks is improved compared to existing methods, for example, the pixel-level task is improved by 3.67% compared to SegEarth-R1;

[0170] The accuracy of the zero-shot visual positioning task reaches 49.85%, and the average accuracy of out-of-distribution target recognition reaches 36%, verifying the strong generalization of the model in unseen scenarios or targets, and solving the problem of poor performance of traditional methods in distribution shift scenarios.

[0171] Fourth, the inference speed is greatly accelerated. The contour extraction link takes only 0.09 seconds per image, which is 4.7 times faster than the existing EPOC method, and the lightweight inference process significantly improves the processing efficiency in practical applications.

[0172] Fifth, the task expansion and compatibility are enhanced. The single multi-modal large language model (MLLM) output drives the region-level, pixel-level and contour-level tasks, avoiding the redundant design of task-specific decoders, reducing the system expansion cost, and realizing the efficient cooperation of cross-granularity tasks.

[0173] In some other embodiments, the present application discloses a multi-granularity visual inference model construction device based on reinforcement learning, comprising:

[0174] A training data set construction module is used to construct a training data set, which is a "image-inference query-boundary box" triple. By adding a preset prompt word in the inference sample, the prompt word is spliced with the original query information, and after obtaining the inference information, the image and the corresponding boundary box label are combined to form a triple.

[0175] A composite reward function design module is used to design the following composite reward function :

[0176] ;

[0177] Wherein: is the positioning accuracy reward, is the target counting accuracy reward, is the format reward;

[0178] A training module is used to train the multi-modal large language model using the GRPO algorithm, and the output quality is dynamically evaluated by the composite reward function.

[0179] The optimization objective function of the GRPO algorithm is:

[0180] ;

[0181] ;

[0182] Wherein:

[0183] is the parameter of the multi-modal large language model to be optimized;

[0184] is a set of N candidate outputs sampled from the old policy distribution, is the i-th candidate output;

[0185] is the old policy distribution;

[0186] for the input query information, i.e., the "reasoning query" in the "image-reasoning query-bounding box" triple in step S1;

[0187] for the current model parameters , given the query , the probability of generating the candidate output ;

[0188] for the flow model parameters , given the query , the probability of generating the candidate output ;

[0189] is the relative advantage value corresponding to the i-th candidate output, is the reward value corresponding to the i-th candidate output, obtained by the composite reward function;

[0190] is the mean of all reward values;

[0191] is the standard deviation of all reward values;

[0192] is the clipping parameter;

[0193] is the clipping function;

[0194] is the KL divergence regularization term;

[0195] is the coefficient of the KL divergence regularization term;

[0196] a reasoning module configured to perform multi-granularity visual reasoning based on the trained multi-modal large language model, comprising:

[0197] the trained multi-modal large language model can directly output a region-level target bounding box;

[0198] input the bounding box and its center point as a prompt to the segmentation model to generate a pixel-level segmentation mask;

[0199] analyze the segmentation mask to extract the outermost vertices of the target to obtain a contour-level result.

[0200] Further, in the training data set construction module, the preset prompt words include text understanding rules, output format requirements, coordinate rules, and output specifications.

[0201] Further, in the composite reward function design module,

[0202] Positioning accuracy reward For measuring the positioning accuracy, the intersection over union of the positioning box is calculated after the Hungarian matching, and the calculation formula is as follows:

[0203] ;

[0204] Wherein: is a set of real boxes, is a predicted box, is a real box;

[0205] Target counting accuracy reward For measuring the target number accuracy predicted by the model, an exponential decay function is used to punish the target number prediction deviation, and the calculation formula is as follows:

[0206] ;

[0207] Wherein: is the number of real boxes, is the number of predicted boxes;

[0208] Format reward For checking whether the output conforms to the preset format, the specified parameters are returned according to the compliance.

[0209] Further, in the inference module, the morphological method or the deep learning model is used to analyze the segmentation mask, the outermost polygon vertices of the target are extracted, and the contour level result is obtained.

[0210] Further, it should be noted that: the above embodiment provides a multi-granularity visual inference model construction device based on reinforcement learning, which is only used as an example for dividing the above functional modules during model construction. In actual application, the above functions can be completed by different functional modules according to the needs, that is, the internal structure of the multi-granularity visual inference model construction device based on reinforcement learning is divided into different functional modules to complete all or part of the functions described above.

[0211] In addition, the multi-granularity visual inference model construction device based on reinforcement learning and the multi-granularity visual inference model construction method provided in the above embodiment belong to the same concept, and the specific implementation process is described in the method embodiment, which will not be repeated here.

[0212] In some other embodiments, the present application discloses a computing device, comprising:

[0213] One or more processors;

[0214] Memory;

[0215] and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions of any of the above-mentioned reinforcement learning-based multi-granularity visual reasoning model construction methods.

[0216] In some other embodiments, the present application discloses a storage medium, which stores one or more computer-readable programs, and the one or more programs include instructions suitable for being loaded by a memory and performing any of the above-mentioned reinforcement learning-based multi-granularity visual reasoning model construction methods.

[0217] The present application discloses a reinforcement learning-based multi-granularity visual reasoning model construction method and device, which has the following beneficial effects:

[0218] Firstly, the present application only needs to complete the training data construction by boundary box labeling, without pixel-level or contour-level labels, greatly reducing the data labeling cost and workload, and solving the problem of multi-granularity task relying on multi-type labeling in the prior art.

[0219] Secondly, the present application drives the region-level, pixel-level and contour-level tasks through a single multi-modal large language model output, avoiding the redundant design of task-specific decoders, reducing the system expansion cost, and realizing the efficient cooperation of cross-granularity tasks.

[0220] Thirdly, the present application improves the generalization ability through the general strategy learning of reinforcement learning, and improves the generalization performance of the model in the distribution offset scene.

[0221] The above shows and describes the basic principles and main features of the present application and the advantages of the present application, and those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles of the present application, and various changes and improvements can be made to the present application without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the present application, and the scope of protection of the present application is defined by the appended claims and their equivalents.< / answer> < / think>

Claims

1. A method for constructing a multi-granularity visual reasoning model based on reinforcement learning, characterized in that, include: Step S1: Construct a training dataset, which is a "image-inference query-bounding box" triple. By adding preset prompt words to the inference samples, the prompt words are concatenated with the original query information to obtain the inference information. Then, the inference information is combined with the image and the corresponding bounding box annotation to form the triple. Step S2: Design the following composite reward function : ； in: As a reward for positioning accuracy, Rewards for target counting accuracy. Formatted rewards; Step S3: Train a multimodal large language model using the GRPO algorithm, and dynamically evaluate the output quality using a composite reward function; The objective function of the GRPO algorithm is: ； ； in: The parameters are those of the multimodal large language model to be optimized. Given a set of N candidate outputs sampled from the old policy distribution, This is the i-th candidate output; The distribution follows the old strategy. The input query information is the "inference query" in the "image-inference query-bounding box" triplet in step S1; To the current model parameters Below, given a query Generate candidate outputs in time The probability of; For the parameters of the flow model Below, given a query Generate candidate outputs in time The probability of; Let be the relative advantage value corresponding to the i-th candidate output. The reward value corresponding to the i-th candidate output is obtained through a composite reward function; The average of all reward values; The standard deviation of all reward values; For editing parameters; For editing functions; This is the KL divergence regularization term; The coefficient of the KL divergence regularization term; Step S4: Perform multi-granularity visual reasoning based on the trained multimodal large language model, including: The trained multimodal large language model can directly output region-level target bounding boxes; The bounding box and its center point are used as prompts input to the segmentation model to generate a pixel-level segmentation mask; Analyze the segmentation mask, extract the outermost polygon vertices of the target, and obtain the contour-level result.

2. The method for constructing a multi-granularity visual reasoning model according to claim 1, characterized in that, In step S1, the preset prompt words include text comprehension rules, output format requirements, coordinate rules, and output specifications.

3. The method for constructing a multi-granularity visual reasoning model according to claim 1, characterized in that, In step S2, the positioning accuracy reward Used to measure positioning accuracy, it is obtained by calculating the intersection-union ratio of the positioning boxes after Hungarian matching. The calculation formula is as follows: ； in: For the set of real bounding boxes, For the prediction box, For true bounding boxes; The target counting accuracy reward To measure the accuracy of the model's predicted target quantity, an exponential decay function is used to penalize the target quantity prediction bias. The calculation formula is as follows: ； in: The number of true bounding boxes. The number of predicted boxes; The format reward Used to check whether the output conforms to the preset format, and returns the specified parameters based on compliance.

4. The method for constructing a multi-granularity visual reasoning model according to claim 1, characterized in that, In step S4, the segmentation mask is analyzed using morphological methods or deep learning models to extract the outermost polygon vertices of the target and obtain the contour-level result.

5. A device for constructing multi-granularity visual reasoning models based on reinforcement learning, characterized in that, include: The training dataset construction module is used to construct the training dataset, which is a "image-inference query-bounding box" triple. By adding preset prompt words to the inference samples, the prompt words are concatenated with the original query information to obtain the inference information. Then, the inference information is combined with the image and the corresponding bounding box annotation to form the triple. The composite reward function design module is used to design the following composite reward functions. : ； in: As a reward for positioning accuracy, Rewards for target counting accuracy. Formatted rewards; The training module is used to train a multimodal large language model using the GRPO algorithm and dynamically evaluate the output quality through a composite reward function. The objective function of the GRPO algorithm is: ； ； in: The parameters are those of the multimodal large language model to be optimized. Given a set of N candidate outputs sampled from the old policy distribution, This is the i-th candidate output; The distribution follows the old strategy. The input query information is the "inference query" in the "image-inference query-bounding box" triplet in step S1; To the current model parameters Below, given a query Generate candidate outputs in time The probability of; For the parameters of the flow model Below, given a query Generate candidate outputs in time The probability of; Let be the relative advantage value corresponding to the i-th candidate output. The reward value corresponding to the i-th candidate output is obtained through a composite reward function; The average of all reward values; The standard deviation of all reward values; For editing parameters; For editing functions; This is the KL divergence regularization term; The coefficient of the KL divergence regularization term; The reasoning module, used for multi-granularity visual reasoning based on the trained multimodal large language model, includes: The trained multimodal large language model can directly output region-level target bounding boxes; The bounding box and its center point are used as prompts input to the segmentation model to generate a pixel-level segmentation mask; Analyze the segmentation mask, extract the outermost polygon vertices of the target, and obtain the contour-level result.

6. The multi-granularity visual reasoning model construction device according to claim 5, characterized in that, In the training dataset construction module, the preset prompt words include text understanding rules, output format requirements, coordinate rules, and output specifications.

7. The multi-granularity visual reasoning model construction device according to claim 6, characterized in that, In the composite reward function design module The positioning accuracy reward Used to measure positioning accuracy, it is obtained by calculating the intersection-union ratio of the positioning boxes after Hungarian matching. The calculation formula is as follows: ； in: For the set of real bounding boxes, For the prediction box, For true bounding boxes; The target counting accuracy reward To measure the accuracy of the model's predicted target quantity, an exponential decay function is used to penalize the target quantity prediction bias. The calculation formula is as follows: ； in: The number of true bounding boxes. The number of predicted boxes; The format reward Used to check whether the output conforms to the preset format, and returns the specified parameters based on compliance.

8. The multi-granularity visual reasoning model construction device according to claim 6, characterized in that, In the inference module, morphological methods or deep learning models are used to analyze the segmentation mask, extract the outermost polygon vertices of the target, and obtain the contour-level result.

9. A computing device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions for the reinforcement learning-based multi-granularity visual reasoning model construction method as described in any of claims 1-4.

10. A storage medium, characterized in that, The storage medium stores one or more computer-readable programs, the programs including instructions adapted to be loaded by memory and executed as described in any of claims 1-4 above, for constructing a multi-granularity visual reasoning model based on reinforcement learning.

Images