Training methods, equipment, and storage media for visual language models

Abstract

This application discloses a training method, device, and storage medium for a visual language model, comprising: supervising semantic control tokens using cross-entropy loss; constructing a Gaussian soft target distribution for spatial coordinate tokens based on their real coordinates and local neighborhoods, and calculating spatial tolerance loss; fine-tuning the visual language model by combining the two losses; using the fine-tuned model as the initial policy, cropping the image according to the bounding boxes output by the model in a dynamic cropping environment, and returning the cropped visual feedback to the model context; calculating a comprehensive reward based on the output trajectory, and then calculating the group relative advantage of multiple trajectories to update the policy. This application, by decoupling semantics and coordinate tokens and applying a Gaussian soft target distribution to the coordinates, matches the supervision signal with the visual semantic tolerance, and combines dynamic visual feedback with group relative policy optimization to guide the model to adaptively generate compact cropping boxes, thereby improving inference accuracy.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a training method, device and storage medium for a visual language model. Background Technology

[0002] Visual language models are evolving from passive image understanding to interactive visual reasoning. In this process, models need to utilize visual tools such as cropping and scaling to obtain fine-grained visual evidence, requiring reliable coordinate localization capabilities. Currently, visual language models based on autoregressive architectures typically serialize continuous two-dimensional spatial coordinates into discrete token sequences and employ standard token-level cross-entropy loss functions for pixel-level supervised training.

[0003] Traditional coordinate token supervision treats each coordinate token as an independent classification label, requiring the model's predicted coordinates to be perfectly identical to the ground truth labels at the pixel level. Under this objective, even if the predicted coordinates deviate from the ground truth labels by only one or two pixels, the resulting cropped image will be visually and semantically almost identical to a correctly cropped image, yet the model will be penalized as severely as with completely incorrect tokens. This pixel-perfect hard-label supervision excessively penalizes spatial "short-range errors" and fails to reflect the inherent tolerance of visual features within local space.

[0004] Furthermore, visual reasoning tasks inherently benefit more from compact and focused visual evidence, i.e., precise cropped regions that can exclude irrelevant backgrounds. However, due to the unreliability of the model's localization ability, in multi-turn interactive reasoning, the model often tends to generate larger and noisier cropped boxes as a margin of error to ensure target coverage. While this conservative strategy improves recall, it introduces a large amount of background noise, which in turn weakens the accuracy of subsequent reasoning. The model falls into a vicious cycle of "enlarging cropped boxes to achieve coverage, resulting in inaccurate reasoning due to background interference."

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of this application is to provide a training method, device and storage medium for a visual language model, aiming to solve the technical problem of mismatch between the coordinate prediction supervision signal and the visual semantic tolerance of the visual language model.

[0007] To achieve the above objectives, this application proposes a method for training a visual language model, the method comprising: Decouple the training sequences in the visual language model by tokens, and divide the tokens in the sequence into semantic control tokens and spatial coordinate tokens; The semantic control tokens are trained under supervision using cross-entropy loss. The spatial tolerance loss is calculated by constructing a Gaussian smooth soft target probability distribution for the spatial coordinate tokens based on the real coordinates and the local neighborhood within a preset step size. By combining the cross-entropy loss and the spatial tolerance loss, the visual language model is subjected to spatial tolerance supervised fine-tuning. The supervised fine-tuned visual language model is used as the initial strategy. In the dynamic cropping environment, the bounding box coordinates output by the visual language model are used as the executable action. The visual feedback image is cropped according to the bounding box coordinates, and the cropped visual feedback image is returned to the context of the visual language model. A comprehensive reward is calculated based on the output trajectory of the visual language model, and the comprehensive reward includes a format validity reward, a spatial alignment reward, and an answer correctness reward. The relative group strategy optimization algorithm is used to calculate the relative group advantage of multiple trajectories based on the comprehensive reward, and the strategy of the visual language model is updated according to the relative group advantage.

[0008] In one embodiment, prior to the step of token decoupling the training sequences in the visual language model, the method further includes: Obtain the target sequence of the training samples, wherein the target sequence contains a semantic control token and a spatial coordinate token; A text mask is constructed for the target sequence to mark the position of the semantic control token, and a bounding box mask is constructed for the target sequence to mark the position of the spatial coordinate token; The text mask and the bounding box mask are stored separately.

[0009] In one embodiment, the step of constructing a Gaussian-smoothed soft target probability distribution for the spatial coordinate token based on its true coordinates and a local neighborhood within a preset step size range to calculate the spatial tolerance loss includes: Based on the value distribution of the spatial coordinate token, the preset step size range of the neighborhood is adaptively determined; Using the discrete token position corresponding to the real coordinates as the center, a local neighborhood is determined within the preset step size range; The probability mass is assigned to each adjacent coordinate in the local neighborhood using a Gaussian distribution function centered on the true coordinate, wherein the adjacent coordinates that are closer to the true coordinates receive a higher probability mass. The allocated probability mass is normalized, and the spatial tolerance loss is calculated based on the difference between the normalized soft target probability distribution and the probability distribution predicted by the model.

[0010] In one embodiment, the step of combining the cross-entropy loss and the spatial tolerance loss to perform spatial tolerance supervised fine-tuning of the visual language model includes: Assign a first weighting coefficient to the cross-entropy loss and a second weighting coefficient to the spatial tolerance loss; The total fine-tuning loss is obtained by summing the product of the first weight coefficient and the cross-entropy loss, and the product of the second weight coefficient and the spatial tolerance loss. The parameters of the visual language model are updated using the total fine-tuning loss to perform spatially tolerant supervised fine-tuning.

[0011] In one embodiment, before the step of using the supervised fine-tuned visual language model as the initial strategy and using the bounding box coordinates output by the visual language model as an executable action in a dynamic pruning environment, the method further includes: Initialize the dynamic pruning environment and load the supervised fine-tuned visual language model as the current policy network; The current policy network is interfaced with the dynamic cropping environment, so that the bounding box coordinates output by the policy network can generate cropped images in real time through the image cropping interface of the dynamic cropping environment.

[0012] In one embodiment, after the steps of using the bounding box coordinates output by the visual language model as an executable action in a dynamic cropping environment, cropping the visual feedback image according to the bounding box coordinates, and returning the cropped visual feedback image to the context of the visual language model, the method further includes: The cropped visual feedback image is used as a new visual input and appended to the current context alongside the original image; The context containing the cropped image is re-inputted into the visual language model to generate the next round of inference or cropping action.

[0013] In one embodiment, the step of calculating the comprehensive reward based on the output trajectory of the visual language model includes: Determine the integrity of the bounding box coordinates in the current output trajectory. If the bounding box coordinates are determined to be complete and the final answer text is not empty, assign a format validity reward. The spatial alignment reward is calculated by weighting the generalized intersection-union ratio and recall rate between the bounding box coordinates and the corresponding real bounding box coordinates. The final answer text is matched with the preset standard answer corresponding to the output trajectory, and the answer correctness reward is determined based on the matching result. The comprehensive reward is obtained by weighting and summing the format validity reward, spatial alignment reward, and answer correctness reward according to their respective preset weights.

[0014] In one embodiment, the step of using a group relative policy optimization algorithm to calculate the group relative advantage of multiple trajectories based on the comprehensive reward, and updating the policy of the visual language model according to the group relative advantage, includes: Sample a set of multiple trajectories from the current strategy. Each trajectory contains a series of bounding box actions and corresponding visual feedback and comprehensive reward. The average cumulative comprehensive reward of all trajectories in the group of multiple trajectories is used as the baseline within the group, and the cumulative comprehensive reward of each trajectory is subtracted from the baseline within the group to obtain the relative advantage of the group of multiple trajectories. Based on the relative advantages of the groups, the policy parameters of the visual language model are updated using a policy gradient approach.

[0015] In addition, to achieve the above objectives, this application also proposes a training device for a visual language model, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the training method for the visual language model as described above.

[0016] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the training method for the visual language model as described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: The training sequence in the visual language model is decoupled into tokens, dividing the tokens in the sequence into semantic control tokens and spatial coordinate tokens. Supervised training is performed on the semantic control tokens using cross-entropy loss, while spatial tolerance loss is calculated for the spatial coordinate tokens by constructing a Gaussian-smooth soft target probability distribution based on the real coordinates and local neighborhoods within a preset step size. The cross-entropy loss and spatial tolerance loss are combined to perform supervised fine-tuning of the visual language model. The supervised fine-tuned visual language model is used as the initial policy. In a dynamic cropping environment, the bounding box coordinates output by the visual language model are used as executable actions to crop the visual feedback image based on the bounding box coordinates, and the cropped visual feedback image is returned to the context of the visual language model. A comprehensive reward is calculated based on the output trajectory of the visual language model, including format validity reward, spatial alignment reward, and answer correctness reward. A group relative policy optimization algorithm is used to calculate the group relative advantage of multiple trajectories based on the comprehensive reward, and the policy of the visual language model is updated according to the group relative advantage.

[0018] The technical solution of this application, while ensuring the rigor of semantic reasoning logic, gives coordinate prediction a reasonable spatial tolerance, and achieves adaptive optimization of the pruning strategy by means of closed-loop dynamic visual feedback. It solves the technical problem of the mismatch between hard label penalty distortion and reasoning positioning scale in coordinate supervision, and realizes the technical leap of interactive visual reasoning training from pixel-level strong supervision to spatially tolerant soft supervision, and from static open-loop reasoning to dynamic closed-loop strategy optimization. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the first embodiment of the training method for the visual language model of this application; Figure 2 This is a detailed process diagram based on step S20 in the first embodiment; Figure 3 This is a detailed schematic diagram of step S30 in the first embodiment; Figure 4 This is a detailed process diagram based on step S50 in the first embodiment; Figure 5 This is a detailed schematic diagram of step S60 in the first embodiment; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the training method of the visual language model in the embodiments of this application.

[0022] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0024] In the field of interactive visual reasoning, coordinate localization training mainly follows the technical path of discrete tokenization plus pixel-level cross-entropy supervision, but it has inherent defects and is difficult to meet the dual requirements of pixel accuracy of coordinate prediction and visual semantic tolerance.

[0025] This type of approach typically serializes continuous 2D coordinates into discrete tokens and uses standard token-level cross-entropy loss for supervised training, requiring predicted coordinates to be perfectly consistent with the ground truth labels at the pixel level. In scenarios where image content changes gradually and minor shifts do not affect semantic understanding, this approach can achieve basic localization with a relatively simple supervised paradigm, facilitating model convergence. However, this approach is essentially a hard-label, strong-supervision paradigm that fails to incorporate visual tolerance within the spatial neighborhood into the loss design. When the predicted coordinates deviate from the ground truth labels by only one or two pixels, the cropped image is semantically almost identical to the correct crop, yet the model is penalized as severely as with completely incorrect tokens. This pixel-perfect supervision over-penalizes "near-range errors," ignoring the tolerance range naturally present in visual features within the local space. More seriously, due to the unreliable localization ability of the model, in interactive inference, the model tends to generate larger and noisier cropping boxes as a margin for error, introducing a large amount of background noise, leading to a vicious cycle of scale mismatch between inference and localization.

[0026] A comprehensive analysis reveals that the core dilemma faced by the aforementioned technical approaches lies in the fact that, while the training paradigm of using pixel-level hard labels for coordinate supervision and static open-loop execution of pruning inference is intuitive, its spatial penalty distortion and lack of feedback loops fundamentally contradict the inherent requirements of interactive visual inference for localization tolerance and pruning compactness. It cannot proactively correct minor deviations in coordinate prediction during training, nor can it guide the model to adaptively generate focused visual evidence.

[0027] To address the aforementioned shortcomings, this application proposes a training method for a visual language model used in interactive visual reasoning. This method solves the core pain points through a two-stage framework: In the first stage, sequence tokens are decoupled into semantic control tokens and spatial coordinate tokens. A strict cross-entropy loss is applied to the semantic tokens, while a Gaussian-smooth soft target probability distribution is constructed for the coordinate tokens, centered on the ground coordinates and their local neighborhoods. A spatial tolerance loss is calculated and combined with the cross-entropy loss to complete supervised fine-tuning, enabling the model to learn to tolerate visually harmless small spatial shifts. In the second stage, the fine-tuned model is used as the initial policy. In a dynamic cropping environment, the bounding box coordinates output by the model are used as executable actions. Images are cropped based on these coordinates, and visual feedback is returned to the model context. A comprehensive reward, including format validity, spatial alignment (integrating generalized intersection-union ratio and recall), and answer correctness, is calculated based on the model's output trajectory. A group relative policy optimization algorithm is used to calculate the group relative advantage of multiple trajectories and update the model policy, guiding the model to find the optimal balance between target coverage and background suppression.

[0028] Through the above-mentioned technical means, this application provides reasonable spatial tolerance for coordinate prediction while ensuring the rigor of semantic reasoning logic. It also achieves adaptive optimization of the pruning strategy by using closed-loop dynamic visual feedback, which solves the technical problem of mismatch between hard label penalty distortion and reasoning positioning scale in coordinate supervision. This realizes the technical leap of interactive visual reasoning training from pixel-level strong supervision to spatially tolerant soft supervision, and from static open-loop reasoning to dynamic closed-loop strategy optimization.

[0029] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0030] Based on this, embodiments of this application provide a method for training a visual language model, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the visual language model training method of this application. In this embodiment, the visual language model training method includes steps S10 to S60: Step S10: Decouple the training sequence in the visual language model by dividing the tokens in the sequence into semantic control tokens and spatial coordinate tokens; In interactive visual reasoning tasks, the sequences generated by the model typically contain multiple types of information: one part is semantic content used for logical reasoning, such as the thought chain reasoning steps within the model, the names of the tools invoked, and the final output answer text; the other part is spatial coordinate values ​​used for localization, such as continuous values ​​representing the x and y coordinates of the top-left and bottom-right corners of the bounding box. These two types of tokens differ fundamentally in their geometric properties. The accuracy of semantic control tokens depends on precise word selection and grammatical structure; any tiny error can lead to a complete change in semantics, thus requiring rigorous supervision. Spatial coordinate tokens, on the other hand, are essentially discrete projections of continuous physical space; the cropped images corresponding to adjacent coordinate values ​​are visually almost indistinguishable, therefore a certain degree of spatial offset tolerance should be allowed.

[0031] Based on this understanding, the first step is to identify and classify the type of each token in the training sequence. Specifically, during the data preprocessing stage, for each training sample's target output sequence, the token category is distinguished by parsing the sequence structure. For example, parts of the sequence belonging to thought chain reasoning text, tool call syntax tags, and final answer text are marked as semantic control tokens; parts belonging to bounding box numerical sequences are marked as spatial coordinate tokens. This classification operation relies on predefined sequence format specifications; for example, special start and end identifiers can be used to wrap the coordinate parts, or the token can be distinguished based on its index range in the vocabulary.

[0032] After the partitioning is complete, the two types of tokens are stored in different mask structures to apply different types of supervision signals when calculating the loss function later. Through this decoupling operation, the model can clearly know which tokens need to follow strict semantic consistency and which tokens can accept spatial proximity tolerance in the early stages of training, thus laying the data foundation for subsequent spatial tolerance loss calculation.

[0033] In addition, before the step of decoupling the training sequences in the visual language model using tokens, the method further includes: Obtain the target sequence of the training samples, wherein the target sequence contains a semantic control token and a spatial coordinate token; A text mask is constructed for the target sequence to mark the position of the semantic control token, and a bounding box mask is constructed for the target sequence to mark the position of the spatial coordinate token; The text mask and the bounding box mask are stored separately.

[0034] This process, executed before token decoupling, aims to prepare a structured data foundation for subsequent token partitioning and differentiated supervision. In the training process of interactive visual reasoning models, the raw labeled data typically exists in the form of natural language dialogues or structured instructions, which mix semantic content and spatial coordinate values. To distinguish between semantic control tokens and spatial coordinate tokens, it is necessary to first parse the complete target sequence from the raw labels and create a positional index mask for it.

[0035] In practice, the first step is to obtain the target sequence for each training sample. This target sequence is the ideal answer that the model should output given an image and a user question. It includes the complete thought process, tool call syntax, bounding box coordinates, and the final answer. For example, a typical target sequence might look like this: "Let me analyze this image carefully. First, I need to locate the target object, whose coordinates are [100, 200, 300, 400]. Based on the cropped area, we can determine that the object is a cat. Therefore, the final answer is 'cat'." In this sequence, "Let me analyze this image carefully," "First, I need to locate the target object, whose coordinates are [100, 200, 300, 400]," "Based on the cropped area, we can determine that the object is a cat," and "Therefore, the final answer is 'cat'" are semantic control tokens, while the four values ​​"100," "200," "300," and "400" are spatial coordinate tokens.

[0036] After obtaining the complete target sequence, the next step is to classify and label each token position in the sequence, constructing two mask vectors. The text mask is a Boolean or numeric vector of the same length as the target sequence, used to mark whether each token position belongs to a semantic control category. For the non-numerical part in the example above, the corresponding position in the text mask is set to 1 or True; while for the coordinate numeric part, the corresponding position in the text mask is set to 0 or False. The bounding box mask, conversely, is used to mark whether each token position belongs to the spatial coordinate category, i.e., the coordinate numeric part is set to 1, and the non-numerical part is set to 0. These two masks are mutually exclusive; that is, the same token position cannot belong to both masks simultaneously. However, for data integrity, two independent mask vectors are usually still stored separately.

[0037] The process of constructing a mask relies on predefined rules or a parser. A common approach is to utilize special delimiters within the sequence. For example, during data processing, special markers, such as "", can be inserted before and after the coordinate value sequence.<bbox_start> "and"<bbox_end> In this way, when constructing the mask, the program only needs to identify the token position between these two special markers and mark it as the bounding box mask, while the remaining positions are marked as text masks. Another method is to determine the token based on its index range in the vocabulary. If it is pre-agreed that all coordinate values ​​are mapped to a continuous special interval in the vocabulary, then any token falling within this interval is considered a spatial coordinate token; otherwise, it is a semantic control token.

[0038] After constructing the text mask and bounding box mask, these two masks and their corresponding target sequences are stored separately. The storage format can be independent files or structured data records for quick loading and use in subsequent training iterations. The advantage of separate storage is that when calculating the loss function, the semantic control token positions requiring cross-entropy loss and the spatial coordinate token positions requiring spatial tolerance loss can be quickly selected based on the masks, without needing to re-parse the sequence structure during each forward propagation. This improves training efficiency while ensuring data processing accuracy. Through the above mask construction and storage operations, clear and efficient label information is provided for token decoupling, enabling the model to clearly distinguish the different roles of the two types of tokens during training.

[0039] Step S20: Supervised training is performed on the semantic control token using cross-entropy loss. For the spatial coordinate token, a Gaussian smooth soft target probability distribution is constructed based on the real coordinates and the local neighborhood within a preset step size range to calculate the spatial tolerance loss. For semantic control tokens, the supervision method follows the traditional classification cross-entropy loss. Specifically, for each position of a semantic control token, the model outputs a probability distribution vector, representing the probability that the position belongs to any token in the vocabulary. This probability distribution is compared with the one-hot encoded label of the real token, and the cross-entropy loss is calculated. This loss function applies a full penalty to any prediction that deviates from the real token, ensuring that the semantic content generated by the model is strictly correct.

[0040] A novel soft-target loss calculation logic is introduced for spatial coordinate tokens. Since spatial coordinates have been serialized into discrete tokens, each token corresponds to a discrete numerical range. The token position corresponding to the true coordinate is used as the center point. A local neighborhood within a preset step size is defined, containing the true coordinate token and its neighboring tokens within several step sizes to its left and right. The step size can be set according to the distribution range of the coordinate values ​​and the required spatial tolerance; for example, in a coordinate system normalized to 0 to 1000, the step size can be set to 1 to 5 token units.

[0041] Then, a Gaussian-smooth probability distribution is constructed as the soft target, centered on the real coordinate token location. The peak of the Gaussian function is located at the center, decaying symmetrically to both sides. For each neighboring coordinate token within the local neighborhood, its soft target probability quality is determined by the value of the Gaussian function at that location. Tokens closer to the real coordinate receive higher probability quality, while tokens farther away receive lower probability quality. The soft target probability quality of all tokens within this neighborhood is normalized so that the sum of these probability qualities equals one. The normalized Gaussian distribution represents the reasonable range of deviations allowed by the model when predicting coordinates: predicting a location at the center is optimal, predicting a location nearby is still acceptable, but the reward is reduced; predicting a location far from the neighborhood is severely penalized.

[0042] After constructing the soft target, the cross-entropy between the probability distribution predicted by the model and the Gaussian soft target distribution is used as the spatial tolerance loss for that coordinate token. This loss function allows the model to obtain partial positive gradients at locations close to the true coordinates, thus avoiding gradient oscillations and convergence difficulties caused by hard label penalties.

[0043] Step S30: Combine the cross-entropy loss and the spatial tolerance loss to perform spatial tolerance supervised fine-tuning on the visual language model; The two types of losses calculated in the previous step are organically combined to form a unified total loss function, and the parameters of the visual language model are updated accordingly, completing the spatial tolerance supervised fine-tuning stage. First, weight coefficients are assigned to the cross-entropy loss and the spatial tolerance loss. The setting of the weight coefficients reflects the relative importance placed on semantic accuracy and spatial tolerance during training. Typically, since the number of semantic control tokens may far exceed the number of spatial coordinate tokens, and semantic accuracy is the foundation for interactive inference, a higher weight coefficient, such as 1.0, can be assigned to the cross-entropy loss. However, as an auxiliary soft supervision signal, the weight coefficient of the spatial tolerance loss can be appropriately reduced, for example, to a value between 0.1 and 0.5, to avoid overly relaxing the localization requirements and causing the model to completely ignore precise coordinates.

[0044] The weighted semantic loss component is obtained by summing the product of the first weight coefficient and the cross-entropy loss of all semantic control tokens; the weighted spatial loss component is obtained by summing the product of the second weight coefficient and the spatial tolerance loss of all spatial coordinate tokens. These two loss components are then added together to obtain the total fine-tuning loss. This total fine-tuning loss incorporates both rigorous semantic supervision and tolerant spatial supervision, achieving the training objective of "strict semantics and broad coordinates."

[0045] After obtaining the total fine-tuning loss, the loss value is backpropagated to each network layer of the visual language model. Gradient descent optimization algorithms, such as the AdamW optimizer, are used to update the model's weight parameters in the direction that reduces the total loss. In each iteration, a batch of samples is sampled from the training dataset, and the process of forward propagation, loss calculation, backpropagation, and parameter update is repeated. After multiple rounds of iterative training, the model gradually learns to maintain high accuracy in outputting semantic content while exhibiting reasonable spatial flexibility in outputting coordinates. Compared to traditional pixel-level hard-label supervision, this supervised fine-tuning method makes the model's convergence process smoother and more physically intuitive because the model no longer needs to bear the full penalty for a one-pixel deviation, thus reducing gradient noise during training.

[0046] Step S40: Using the supervised fine-tuned visual language model as the initial strategy, in the dynamic cropping environment, the bounding box coordinates output by the visual language model are used as an executable action to crop the visual feedback image according to the bounding box coordinates, and the cropped visual feedback image is returned to the context of the visual language model. The second phase of training begins by using the spatially tolerant supervised fine-tuning model as the initial strategy for the reinforcement learning phase, establishing an interactive closed loop between the model and the environment. In traditional visual language model training, the coordinates output by the model are only used for static comparison with the labeled parameters; the model is unaware of the subsequent impact of its generated cropping boxes on actual inference. This environment includes an image database and a cropping execution module. When the model outputs a set of bounding box coordinates for the current image, these four values ​​are passed to the cropping execution module as an executable action. After receiving the coordinates, this module extracts the corresponding sub-image regions from the original image according to the coordinates of the top-left and bottom-right corners, thus cropping the image.

[0047] Then, the cropped image is used as new visual feedback, along with the original image and the current dialogue history, and returned to the context of the visual language model. This means the model can see the actual visual result of its cropping action, thus forming an "action-observation" closed loop. During the coordinate-to-action conversion, the original token sequence output by the model needs to be parsed. Since the coordinates output by the model exist in the form of discrete tokens, these tokens are first restored to continuous numerical coordinates. This restoration process is achieved by finding the numerical interval corresponding to the token and taking the midpoint value or the representative value of the interval. The restored continuous coordinates are then fed into the cropping environment.

[0048] Simultaneously, to ensure the stability of environmental interactions, the bounding box coordinates can be validated for reasonableness, such as checking whether the coordinates exceed the limits or whether the cropping box width and height are positive. For invalid coordinate actions, a penalty reward signal can be assigned, and the current trajectory can be terminated. After returning the cropped visual feedback to the model context, the model can continue to generate the next round of inference or cropping actions based on the new visual information. This multi-round interaction mechanism allows the model to dynamically adjust subsequent localization strategies based on the content of the cropped image. For example, if the first cropped area is insufficient, the model can expand the cropping range in the next interaction; if the cropped area contains too much background noise, the model can try to reduce the range. The closed-loop design provides the necessary data foundation for subsequent reinforcement learning reward calculation and policy optimization.

[0049] In addition, before the step of using the supervised fine-tuned visual language model as the initial strategy and using the bounding box coordinates output by the visual language model as an executable action in the dynamic pruning environment, the method further includes: Initialize the dynamic pruning environment and load the supervised fine-tuned visual language model as the current policy network; The current policy network is interfaced with the dynamic cropping environment, so that the bounding box coordinates output by the policy network can generate cropped images in real time through the image cropping interface of the dynamic cropping environment.

[0050] The core objective of this embodiment is to construct a complete reinforcement learning interactive environment and correctly connect the model with the environment. In traditional visual language model training, the model outputs coordinates and directly compares them with static labels, without an interactive environment. To implement dynamic visual feedback training, a pruned environment is explicitly initialized, and the fine-tuned model is deployed as the agent's policy network into this environment.

[0051] First, the dynamic cropping environment is initialized. This environment consists of three core components: an image database, a cropping execution module, and a state management unit. The image database stores all the original training images, each with a unique identifier for quick indexing during interaction. The cropping execution module is a function or service that receives bounding box coordinate parameters and outputs sub-images; its internal implementation relies on standard image processing libraries. The state management unit is responsible for maintaining information about the current interaction round, such as the currently processed image identifier, the cropping history, and the current dialogue context. During initialization, basic environment parameters need to be set, such as the image loading path, cropping output format resolution, and whether to perform normalization preprocessing on the cropping results. Simultaneously, the environment's interaction protocol needs to be defined, i.e., how the model's output actions are parsed and how the cropped feedback is organized into a model-readable format.

[0052] After environment initialization, the spatially tolerant supervised fine-tuning visual language model is loaded as the current policy network. The loading process involves reading the weight file saved after model training and restoring these weights to the same model architecture. To adapt to the training requirements of reinforcement learning, the model needs to be set to inference mode to generate actions, while retaining gradient computation capabilities for subsequent policy updates. The core function of this policy network is to receive the current observation state—the original image, the cropping history, and the text encoding of the user's question—and then output an action probability distribution from which specific bounding box coordinate sequences are sampled. Since the model outputs a discrete token sequence, the mapping relationship between coordinate tokens and numerical ranges must be maintained during loading to ensure the correctness of action parsing.

[0053] Finally, the current policy network is integrated with the dynamic cropping environment. The core of this integration is establishing a data path from the policy network's output to the cropping environment's input. Specifically, an interaction adapter is first written to extract bounding box coordinate actions from the policy network's output. After the model generates a complete output sequence, the adapter parses four coordinate values ​​according to predefined format rules and validates their validity, such as checking if the coordinates are within the image size range and if the cropping box's width and height are positive. After successful validation, the adapter uses these coordinate values ​​as parameters to call the image cropping interface of the cropping execution module. The cropping interface extracts the corresponding original image region from the image database based on the coordinates and generates a cropped sub-image in real time. This sub-image, as visual feedback, along with the current dialogue history and original image information, is encapsulated into the next round's observation state and returned to the policy network. Through this integration, the model no longer outputs coordinates in isolation but forms a closed-loop interaction with the environment. Each bounding box action output by the model immediately produces a visual cropping result, which in turn influences the model's subsequent decisions.

[0054] Furthermore, after the step of using the bounding box coordinates output by the visual language model as an executable action in the dynamic cropping environment, cropping the visual feedback image according to the bounding box coordinates, and returning the cropped visual feedback image to the context of the visual language model, the method further includes: The cropped visual feedback image is used as a new visual input and appended to the current context alongside the original image; The context containing the cropped image is re-inputted into the visual language model to generate the next round of inference or cropping action.

[0055] The core objective of this embodiment is to achieve memory maintenance and state updates in multi-round interactive reasoning, enabling the model to continuously reason or further adjust its positioning based on the already cropped image content. In traditional methods, the model typically outputs the final answer after only one cropping operation, unable to correct its actions based on visual feedback. The above steps extend the number of interaction rounds, giving the model the ability to make continuous decisions.

[0056] First, the cropped image is processed as new visual input. The cropped image is a sub-image whose size and content depend on the bounding box coordinates output by the model. To preserve global information from the original image while introducing local details, the above steps process the cropped image alongside the original image, rather than replacing the original. This juxtaposition can be achieved by placing the original and cropped images in two different visual channels, or by embedding the cropped image as a thumbnail at the end of the dialogue history. In a concrete implementation, the above steps encode the cropped image into a feature vector or tensor in the same format as the original image, and then append it to the current context sequence in chronological order. The context is a structured data structure containing the user's question, system prompts, previously output text from the model, and all visual observations already generated. Each append operation generates a new version of the context, which retains the complete original scene information while adding visual evidence of the local region focused on by the model's most recent actions.

[0057] After image appending is complete, the above steps re-input the complete context of the cropped image into the visual language model. This means the model no longer simply reasones step-by-step from the original image, but can utilize the visual feedback generated by its previous cropping action as new clues. The re-input context includes the original image for global reference and the cropped image for detailed analysis. The model can comprehensively judge whether the current cropping is appropriate based on these two types of visual information. For example, if the cropped image contains the target subject but the background is noisy, the model can choose to output a new, more compact bounding box to refine the localization; if the target is missing in the cropped image, the model can choose to expand the cropping area and try again; if the cropped image already contains enough features to identify the target, the model can directly output the final answer.

[0058] The above steps also support multi-round continuous interaction. After one round of cropping and feedback, the model outputs a new bounding box action or answer based on the new context. Then, the above steps append the newly cropped image to the context and re-input it to the model. This loop can be repeated multiple times until the model outputs a termination symbol or reaches the preset maximum number of rounds. Through this mechanism, the model can continuously calibrate its localization strategy during interaction, gradually approaching the optimal cropping region. The model is no longer afraid of insufficient accuracy in a single localization, as subsequent rounds provide opportunities for correction. The context appending and re-input mechanism implemented in the above steps is a key connecting link in the dynamic visual feedback loop. It expands a single action output into a sequential decision-making process, enabling the model to find the optimal balance between target coverage and background suppression through multiple rounds of exploration, ultimately generating compact, focused, and low-noise visual evidence for high-quality inference.

[0059] Step S50: Calculate a comprehensive reward based on the output trajectory of the visual language model. The comprehensive reward includes a format validity reward, a spatial alignment reward, and an answer correctness reward. A quantitative evaluation is performed on a complete output trajectory generated by the model in a dynamic cropping environment, fusing multiple metrics into a comprehensive reward value. This comprehensive reward is the core input to the subsequent group relative policy optimization algorithm. A trajectory typically consists of multiple interaction rounds, each round including the bounding box coordinates of the model output, the corresponding cropped image, and the final answer text. The aforementioned comprehensive reward value constructs the reward signal from three dimensions.

[0060] The first dimension is the format validity reward. It checks whether the bounding box coordinates output by the model in the current trajectory are complete, i.e., whether they contain all four numerical components required to form a bounding box, and also checks whether the final answer text is not empty. If both conditions are met, a positive reward value is given, such as adding one point; otherwise, a negative reward value is given, such as subtracting one point, or the trajectory is terminated directly. The format validity reward ensures that the model learns to output complete actions that conform to the interaction specifications.

[0061] The second dimension is the spatial alignment reward. The coordinates of the ground truth bounding boxes corresponding to the current trajectory are obtained, and the generalized intersection-union (GUU) between the model's output bounding box and the ground truth bounding box is calculated. GUU not only evaluates the overlap between the predicted and ground truth bounding boxes but also penalizes predicted boxes that, while containing the target, include a large amount of redundant background area; that is, a lower GUU value indicates more background noise. Simultaneously, recall is calculated, which is the area of ​​the intersection between the predicted and ground truth bounding boxes divided by the area of ​​the ground truth bounding box. This metric measures whether the predicted box completely covers the target. The spatial alignment reward is obtained by weighting and summing the GUU and recall according to preset weights.

[0062] The third dimension is the reward for correct answer. The final answer text output by the model is matched with the preset standard answer corresponding to the current trajectory. The matching method can be exact string comparison or semantic similarity calculation. If the match is successful or the similarity exceeds a preset threshold, a positive reward value is given; otherwise, a negative reward value or zero reward value is given.

[0063] Finally, the format validity reward, spatial alignment reward, and answer correctness reward are weighted and summed according to their respective preset weight coefficients to obtain the comprehensive reward for the trajectory. This comprehensive reward reflects both the quality of the model's pruning action and the accuracy of the final inference, forming a multi-objective comprehensive evaluation of the model's behavior.

[0064] Step S60: Using the group relative strategy optimization algorithm, calculate the group relative advantage of multiple trajectories based on the comprehensive reward, and update the strategy of the visual language model according to the group relative advantage.

[0065] This embodiment performs policy optimization, employing a group-relative policy optimization algorithm to update the policy parameters of the visual language model based on the comprehensive reward calculated in previous steps. Traditional policy gradient methods typically use absolute reward values ​​as the basis for advantage estimation, which is easily affected by fluctuations in reward scale. The group-relative policy optimization algorithm eliminates the scale sensitivity problem of absolute reward values ​​by performing relative comparisons within a sampling group, making training more stable and efficient.

[0066] First, a set of multiple trajectories is sampled from the current policy network, i.e., the visual language model after spatial tolerance fine-tuning. During the generation of each trajectory, an interaction loop is followed, and a comprehensive reward value is calculated for that trajectory. The cumulative comprehensive rewards of all trajectories in this set are summed, and the average of these cumulative comprehensive rewards is calculated. This average value is used as the intra-group baseline. This baseline represents the average performance level of the model in a dynamic pruning environment under the current policy.

[0067] Then, the cumulative composite reward of each trajectory within each group is subtracted from the intra-group baseline to obtain the group relative advantage of that trajectory. If the cumulative composite reward of a trajectory is higher than the intra-group baseline, its group relative advantage is positive, indicating that the trajectory performs better than the average level; conversely, if it is lower than the intra-group baseline, the group relative advantage is negative, indicating that the trajectory performs worse than the average level.

[0068] Furthermore, the relative advantages of these groups are used as weights to update the policy network. Specifically, a policy gradient approach is employed to calculate the gradient direction that increases the generation probability of high-advantage trajectories and decreases the generation probability of low-advantage trajectories. For each action in a high-advantage trajectory, its output probability in that state is increased; for each action in a low-advantage trajectory, its output probability is decreased. Simultaneously, to prevent excessively large policy updates from causing training failure...

[0069] By repeatedly performing the above sampling, baseline calculation, dominance estimation, and policy gradient update on multiple sets of trajectories, the policy of the visual language model is continuously improved. The model gradually learns to generate compact clipping boxes that can effectively cover the target while suppressing background noise, and can accurately answer inference questions based on the visual feedback after clipping. This fundamentally solves the scale mismatch problem of excessively large clipping boxes caused by unreliable localization in traditional methods.

[0070] Furthermore, you can also view Figure 2 , Figure 2 This is a detailed process diagram based on step S20 in the first embodiment. Figure 2 The step of constructing a Gaussian-smoothed soft target probability distribution and calculating the spatial tolerance loss based on the real coordinates of the spatial coordinate token and the local neighborhood within a preset step size range includes S21~S24: Step S21: Based on the value distribution of the spatial coordinate token, adaptively determine the preset step size range of the neighborhood; Step S22: Using the discrete token position corresponding to the real coordinates as the center, determine the local neighborhood within the preset step size range; Step S23: Assign probability mass to each adjacent coordinate in the local neighborhood using a Gaussian distribution function centered on the true coordinate, wherein the adjacent coordinates that are closer to the true coordinates receive a higher probability mass. Step S24: Normalize the allocated probability mass, and calculate the spatial tolerance loss based on the difference between the normalized soft target probability distribution and the probability distribution predicted by the model.

[0071] This embodiment refines the spatial tolerance loss calculation process, forming a complete soft target loss construction and solution process from adaptive step size determination, local neighborhood construction, Gaussian probability mass allocation to normalized loss calculation.

[0072] First, the preset step size range of the neighborhood is adaptively determined based on the value distribution of the spatial coordinate tokens. Spatial coordinate tokens are typically discretized into a fixed number of numerical intervals, for example, mapping continuous coordinates from 0 to 1000 to 1001 discrete tokens. The distribution range of coordinates may differ in different training datasets, and the sparsity of coordinates for different targets within the same dataset may also vary. To ensure the reasonableness of the spatial tolerance, the neighborhood step size needs to be dynamically adjusted according to the statistical characteristics of the actual coordinate distribution. Specifically, the true coordinate values ​​corresponding to all spatial coordinate tokens in the current training batch are collected, and the dispersion of their distribution is calculated, such as variance or interquartile range. If the coordinate distribution is relatively concentrated, i.e., the target size is small or the range of variation is narrow, the neighborhood step size can be set smaller, such as 1 to 3 token units, to avoid including regions with different semantic meanings in the tolerance range. If the coordinate distribution is relatively dispersed, i.e., the target size is large or the range of variation is wide, the neighborhood step size can be appropriately increased, such as 3 to 7 token units, to allow for more meaningful spatial offsets. This adaptive mechanism avoids the overfitting or underfitting problems caused by artificially fixing the step size.

[0073] Using the discrete token positions corresponding to the real coordinates as the center, a local neighborhood is determined within the aforementioned preset step size. Once the step size is determined, for example, k, the local neighborhood consists of k tokens to the left and right of the real coordinate token position, for a total of 2k+1 token positions. For boundary cases on the coordinate axes, such as when the real coordinates are close to the minimum or maximum value of the coordinate range, the neighborhood may exceed the effective token index range. In this case, a pruning process is performed, retaining only tokens within the effective index range as the neighborhood, and renormalizing the probability quality within the effective neighborhood.

[0074] For each neighboring coordinate within the aforementioned local neighborhood, a probability mass is assigned using a Gaussian distribution function centered on the true coordinates. The expression of the Gaussian distribution function determines the rate at which the probability mass decays with the offset from the center. First, a standard deviation parameter is set, which can be related to the step size range; for example, the standard deviation is equal to one-third of the step size range. Then, for each offset d within the neighborhood, the corresponding Gaussian function value is calculated. The closer to the true coordinates, the smaller d, the larger the Gaussian function value, and the higher the assigned probability mass; the farther away, the lower the probability mass. The soft target distribution constructed in this way exhibits unimodal, smooth, and symmetrical characteristics, conforming to the visual semantic intuition of "the closer, the more acceptable."

[0075] The assigned probability quality is normalized so that the sum of the soft target probability quality of all tokens in the local neighborhood equals one. Normalization is achieved by dividing each Gaussian function value by the sum of all Gaussian function values ​​in the neighborhood. This normalized result is a discretized, truncated Gaussian distribution of the soft target probability. The cross-entropy difference between the model's predicted probability distribution and this normalized soft target distribution is then calculated as the spatial tolerance loss for that coordinate token. Specifically, for each token in the neighborhood, the logarithm of the model's predicted probability is multiplied by the soft target probability, then summed and negative. This loss function ensures that during training, the model generates positive gradients not only for the ground truth coordinate tokens but also for neighboring coordinate tokens, with the gradient magnitude proportional to the Gaussian weights. This allows the model to learn that predicting at the ground truth coordinates yields the maximum reward, predicting at neighboring coordinates yields a smaller reward without penalty, and predicting at coordinates far from the neighborhood incurs a severe penalty. This soft target supervision method fundamentally solves the problem of severe penalties for small offsets in hard-label supervision.

[0076] Furthermore, you can also view Figure 3 , Figure 3 This is a detailed process diagram based on step S30 in the first embodiment. Figure 3 The step of combining the cross-entropy loss and the spatial tolerance loss to perform spatial tolerance supervised fine-tuning of the visual language model includes S31~S33: Step S31: Assign a first weighting coefficient to the cross-entropy loss and a second weighting coefficient to the spatial tolerance loss. Step S32: Summing the product of the first weight coefficient and the cross-entropy loss, and the product of the second weight coefficient and the spatial tolerance loss, yields the total fine-tuning loss; Step S33: Update the parameters of the visual language model using the total fine-tuning loss to perform spatially tolerant supervised fine-tuning.

[0077] This embodiment provides a detailed breakdown of the combined loss and model fine-tuning process, covering three stages: weight coefficient allocation, total loss calculation, and parameter update. This constitutes the core execution flow of space-tolerant supervised fine-tuning.

[0078] A first weight coefficient is assigned to the cross-entropy loss, and a second weight coefficient is assigned to the spatial tolerance loss. Since the proportions of semantic control tokens and spatial coordinate tokens in a sequence typically differ significantly, and semantic accuracy has higher priority in interactive inference, directly adding them might lead to one type of loss dominating the training process. Therefore, weight coefficients are needed to balance the influence of the two types of losses. The first weight coefficient is assigned to the cross-entropy loss, and this coefficient is usually set relatively large, for example, between 0.8 and 1.2, with a typical value of 1.0. The second weight coefficient is assigned to the spatial tolerance loss, and this coefficient is set relatively small, for example, between 0.1 and 0.5, with a typical value of 0.2. The specific values ​​of the weight coefficients can be optimized based on performance on the validation set. If the model's coordinate localization is too divergent, the second weight can be increased appropriately; if the model's semantic output is incorrect, the first weight can be increased appropriately. Additionally, the weights can be dynamically adjusted for each sample, for example, adaptively adjusting the second weight coefficient based on the number of coordinate tokens or the size of the target object in the current sample.

[0079] The total fine-tuning loss is obtained by summing the products of the first weight coefficient and the cross-entropy loss, and the second weight coefficient and the spatial tolerance loss. The cross-entropy loss is the sum or average of the cross-entropy losses at all semantic control token positions, while the spatial tolerance loss is the sum or average of the spatial tolerance losses at all spatial coordinate token positions. First, for each token position in the model output sequence, it is determined whether it belongs to a semantic control token or a spatial coordinate token based on the constructed mask. For semantic control tokens, the cross-entropy loss is calculated and accumulated; for spatial coordinate tokens, the spatial tolerance loss is calculated and accumulated. Then, the two types of losses are multiplied by their corresponding weight coefficients and summed to obtain the total loss. This total loss function implements differentiated supervision for the two types of tokens: semantic tokens are required to strictly match the ground truth label, while coordinate tokens are allowed to receive partial rewards within their neighborhood.

[0080] The parameters of the visual language model are updated using the total fine-tuning loss described above to achieve spatially tolerant supervised fine-tuning. In practice, the total loss is first backpropagated to calculate the gradient of the loss with respect to each trainable parameter of the model. Then, an optimizer such as AdamW or SGD is used to update the parameters based on a preset learning rate and gradient information. The update process adjusts the model weights in the direction that reduces the total loss. Parameter updates are performed once after processing each batch of data. Over multiple training epochs, the model repeatedly observes a large number of samples, simultaneously optimizing both semantic output accuracy and coordinate output tolerance. Notably, parameter updates not only adjust the output probability distribution of semantic control tokens but also the representational power of spatial coordinate tokens. Because the soft-target loss of coordinate tokens guides the model to generate positive gradients on neighboring tokens, the model's coordinate prediction ability becomes smoother. Small perturbations in the prediction results in continuous space do not cause drastic changes in loss, making the model more robust to pixel-level noise in actual inference.

[0081] Furthermore, you can also view Figure 4 , Figure 4 This is a detailed process diagram based on step S50 in the first embodiment. Figure 4 The step of calculating the comprehensive reward based on the output trajectory of the visual language model includes S51~S54: Step S51: Determine the integrity of the bounding box coordinates in the current output trajectory. When the bounding box coordinates are determined to be complete and the final answer text is not empty, allocate a format validity reward. Step S52: Based on the generalized intersection-union ratio and recall rate between the bounding box coordinates and the corresponding real bounding box coordinates, the spatial alignment reward is calculated using a weighted average. Step S53: Match the final answer text with the preset standard answer corresponding to the output trajectory, and determine the answer correctness reward based on the matching result; Step S54: The format validity reward, spatial alignment reward, and answer correctness reward are weighted and summed according to their respective preset weights to obtain the comprehensive reward.

[0082] This embodiment represents the core process of comprehensive reward calculation, forming a complete trajectory evaluation system from format checking, spatial alignment assessment, answer correctness judgment to final weighted fusion.

[0083] First, the integrity of the bounding box coordinates in the current output trajectory is determined, and a format validity reward is assigned when the bounding box coordinates are complete and the final answer text is not empty. In the dynamic pruning environment of interactive visual reasoning, the output sequence generated by the model may exhibit format abnormalities due to decoding errors or model instability, such as outputting only three coordinate values ​​or an empty string in the final answer section. Such incomplete output cannot be correctly parsed by the pruning execution module, therefore a negative reward is needed to guide the model to learn standardized output.

[0084] In practice, the complete text sequence output by the model in the final round is obtained from the current trajectory, and regular expressions are used to match the bounding box pattern. A standard bounding box coordinate sequence should contain four numerical components, representing the x and y coordinates of the top-left and bottom-right corners, respectively. The system sequentially checks whether four valid numerical components exist, whether each component falls within a preset coordinate range, and whether the four components form a non-degenerate rectangle. It also checks whether the final answer text, after removing whitespace, is non-empty. When all the above conditions are met, a positive reward value is assigned, such as adding 1.0 point; otherwise, a negative reward value is assigned, such as subtracting 1.0 point. This reward ensures that the actions and answers output by the model are structurally parsable.

[0085] The spatial alignment reward is calculated by weighting the generalized intersection-union ratio (GURR) and recall between the bounding box coordinates and the corresponding ground truth bounding box coordinates. The ground truth bounding box coordinates can be obtained from the training samples. Traditional localization evaluation only uses GURR, but it cannot distinguish between cases where the predicted box is too large, resulting in a lot of background noise but the target is completely covered, and cases where the predicted box is too small, resulting in partial target loss. This paper introduces two complementary metrics: GURR and recall. GURR adds a penalty term to the standard GURR, which is between -1 and 1, penalizing the difference in the area of ​​the minimum bounding rectangle between the predicted and ground truth bounding boxes. This penalizes predicted boxes containing a large amount of irrelevant background. Recall is defined as the area of ​​the intersection of the predicted and ground truth bounding boxes divided by the area of ​​the ground truth bounding box, ranging from 0 to 1, measuring the degree of coverage of the target subject. Using recall alone can lead to the model generating large cropped boxes that cover the entire image. The spatial alignment reward is obtained by weighting and summing GURR and recall with preset weights, such as 0.6 and 0.4. The reward-guided model seeks the optimal clipping size, such that the clipping box exactly surrounds the target and is as compact as possible.

[0086] The final answer text is matched against the preset standard answer corresponding to the output trajectory. A reward is determined based on the matching result, and the preset standard answer must be obtained from the training samples. Even if the cropping action performs perfectly in spatial alignment, the entire interaction is still considered a failure if the final semantic inference result is incorrect. First, the final answer text output by the model is extracted from the current trajectory, and the preset standard answer corresponding to that trajectory is also obtained. Matching methods can include exact string matching, which involves normalizing the two strings (e.g., converting to lowercase, removing punctuation), and then comparing them character by character for equality. Alternatively, semantic similarity matching can be used, where a pre-trained text embedding model encodes the answer as a semantic vector, and the cosine similarity is calculated. If the similarity exceeds a preset threshold (e.g., 0.85), the match is considered successful. Based on the matching result, a positive reward (e.g., adding 1.0 point) is given for success, and a negative reward or zero reward is given for failure. This reward system enables the model not only to learn cropping but also to learn accurate inference based on the cropped image.

[0087] The format validity reward, spatial alignment reward, and answer correctness reward are weighted and summed according to their respective preset weights to obtain the comprehensive reward. First, the reward values ​​calculated in the first three steps are obtained. These three reward values ​​may have different ranges, and directly adding them could lead to one dimension dominating the result. Therefore, a preset weight coefficient is assigned to each reward; for example, the format validity reward has a weight of 0.2, the spatial alignment reward has a weight of 0.3, and the answer correctness reward has a weight of 0.5. The weight coefficients can be flexibly adjusted according to task requirements. If more emphasis is placed on positioning accuracy, the weight of the spatial alignment reward is increased; if more emphasis is placed on the final answer, the weight of the answer correctness reward is increased. Each reward is multiplied by its corresponding weight and then summed to obtain a scalar comprehensive reward value. This comprehensive reward serves as a single indicator for evaluating the quality of a trajectory in subsequent relative policy optimization algorithms. It integrates information from three dimensions: format regularity, spatial positioning quality, and semantic correctness, enabling the model to simultaneously consider multiple objectives during policy optimization.

[0088] Furthermore, you can also view Figure 5 , Figure 5 This is a detailed process diagram based on step S60 in the first embodiment. Figure 5 The step of using the group relative policy optimization algorithm to calculate the group relative advantage of multiple trajectories based on the comprehensive reward, and updating the policy of the visual language model according to the group relative advantage, includes S61~S63: Step S61: Sample a set of multiple trajectories from the current policy. Each trajectory contains a series of bounding box actions and corresponding visual feedback and comprehensive reward. Step S62: Take the average of the cumulative comprehensive rewards of all trajectories in the group of multiple trajectories as the baseline within the group, and subtract the baseline within the group from the cumulative comprehensive reward of each trajectory to obtain the relative advantage of the group of multiple trajectories. Step S63: Based on the relative advantage of the group, update the policy parameters of the visual language model using a policy gradient approach.

[0089] This embodiment outlines the core process of updating the model policy using a group relative policy optimization algorithm. From trajectory sampling, intra-group baseline calculation and relative advantage estimation to policy parameter updating, it forms a complete reinforcement learning optimization closed loop.

[0090] A set of multiple trajectories is sampled from the current policy. Each trajectory contains a series of bounding box actions, corresponding visual feedback, and a comprehensive reward. In group relative policy optimization, absolute reward values ​​are not used; instead, relative comparisons within the group are used to eliminate the influence of fluctuations in the reward scale. Step S61 first uses the spatially tolerant supervised fine-tuning visual language model as the current policy network. Then, under this policy, multiple trajectories are generated according to the interaction loop. The generation process of each trajectory includes: the model outputs bounding box coordinates as executable actions; the dynamic cropping environment crops the image according to the coordinates and returns visual feedback; the model continues to output the next round of actions or answers based on the new context; and finally, the comprehensive reward of the trajectory is calculated. A set of trajectories is sampled at a time, and the group size is usually set to 4 to 16. This set of trajectories represents multiple behavioral results of the current policy under the same or similar initial conditions, providing a sample basis for subsequent relative comparisons. The sampling process can be accelerated using a parallel environment, with multiple trajectories generated and the comprehensive reward collected simultaneously.

[0091] The group relative advantage is obtained by taking the average of the cumulative composite rewards of all trajectories in a set of multiple trajectories as the within-group baseline, and then subtracting the within-group baseline from the cumulative composite reward of each trajectory. Traditional policy gradient methods use the absolute value of cumulative rewards as the advantage function, which is easily affected by changes in reward scaling. First, the cumulative composite reward of each trajectory within the group is calculated, which is the sum of all immediate composite rewards obtained from the start to the end of the trajectory. Then, the arithmetic mean of the cumulative composite rewards of all trajectories in this group is calculated, and this mean is the within-group baseline. The within-group baseline represents the average performance level of the current policy on this group of trajectories. Next, the group relative advantage of each trajectory is obtained by subtracting the within-group baseline from the cumulative composite reward of that trajectory. If the cumulative composite reward of a trajectory is higher than the within-group baseline, its group relative advantage is positive, indicating that the trajectory performs better than the average level; conversely, the group relative advantage is negative, indicating that the performance is worse than the average level. This method of estimating relative advantage naturally eliminates the influence of the magnitude of the absolute value of the reward, making policy updates more stable and reducing variance.

[0092] Based on the relative advantage of the groups, the policy parameters of the visual language model are updated using a policy gradient approach. The calculated relative advantage of the groups is used as weights to adjust the parameters of the policy network. Specifically, using an approximate form of the policy gradient theorem, for trajectories with a positive relative advantage within a group, the probability of each action in that trajectory being selected in the current state is increased; for trajectories with a negative relative advantage, the probability of their actions is decreased.

[0093] In practice, an objective function is constructed, which equals the sum of the log probabilities of all trajectories within a group multiplied by the relative group advantages of the corresponding trajectories. The gradient of this objective function is then calculated, and the policy network parameters are updated along the gradient's ascending direction. To prevent policy collapse due to excessively large single updates, a pruning mechanism can be introduced to limit the probability ratio between the old and new policies to a preset range, such as 0.8 to 1.2. Simultaneously, an entropy regularization term can be used to encourage the policy to maintain a certain degree of exploratory activity. Through the process of updating policy parameters across multiple groups of trajectories, the visual language model's policy is continuously iterated and improved. The model gradually learns to generate compact clipping boxes that effectively cover the target while suppressing background noise, and can accurately answer inference questions based on the visual feedback after pruning. This fundamentally solves the scale mismatch problem of excessively large clipping boxes caused by unreliable localization in traditional methods.

[0094] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the training method of the visual language model of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0095] This application provides a training device for a visual language model, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the visual language model training method in Embodiment 1 above.

[0096] The following is for reference. Figure 6 The diagram illustrates a structural schematic of a training device suitable for implementing the visual language model in the embodiments of this application. The training device for the visual language model in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The training device for the visual language model shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0097] like Figure 6 As shown, the training device for the visual language model may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the visual language model training device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the training device for the visual language model to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows training devices with various visual language models, it should be understood that implementation or possession of all of them is not required. More or fewer models may be implemented alternatively.

[0098] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0099] The visual language model training device provided in this application, employing the visual language model training method described in the above embodiments, can solve the technical problem of mismatch between the coordinate prediction supervision signal and the visual semantic tolerance of the visual language model. Compared with the prior art, the beneficial effects of the visual language model training device provided in this application are the same as those of the visual language model training method provided in the above embodiments, and other technical features in this visual language model training device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0100] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0101] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0102] This application provides a storage medium, which is a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the training method of the visual language model in the above embodiments.

[0103] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be executed by instructions, used by devices, or used in conjunction with them. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0104] The aforementioned computer-readable storage medium may be included in the training device for the visual language model; or it may exist independently and not be incorporated into the training device for the visual language model.

[0105] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by a visual language model training device, enable the visual language model training device to implement the technical content of the above-described visual language model training method embodiment.

[0106] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using dedicated hardware-based implementations that perform the specified functions or operations, or can be implemented using a combination of dedicated hardware and computer instructions.

[0108] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0109] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the training method of the above-described visual language model, which can solve the technical problem of mismatch between the coordinate prediction supervision signal and the visual semantic tolerance of the visual language model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the training method of the visual language model provided in the above embodiments, and will not be repeated here.

Claims

1. A training method for a visual language model, characterized in that, The training method for the visual language model includes the following steps: Decouple the training sequences in the visual language model by tokens, and divide the tokens in the sequence into semantic control tokens and spatial coordinate tokens; The semantic control tokens are trained under supervision using cross-entropy loss. The spatial tolerance loss is calculated by constructing a Gaussian smooth soft target probability distribution for the spatial coordinate tokens based on the real coordinates and the local neighborhood within a preset step size. By combining the cross-entropy loss and the spatial tolerance loss, the visual language model is subjected to spatial tolerance supervised fine-tuning. The supervised fine-tuned visual language model is used as the initial strategy. In the dynamic cropping environment, the bounding box coordinates output by the visual language model are used as the executable action. The visual feedback image is cropped according to the bounding box coordinates, and the cropped visual feedback image is returned to the context of the visual language model. A comprehensive reward is calculated based on the output trajectory of the visual language model, and the comprehensive reward includes a format validity reward, a spatial alignment reward, and an answer correctness reward. The relative group strategy optimization algorithm is used to calculate the relative group advantage of multiple trajectories based on the comprehensive reward, and the strategy of the visual language model is updated according to the relative group advantage.

2. The training method for the visual language model as described in claim 1, characterized in that, Before the step of decoupling the training sequences in the visual language model using tokens, the method further includes: Obtain the target sequence of the training samples, wherein the target sequence contains a semantic control token and a spatial coordinate token; A text mask is constructed for the target sequence to mark the position of the semantic control token, and a bounding box mask is constructed for the target sequence to mark the position of the spatial coordinate token; The text mask and the bounding box mask are stored separately.

3. The training method for the visual language model as described in claim 1, characterized in that, The step of constructing a Gaussian-smoothed soft target probability distribution and calculating the spatial tolerance loss based on the spatial coordinate token's real coordinates and a local neighborhood within a preset step size range includes: Based on the value distribution of the spatial coordinate token, the preset step size range of the neighborhood is adaptively determined; Using the discrete token position corresponding to the real coordinates as the center, a local neighborhood is determined within the preset step size range; The probability mass is assigned to each adjacent coordinate in the local neighborhood using a Gaussian distribution function centered on the true coordinate, wherein the adjacent coordinates that are closer to the true coordinates receive a higher probability mass. The allocated probability mass is normalized, and the spatial tolerance loss is calculated based on the difference between the normalized soft target probability distribution and the probability distribution predicted by the model.

4. The training method for the visual language model as described in claim 1, characterized in that, The step of combining the cross-entropy loss and the spatial tolerance loss to perform spatial tolerance supervised fine-tuning of the visual language model includes: Assign a first weighting coefficient to the cross-entropy loss and a second weighting coefficient to the spatial tolerance loss; The total fine-tuning loss is obtained by summing the product of the first weight coefficient and the cross-entropy loss, and the product of the second weight coefficient and the spatial tolerance loss. The parameters of the visual language model are updated using the total fine-tuning loss to perform spatially tolerant supervised fine-tuning.

5. The training method for the visual language model as described in claim 1, characterized in that, Before the step of using the supervised fine-tuned visual language model as the initial strategy and the bounding box coordinates output by the visual language model as the executable action in the dynamic pruning environment, the method further includes: Initialize the dynamic pruning environment and load the supervised fine-tuned visual language model as the current policy network; The current policy network is interfaced with the dynamic cropping environment, so that the bounding box coordinates output by the policy network can generate cropped images in real time through the image cropping interface of the dynamic cropping environment.

6. The training method for the visual language model as described in claim 1, characterized in that, After the step of using the bounding box coordinates output by the visual language model as an executable action in a dynamic cropping environment, cropping the visual feedback image according to the bounding box coordinates, and returning the cropped visual feedback image to the context of the visual language model, the method further includes: The cropped visual feedback image is used as a new visual input and appended to the current context alongside the original image; The context containing the visual feedback image is re-inputted into the visual language model to generate the next round of inference or cropping actions.

7. The training method for the visual language model as described in claim 1, characterized in that, The step of calculating the comprehensive reward based on the output trajectory of the visual language model includes: Determine the integrity of the bounding box coordinates in the current output trajectory. If the bounding box coordinates are determined to be complete and the final answer text is not empty, assign a format validity reward. The spatial alignment reward is calculated by weighting the generalized intersection-union ratio and recall rate between the bounding box coordinates and the corresponding real bounding box coordinates. The final answer text is matched with the preset standard answer corresponding to the output trajectory, and the answer correctness reward is determined based on the matching result. The comprehensive reward is obtained by weighting and summing the format validity reward, spatial alignment reward, and answer correctness reward according to their respective preset weights.

8. The training method for the visual language model as described in claim 1, characterized in that, The step of using the group relative policy optimization algorithm to calculate the group relative advantage of multiple trajectories based on the comprehensive reward, and updating the policy of the visual language model according to the group relative advantage, includes: Sample a set of multiple trajectories from the current strategy. Each trajectory contains a series of bounding box actions and corresponding visual feedback and comprehensive reward. The average cumulative comprehensive reward of all trajectories in the group of multiple trajectories is used as the baseline within the group, and the cumulative comprehensive reward of each trajectory is subtracted from the baseline within the group to obtain the relative advantage of the group of multiple trajectories. Based on the relative advantages of the groups, the policy parameters of the visual language model are updated using a policy gradient approach.

9. A training device for a visual language model, characterized in that, The training device for the visual language model stores a computer program, which, when executed by a processor, implements the training method for the visual language model according to any one of claims 1-8.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the training method for the visual language model according to any one of claims 1-8.

Images