Method and device for enhancing fine-grained perception of visual language model

Abstract

The invention discloses a method and a device for enhancing fine-grained perception of a visual language model. The method comprises the following steps: acquiring normalized coordinates of an image and a candidate region; extracting global semantic features and fine-grained visual features by using a main visual encoder and an auxiliary visual encoder in parallel; performing multi-scale transformation on the main features by using a feature pyramid through a mixed fine-grained region encoder, performing splicing in combination with the auxiliary features, generating geometric position embedding vectors based on coordinates by using linear projection, and performing element-by-element fusion with the spliced features to obtain mixed fine-grained region features; and finally, projecting the region reference tag into a region index Token, inputting the region index Token into a visual language model, and generating a text response containing the region reference tag. According to the method, through double-flow feature complementation and explicit geometric injection, the problem of low VLM positioning precision is solved, accurate target positioning and region understanding are realized, and meanwhile, the universal semantic capability of the model is reserved.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence and computer vision, and particularly to multimodal large model technology. Specifically, this invention relates to a method and apparatus for enhancing the fine-grained perception capabilities of visual language models (VLMs), applicable to computer vision tasks such as target localization, region generation and understanding, and visual region reasoning. Background Technology

[0002] In recent years, visual language models (VLMs) have demonstrated excellent performance in high-level semantic understanding tasks such as visual question answering (VQA) and image captioning by mapping visual features to the language model embedding space. However, existing general-purpose VLMs still have significant shortcomings in fine-grained perception tasks that require precise spatial localization (such as object detection and object grounding).

[0003] First, there is an architectural mismatch. Mainstream Virtual Logic Models (VLMs) (such as LLaVA, Qwen-VL, etc.) are designed based on sequence language generation, tending to output discrete text tokens. When the model is required to directly regress and generate precise floating-point coordinate sequences (such as [0.45, 0.32, ...]), it is prone to errors. A prediction error of a single token will cause the entire rectangular coordinate box to become invalid, and in multi-object scenarios, error accumulation is likely to occur, resulting in low recall.

[0004] Secondly, the existing improvement plans have shortcomings: 1. Coordinate quantization schemes (Pix2Seq, etc.): Discretize coordinates into words in a vocabulary. This method suffers from quantization errors in high-resolution images and struggles to handle complex scenes with overlapping instances.

[0005] 2. External Prediction Head Scheme: An independent localization module (such as a DETR head) is added after the output layer of VLMs. This not only increases inference latency but also requires the design of complex task loss functions, disrupting the unified generative paradigm of LLMs.

[0006] 3. Full Joint Training Scheme: This requires end-to-end joint training of the detection model and the VLM. This cannot reuse the rich semantic understanding and world knowledge already existing in the pre-trained VLM, resulting in extremely high training costs and a high risk of the model forgetting its original general capabilities (catastrophic forgetting).

[0007] Therefore, the localization performance of existing VLMs on standard detection datasets such as COCO is far lower than that of dedicated detection models (for example, the recall rate of some 72-parameter models is even less than 40%). How to improve the fine-grained perception capability of VLMs to achieve accurate target localization and region understanding, while not losing their original high-level scene understanding and general reasoning capabilities, is a technical problem that urgently needs to be solved. Summary of the Invention

[0008] The present invention mainly addresses the aforementioned technical problems existing in the prior art, and provides a method and apparatus for enhancing fine-grained perception of visual language models. By introducing a plug-in hybrid fine-grained region encoder (HFRE) and a decoupled region reference generation mechanism, it significantly improves the localization accuracy of small and dense targets while retaining the general semantic capabilities of VLM.

[0009] The present invention addresses the aforementioned technical problems primarily through the following technical solution: a method for enhancing fine-grained perception in visual language models, comprising the following steps: S1: Obtain the image to be processed and several candidate region information corresponding to the image to be processed, wherein the candidate region information includes the normalized spatial coordinates of potential target objects in the image to be processed; S2: Extract global semantic feature maps from the image to be processed using the main visual encoder; in parallel, extract fine-grained visual features from the image to be processed using the auxiliary visual encoder; S3: Generate hybrid fine-grained region features through a hybrid fine-grained region encoder. Specific steps include: S31: The global semantic feature map output by the main visual encoder is scaled using the simple feature pyramid module to generate several layers of main visual features. S32: Extract a first region feature from the main visual features based on the aforementioned candidate region information; in parallel, extract a second region feature from the fine-grained visual features based on the aforementioned candidate region information. S33: Concatenate the first region features with the second region features; S34: Based on the normalized spatial coordinates of the candidate region information, a geometric position embedding vector is generated using a linear projection layer, and the geometric position embedding vector is fused with the concatenated features element by element to obtain hybrid fine-grained region features. S4: The hybrid fine-grained region features are mapped to the embedding space of the visual language model through the feature projection layer to generate a region index Token; S5: The global semantic feature map is converted into image tokens through a vision-language connector, and the text instructions are converted into text tokens through a text segmenter. Then, the region index tokens, image tokens, and text tokens are input into the visual language model to generate a text response containing region reference tags. The region reference tags are associated with the region index tokens to indicate the location of the target object in the image.

[0010] This method aims to serve as a "plug-in" or "adapter" solution, seamlessly integrating into existing pre-trained visual language models (VLMs). The main visual encoder typically refers to the visual encoder inherent in the original VLM (such as CLIP-ViT), responsible for providing global features aligned with text semantics; the visual language model refers to the original VLM's Large Language Model (LLM), responsible for logical reasoning and text generation.

[0011] To address the issue that VLMs are not adept at generating anchor boxes from scratch, this invention introduces explicit candidate regions (Proposals) at the input. These candidate region information can originate from existing independent object detection models (Proposal Networks, such as RPN, YOLO, etc.) or be obtained through heuristic algorithms such as sliding windows. The obtained coordinates need to be normalized (mapped to the [0,1] interval) to eliminate scale differences caused by different image resolutions. Preferably, to balance efficiency and coverage, the top N (e.g., 100) high-potential regions can be selected as input based on the confidence level output by the external detector.

[0012] This invention employs an asymmetric dual-stream architecture. The main stream reuses the original VLM encoder, preserving semantic alignment capabilities; the auxiliary stream introduces a high-resolution visual network (such as ConvNext or DaViT) specifically to capture fine-grained information such as edges and textures. This design resolves the contradiction that a single encoder cannot simultaneously handle high-level semantics and low-level details.

[0013] Geometric injection and fusion are among the core innovations of this invention. Traditional RoI feature extraction often loses absolute positional information. This invention explicitly encodes normalized coordinates through geometric position embedding vectors and injects them into visual features in an element-wise additive manner. This allows the final generated Region Token to contain both "what it looks like" (visual texture) and "where it is" (absolute spatial prior), thereby significantly improving localization accuracy.

[0014] In terms of reference-based generation, this invention abandons the approach of directly regressing coordinate values, and instead projects regional features into special regional index tokens (such as...).<region_0> LLM only needs to determine which token corresponds to the user-described target and then use the reference tags (such as...). <object>This citation mechanism achieves logical decoupling between visual perception and language generation, avoiding the cumulative error of numerical regression.

[0015] Preferably, in step S31, the multi-scale transformation of the global semantic feature map using the simple feature pyramid module specifically includes: using a set of convolutional and deconvolutional layers with different strides to process the global semantic feature map and construct a four-layer main visual feature that includes resolution preservation, downsampling and upsampling scales. In step S32, extracting the first region feature from the main visual features based on the aforementioned candidate region information specifically includes: For each candidate region, a region of interest alignment (RoIAlign) operation is performed on each of the four main visual feature layers to obtain feature blocks at four scales. The feature blocks of the four scales are concatenated along the channel dimension to form the first region feature.

[0016] Existing pre-trained master vision encoders (such as ViT-Large) typically output feature maps at only a single scale, which is not conducive to multi-scale object detection. In order to avoid modifying the original model structure (avoiding re-pre-training), this invention uses a simple feature pyramid (SimpleFP) as a post-processing module.

[0017] SimpleFP uses a set of convolutional and deconvolutional layers to transform the single-layer output of the main encoder into four feature maps with strides of 4, 8, 16, and 32.

[0018] For each candidate bounding box, instead of enlarging and concatenating the feature maps before extraction (which consumes a lot of GPU memory), RoIAlign is performed on each of the four feature maps to obtain four feature patches at different scales. These patches are then concatenated along the channel dimension. This approach preserves the multi-scale context while significantly reducing computation and GPU memory usage, ensuring the richness of the 2048-dimensional main features. It is an efficient extraction strategy.

[0019] Preferably, the auxiliary visual encoder is a high-resolution visual Transformer or a convolutional neural network; In step S32, the extraction of the corresponding second region features specifically includes: The fine-grained visual features (generally multi-layered structures) output in step S2 are upsampled to a uniform size and stitched together to form a combined feature map; Perform a Region of Interest Align (RoIAlign) operation on the combined feature map to extract the second region feature.

[0020] Unlike mainstream approaches, auxiliary visual encoders (such as DaViT) inherently possess a strong multi-scale hierarchical structure and are primarily responsible for providing high-frequency details (texture, edges). To maximize pixel-level spatial accuracy, this invention employs a strong fusion strategy in the auxiliary stream. This involves first upsampling and aligning feature maps from different levels and then physically stitching them together to form a high-resolution combined feature map, which is then used for RoIAlign. Although this slightly increases computational cost, it is crucial for correcting localization errors of small targets.

[0021] Preferably, in step S34, generating the geometric position embedding vector specifically includes: Obtain the normalized spatial coordinates [x1, y1, x2, y2]; Each coordinate is mapped to an initial position vector using sine and cosine functions containing a series of preset frequencies, as shown in the following formula: PE(u,2i)=sin(u / T) 2i / d' ); PE(u,2i+1)=cos(u / T) 2i / d' ); Where u is the coordinate value, i is the dimension index, d' is the length of a single coordinate vector (partially embedded dimension), and T is the temperature coefficient; The initial position vectors corresponding to the four coordinates are concatenated and mapped to the same dimension as the concatenated feature through a learnable linear projection layer to obtain the geometric position embedding vector.

[0022] Directly concatenating floating-point coordinates into features is difficult for Transformer structures to utilize effectively. This invention, referencing positional encoding principles, uses a set of preset frequencies covering variations from high to low frequencies to map continuous coordinate values ​​[x, y] into a high-dimensional vector. High-frequency components capture fine-grained positional offsets, while low-frequency components capture global relative positions.

[0023] The generated initial position vector (e.g., 1024-dimensional) is often inconsistent with the dimensions of the visual features (e.g., 5888-dimensional). Therefore, this invention introduces a learnable linear projection layer, which serves two purposes: first, to achieve dimensional alignment and meet the mathematical requirements of element-wise fusion; and second, to introduce a nonlinear transformation to map the physical information of the geometric space to the semantic feature space, enabling deep fusion with the visual features.

[0024] Preferably, in step S5, the text instruction includes a region placeholder; The text response containing the region reference tag is specifically in the format " <ground>Target Name< / ground> <object> Regional Index Token< / object> The sequence of "" is used to achieve explicit alignment between text semantics and image regions.

[0025] This formatted output (such as "detected in the figure") <ground> car< / ground> <object><region_5>< / object> This implements hard binding between natural language noun phrases and image region tokens. This not only facilitates subsequent parsing of structured data, but also makes the model interpretable, allowing users to clearly know which part of the image the model refers to as the "car".

[0026] Preferably, this method further includes a model training step, which comprises: The first stage of alignment training involves freezing the parameters of the main visual encoder, the auxiliary visual encoder, and the visual language model, and training only the parameters of the simple feature pyramid module, the linear projection layer, and the feature projection layer to achieve alignment between region features and the language embedding space. The second stage of fine-tuning training involves unfreezing the auxiliary visual encoder and the visual language model, and then fine-tuning the instructions using mixed data containing both positive and negative samples. The negative samples are image-text pairs that do not contain the target object, and their corresponding training labels are constructed as text descriptions that do not contain the region reference labels, in order to suppress the model's localization output when the target is missing.

[0027] Preferably, the proportion of negative samples in the mixed data is configured to a preset proportion (e.g., 20%), and the mixed data also includes general visual question answering (VQA) task data.

[0028] Direct full-scale fine-tuning can cause VLM to forget its original general capabilities (such as OCR and commonsense reasoning). Two-stage training can solve this problem. The first stage freezes the backbone and trains only the newly added HFRE and connectors, with the aim of quickly aligning the feature spaces of vision and language. The second stage unfreezes some parameters and performs fine-tuning of instructions to adapt to specific question-answering tasks.

[0029] Negative samples are the core method used in this invention to solve the illusion problem. In practical applications, Virtual Learning Models (VLMs) often tend to over-output. This invention constructs approximately 20% negative samples (e.g., an image without a "dog," but asking "Where is the dog?"). For these samples, the training labels are forced to contain no negative samples. <object>Instead of referencing tags, it outputs plain text rejection (such as "Target not detected"). This teaches the model to remain silent when visual features do not match the text query, thus significantly reducing the false positive rate.

[0030] A device for enhancing fine-grained perception in a visual language model, comprising: The main visual encoder and the auxiliary visual encoder are used to receive the image to be processed and output the global semantic feature map and fine-grained visual features respectively; The Hybrid Fine-Grained Region Encoder (HFRE) has its input connected to the main visual encoder and the auxiliary visual encoder. It is used to receive the global semantic feature map, the fine-grained visual features, and several candidate region information, and output the hybrid fine-grained region features corresponding to the candidate regions. The feature projection layer, also known as the region-language connector, has its input connected to the hybrid fine-grained region encoder and is used to map the hybrid fine-grained region features into region index tokens. The Visual Language Model (LLM) takes as input the region index token, image tokens from the global semantic feature map, and text tokens from the text instruction, and outputs a text response containing region reference tags.

[0031] Preferably, the hybrid fine-grained region encoder (HFRE) specifically includes: The Simple Feature Pyramid module is used to perform multi-scale transformation on the global semantic feature map to generate multi-layer main visual features. The region feature extraction module is used to extract first region features and second region features from the multi-layer main visual features and the fine-grained visual features, respectively, based on the information of the several candidate regions. The feature fusion module is used to concatenate the first region features and the second region features, and generate a geometric position embedding vector using a linear projection layer based on the spatial coordinates of the candidate region information. The geometric position embedding vector is then fused element-wise with the concatenated features to generate the hybrid fine-grained region features.

[0032] The device of this invention can be implemented as an integrated computing system, in which HFRE exists as a standalone hardware acceleration module or software algorithm module. The device connects the visual end and the speech end through well-defined interfaces (feature map input, coordinate input, token output), forming a complete end-to-end fine-grained perception system.

[0033] The substantial effects of this invention are: 1. Significantly improved fine-grained perception: Through dual-stream feature complementarity and explicit geometric position injection, the detection performance of this invention on benchmark datasets such as COCO and ODinW is significantly better than existing general-purpose VLMs and approaches that of dedicated detection models.

[0034] 2. Lossless general capabilities: Thanks to the plug-in architecture and two-stage training strategy, the model enhances its localization capabilities without catastrophic forgetting, maintaining a high level of performance on general leaderboards such as OpenCompass.

[0035] 3. Strong robustness: The introduction of a reference-based generation mechanism avoids the cumulative error of coordinate regression; the introduction of a negative sample training mechanism effectively suppresses the illusion phenomenon in object detection.

[0036] 4. Flexible and efficient: The main computational load (HFRE) parameters are small and there is no need to retrain the large-scale main vision encoder, which reduces training costs and deployment threshold. Attached Figure Description

[0037] Figure 1 This is a flowchart of a method for enhancing fine-grained perception of visual language models according to the present invention; Figure 2 This is a schematic diagram of a device structure for enhancing fine-grained perception of visual language models according to the present invention. Detailed Implementation

[0038] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0039] Example: This example describes a device for enhancing fine-grained perception of visual language models, such as... Figure 2 As shown, the device is designed with a "plug-in" architecture (named VLM-FO1 in this embodiment), which can be flexibly integrated into existing pre-trained visual language models.

[0040] The device mainly consists of four core components: a two-stream visual coding system, a hybrid fine-grained region encoder (HFRE), a feature projection layer (region-language connector), and a visual language model (LLM).

[0041] 1. Two-stream visual coding system The system consists of two parallel visual pathways, designed to capture both high-level semantics and low-level details simultaneously: The primary visual encoder is preferably a reused visual encoder from the original pre-trained VLM (e.g., the ViT part of CLIP-ViT-L / 14 or Qwen-VL). This encoder is typically pre-trained on large-scale image-text pairs and excels at extracting global semantic feature maps highly aligned with text semantics. In a preferred embodiment, to preserve generality, the encoder's parameters are typically frozen during training and inference phases.

[0042] Visual-Language Connector: This module connects the main visual encoder and the visual language model (LLM). Since the dimensionality of the global semantic feature map output by the main visual encoder (e.g., 1024 dimensions for CLIP) is typically inconsistent with the embedding dimension of the LLM (e.g., 4096 dimensions for Qwen), this connector (usually implemented as a two-layer MLP or Q-Former) projects the global semantic feature map into the feature space of the LLM and flattens the two-dimensional feature map into serialized image tokens. These image tokens carry global contextual information for the entire image and are fundamental for the LLM to understand image content. In this embodiment, the parameters of this connector are kept synchronized and frozen with the main visual encoder.

[0043] The auxiliary visual encoder employs a high-resolution visual network specifically designed for dense prediction tasks, such as DaViT (Dual Attention Vision Transformer) or ConvNeXt. Its role is to extract fine-grained visual features of the image (such as edges, textures, and minute geometric structures). In this embodiment, the auxiliary encoder outputs four layers of feature maps at different scales to supplement the spatial information lost by the main encoder.

[0044] 2. Hybrid Fine-Grained Region Encoder (HFRE) This is the core module of the device, and its input is connected to the two encoders mentioned above and the external candidate region information interface. The HFRE internally contains three sub-modules: Simple Feature Pyramid Module (SimpleFP): This is a lightweight adapter module. It receives a single-scale feature map (usually the last layer) from the main visual encoder and reconstructs a feature map containing four scales through a series of convolutional or deconvolutional operations with strides of {2, 1, 1 / 2, 1 / 4}. A stride of 2 represents downsampling, step 1 preserves resolution, and strides of 1 / 2 and 1 / 4 represent upsampling. This enables CLIP-ViT, which originally did not support multi-scale operations, to possess multi-scale perception capabilities.

[0045] Region feature extraction module: used to extract the corresponding feature vectors from the main stream (SimpleFP output) and auxiliary stream based on the coordinates of the candidate region.

[0046] Feature fusion module (including geometric injection unit): responsible for deep fusion of main features, auxiliary features and geometric position embedding vectors generated by mathematical transformation, to generate the final hybrid fine-grained region features.

[0047] 3. Visual Language Model (LLM) This invention employs a general-purpose decoder-only architecture for large language models (such as Qwen-7B, LLaMA-3, etc.). Instead of modifying the internal structure of the LLM, this invention expands its vocabulary (by adding...). <region>and <object>Special tokens such as [etc.] and adjust the way of inputting the Prompt to enable it to have the output ability of fine-grained positioning.

[0048] The device also includes a text tokenizer, which is a text preprocessing module配套 with the vision language model (LLM), used to receive the user's natural language text instructions (such as "Please detect all the cars in the picture"), and use a specific tokenization algorithm (such as Byte-Pair Encoding, BPE) to segment and map it into an integer sequence or a vector sequence, that is, text Tokens. In the present invention, the vocabulary of this tokenizer is expanded to include the region placeholder unique to the present invention (such as <region> 、 <object>、 <ground>(etc.). This ensures that these special tokens can be recognized by the model as independent tokens, rather than being incorrectly broken down into ordinary characters.

[0049] The specific processing flow for image target localization and understanding using the aforementioned device, namely, methods to enhance the fine-grained perception of visual language models, such as... Figure 1 As shown, the process is as follows: Step S1: Input Acquisition and Candidate Region Preprocessing The system first obtains the image to be processed and the user's text instructions.

[0050] At the same time, information on several candidate regions corresponding to the image is obtained.

[0051] Source Note: Candidate regions can be generated by external, independent, lightweight object detection models (Proposal Networks, such as RPN, YOLOv8, or DINO). These models are only responsible for "discovering where objects might be," not for "identifying what objects are," and therefore can use general class-agnostic modes.

[0052] Filtering Strategy: To avoid excessive input tokens leading to LLM context overflow, this embodiment employs a confidence-based filtering strategy. Specifically, the Objectness Score output by the external detection model is sorted in descending order, and only the top N (e.g., N=100) high-potential bounding boxes are selected as candidate regions.

[0053] Coordinate normalization: Obtain the original coordinates [x] min ,y min ,x max ,y max It needs to be divided by the width W and height H of the image to convert it into a floating-point number in the range [0,1], denoted as [x1,y1,x2,y2].

[0054] Step S2: Parallel encoding of two-stream features The image is input into the main visual encoder to obtain the global semantic feature map F. main .

[0055] In parallel, the image is input into an auxiliary visual encoder (such as DaViT) to obtain four layers of fine-grained feature maps F at different resolutions. aux_layers .

[0056] Step S3: Construction of fine-grained region features (HFRE internal logic) This step includes four refined sub-steps: S31: Constructing the Main Visual Feature Pyramid Using the SimpleFP module to process F main Specifically, SimpleFP consists of a set of convolutional layers (with a stride of 2 for downsampling) and deconvolutional layers (with strides of 0.5 / 0.25 for upsampling), which converts single-scale F... main The transformation results in a four-layer feature pyramid with step sizes of 4, 8, 16, and 32. This process does not alter the original ViT structure and is only used as a post-processing step.

[0057] S32: Differentiated RoI Feature Extraction Based on the coordinates of the 100 candidate regions selected in step S1, features are extracted respectively: Main region features (F pri Extraction: A hierarchical extraction + concatenation strategy is adopted. For each candidate box, RoIAlign is executed once on each of the four feature maps generated by SimpleFP (output size, for example, 7×7), resulting in four feature blocks. These four feature blocks are then concatenated along the channel dimension. Assuming each layer has 512 channels, the concatenated F... pri The dimension is 2048. This method preserves the complete semantic hierarchy.

[0058] Auxiliary region features (F aux Extraction: An upsampling + concatenation strategy is used. The 4 layers of F output from the auxiliary encoder are extracted. aux_layers All samples are upsampled to the maximum resolution (i.e., a scale with a stride of 4) and stitched together to form an extremely high-resolution combined feature map. Then, RoIAlign is performed once on this combined map. Assuming the total number of auxiliary stream channels is 3840, then F... aux The dimension is 3840. This method preserves pixel-level edge texture information to the greatest extent possible.

[0059] S33: Feature splicing The extracted main region features F pri (2048-dimensional) and auxiliary region features F aux (3840 dimensions) are concatenated along the channel dimension to obtain the combined feature F. comb Its dimensions are 2048 + 3840 = 5888.

[0060] S34: Geometric Position Embedding and Fusion To enable the model to explicitly perceive "where the object is", this embodiment introduces geometric position embedding: Sine and cosine encoding: Obtain normalized coordinates [x1, y1, x2, y2]. For each scalar u, generate a vector using the following formula: PE(u, 2i) = sin(u / T) 2i / d' ); PE(u,2i+1)=cos(u / T) 2i / d' ); Where d' = 256 (single coordinate embedding dimension), and i is an index from 0 to 127. This maps each coordinate to a 256-dimensional vector.

[0061] Concatenation: Concatenate the vectors of the four coordinates to obtain an initial position vector of 4×256=1024 dimensions.

[0062] Linear projection (dimensional alignment): A fully connected layer is set to map (project) the above 1024-dimensional vector to 5888 dimensions, resulting in the geometric position embedding vector E. pos This step not only aligns the dimensions but also introduces learnable parameters to adapt to the semantic space.

[0063] Element-by-element fusion: Calculating F hybrid =F comb +E pos The final result is F. hybrid It contains both rich dual-stream visual information and precise spatial location priors.

[0064] Step S4: Generate a region index token Through a feature projection layer (usually two MLP layers), the 5888-dimensional F... hybrid Mapped to the input embedding dimension of LLM (e.g., 4096 dimensions of Qwen-7B).

[0065] For 100 candidate regions, generate 100 unique region index tokens, denoted as .<region_0> ,<region_1> ,...,<region_99> .

[0066] Step S5: LLM Inference and Reference Generation Before inputting the LLM, the system first processes the global semantic feature map output by the main visual encoder in step S2 into M image tokens (e.g., 256 tokens) through a vision-language connector.

[0067] The text segmenter converts the user's text commands into text tokens.

[0068] Subsequently, these image tokens are combined with the N region index tokens generated in step S4 (<region_i> (and the corresponding feature vectors) and text tokens are concatenated.

[0069] The final input sequence structure is as follows: <Image_Start> [Image Tokens]<Image_End> \n<region_0> <Region Token_0>...<region_N> <Region Token_N>\n[Text Tokens]; in<region_i> It's a placeholder.<Token_i> It is the feature vector calculated in step S4. This construction method allows LLM to see the global picture through image tokens (e.g., "This is a street") and to see detailed anchor points through region tokens (e.g., "There is a red fire hydrant here").

[0070] Output generation: LLM generates text containing citation tags based on semantic understanding.

[0071] For example, a user asks, "Where is the red car in the picture?" Model output: "Red car detected at..." <ground> Red car< / ground> <object><region_5>< / object> . Here, the model does not regress numerical coordinates, but instead references the 6th candidate region (Index5) in the input sequence. The system then returns the original coordinates [x1, y1, x2, y2] of the 6th candidate box to the user by looking up a table, thus achieving precise positioning.

[0072] Example 3: Training Strategies for Anti-Hallucination and Anti-Forgetting To ensure that the model enhances its fine-grained capabilities without degenerating into a machine that can only draw boxes, this invention employs a two-stage training strategy.

[0073] 1. First stage: Region-semantic alignment training Configuration: Freeze the primary visual encoder, secondary visual encoder, and LLM pedestal. Train only the HFRE (including SimpleFP, fusion module, and linear projection layer) and feature projection layer.

[0074] Objective: To make the newly added regional feature F hybrid It can quickly adapt to the semantic space of LLM so that it can be understood by LLM as "a certain piece in the image".

[0075] Data: Use Region Description data, such as RefCOCO.

[0076] 2. Second stage: Fine-tuning of multi-tasking instructions Configuration: Unfreeze the auxiliary vision encoder and LLM, and perform full parameter fine-tuning (or LoRA fine-tuning).

[0077] Negative sample construction (core features): To suppress hallucinations, approximately 20% of the training data was mixed in with negative samples.

[0078] Type A: There are objects in the image, but the user asks for objects that do not exist (e.g., asking "Where is the sofa?" in a kitchen image).

[0079] Type B: The image is an empty scene or blurry.

[0080] Label Construction: For negative samples, the model's Ground Truth label is forced to be a plain text negation, such as "No target object detected." It is strictly forbidden to include [unclear text - possibly related to labels]. <ground>or <object>Label.

[0081] In this way, the cross-entropy loss function is used to penalize the model's random prompting behavior when there is no target, so that it learns to "refuse to answer".

[0082] General data mixing: General VQA data such as LLaVA are mixed in to prevent the model from forgetting world knowledge and general dialogue capabilities.

[0083] This device can be deployed on high-performance computing servers, such as nodes configured with NVIDIA A100 or H800 GPUs.

[0084] Experiments show that compared to full retraining methods (such as Shikra), the "plug-in" method of this invention reduces training time by more than 60%. On the COCO dataset, compared to baseline models such as Qwen-VL-Chat, this invention improves the mAP (mean accuracy) of object detection from about 35% to over 55%, and maintains almost no loss in the OpenCompass general capability evaluation (remaining at around 64.5 points), verifying the perfect balance between fine-grained perception and general understanding achieved by this invention.

[0085] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0086] Although this document uses a variety of terms, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of the invention; interpreting them as any additional limitation would contradict the spirit of the invention.< / object> < / ground> < / ground> < / object> < / region> < / object> < / region> < / object> < / object>

Claims

1. A method for enhancing fine-grained perception in visual language models, characterized in that, Includes the following steps: S1: Obtain the image to be processed and several candidate region information corresponding to the image to be processed, wherein the candidate region information includes the normalized spatial coordinates of potential target objects in the image to be processed; S2: Extract global semantic feature maps from the image to be processed using the main visual encoder; in parallel, extract fine-grained visual features from the image to be processed using the auxiliary visual encoder; S3: Generate hybrid fine-grained region features through a hybrid fine-grained region encoder. Specific steps include: S31: The global semantic feature map output by the main visual encoder is scaled using the simple feature pyramid module to generate several layers of main visual features. S32: Extract a first region feature from the main visual features based on the aforementioned candidate region information; in parallel, extract a second region feature from the fine-grained visual features based on the aforementioned candidate region information. S33: Concatenate the first region features with the second region features; S34: Based on the normalized spatial coordinates of the candidate region information, a geometric position embedding vector is generated using a linear projection layer, and the geometric position embedding vector is fused with the concatenated features element by element to obtain hybrid fine-grained region features. S4: The hybrid fine-grained region features are mapped to the embedding space of the visual language model through the feature projection layer to generate a region index Token; S5: The global semantic feature map is converted into image tokens through a vision-language connector, and the text instructions are converted into text tokens through a text segmenter. Then, the region index token, the image tokens, and the text tokens are input into the visual language model to generate a text response containing region reference tags. The region reference tags are associated with the region index token to indicate the location of the target object in the image.

2. The method for enhancing fine-grained perception of a visual language model according to claim 1, characterized in that, In step S31, the multi-scale transformation of the global semantic feature map using the simple feature pyramid module specifically includes: using a set of convolutional and deconvolutional layers with different strides to process the global semantic feature map and construct a four-layer main visual feature that includes resolution preservation, downsampling and upsampling scales. In step S32, extracting the first region feature from the main visual features based on the aforementioned candidate region information specifically includes: For each candidate region, region of interest alignment is performed on each of the four main visual feature layers to obtain feature blocks at four scales. The feature blocks of the four scales are concatenated along the channel dimension to form the first region feature.

3. The method for enhancing fine-grained perception of a visual language model according to claim 1, characterized in that, The auxiliary visual encoder is a high-resolution visual Transformer or a convolutional neural network. In step S32, the extraction of the corresponding second region features specifically includes: The fine-grained visual features output in step S2 are upsampled to a uniform size and stitched together to form a combined feature map; A region of interest alignment operation is performed on the combined feature map to extract the second region feature.

4. The method for enhancing fine-grained perception of a visual language model according to claim 1, characterized in that, In step S34, generating the geometric position embedding vector specifically includes: Obtain the normalized spatial coordinates [x1, y1, x2, y2]; Each coordinate is mapped to an initial position vector using sine and cosine functions containing a series of preset frequencies, as shown in the following formula: PE(u,2i)=sin(u / T) 2i / d' ); PE(u,2i+1)=cos(u / T) 2i / d' ); Where u is the coordinate value, i is the dimension index, d' is the length of a single coordinate vector, and T is the temperature coefficient; The initial position vectors corresponding to the four coordinates are concatenated and mapped to the same dimension as the concatenated feature through a learnable linear projection layer to obtain the geometric position embedding vector.

5. The method for enhancing fine-grained perception of a visual language model according to claim 1, characterized in that, In step S5, the text instruction includes a region placeholder; The text response containing the region reference tag is specifically in the format " <ground> Target Name< / ground> <object> Regional Index Token< / object> "sequence.

6. The method for enhancing fine-grained perception of a visual language model according to claim 1, characterized in that, It also includes a model training step, which comprises: The first stage of alignment training involves freezing the parameters of the main visual encoder, the auxiliary visual encoder, and the visual language model, and training only the parameters of the simple feature pyramid module, the linear projection layer, and the feature projection layer to achieve alignment between region features and the language embedding space. The second stage of fine-tuning training involves unfreezing the auxiliary visual encoder and the visual language model, and then fine-tuning the instructions using mixed data containing both positive and negative samples. The negative sample is an image-text pair that does not contain the target object, and its corresponding training label is constructed as a text description that does not contain the region reference label.

7. The method for enhancing fine-grained perception of a visual language model according to claim 6, characterized in that, The proportion of negative samples in the mixed data is configured to a preset proportion, and the mixed data also includes general visual question answering task data.

8. A device for enhancing fine-grained perception in a visual language model, characterized in that, include: The main visual encoder and the auxiliary visual encoder are used to receive the image to be processed and output the global semantic feature map and fine-grained visual features respectively; A hybrid fine-grained region encoder, whose input is connected to the main visual encoder and the auxiliary visual encoder, is used to receive the global semantic feature map, the fine-grained visual features and several candidate region information, and output the hybrid fine-grained region features corresponding to the candidate regions. The feature projection layer, also known as the region-language connector, has its input connected to the hybrid fine-grained region encoder and is used to map the hybrid fine-grained region features into region index tokens. A visual language model, whose input is used to receive the region index token, image tokens from the global semantic feature map, and text tokens from text instructions, and outputs a text response containing region reference tags.

9. The device for enhancing fine-grained perception of a visual language model according to claim 8, characterized in that, The hybrid fine-grained region encoder specifically includes: The Simple Feature Pyramid module is used to perform multi-scale transformation on the global semantic feature map to generate multi-layer main visual features. The region feature extraction module is used to extract first region features and second region features from the multi-layer main visual features and the fine-grained visual features, respectively, based on the information of the several candidate regions. The feature fusion module is used to concatenate the first region features and the second region features, and generate a geometric position embedding vector using a linear projection layer based on the spatial coordinates of the candidate region information. The geometric position embedding vector is then fused element-wise with the concatenated features to generate the hybrid fine-grained region features.

Images