A multi-modal visual understanding method based on consistent learning and mixed feature extraction

Abstract

This invention relates to a multimodal visual understanding method based on consistency learning and hybrid feature extraction. It includes constructing an end-to-end fine-grained consistency learning framework, introducing a hybrid region extractor, fusing local details and global semantics to generate high-quality hybrid visual cue embeddings, combining self-reconstruction loss and latent spatial consistency loss to force the model to establish explicit alignment between the input visual cue and the output segmentation label, utilizing the geometric boundary constraints of the localization task for description generation, and simultaneously optimizing localization accuracy using the semantic depth of the description task. Furthermore, it constructs a detailed localization index expression and segmentation task to enhance the model's reasoning ability for complex long text instructions. The aim is to address the problems of feature fragmentation and insufficient accuracy in existing large models for fine-grained visual localization and description tasks. Compared with existing technologies, this invention has advantages such as high accuracy and strong generalization ability in pixel-level localization and fine-grained description.

Description

Technical Field

[0001] This invention relates to the field of visual understanding, and in particular to a multimodal visual understanding method based on consistency learning and hybrid feature extraction. Background Technology

[0002] In recent years, with the rapid development of deep learning technology, Multimodal Large Language Models (MLLMs) have achieved significant breakthroughs in general vision-language tasks such as image captioning and visual question answering. These models, pre-trained on large-scale image-text pairs, have demonstrated powerful generalization and logical reasoning capabilities. As application scenarios deepen, industry and academia are placing higher demands on the visual perception capabilities of these models, shifting from coarse-grained full-image understanding to fine-grained region-level perception and reasoning.

[0003] Fine-grained visual understanding mainly includes two core paradigms: region description and visual localization. Existing technical solutions have the following significant limitations when handling these two types of tasks: First, task modeling is fragmented and lacks feature consistency. Most existing technologies treat region description and visual localization as two independent tasks for separate optimization, or, while trained within the same model framework, only perform simple multi-task learning using mixed datasets. In reality, the input visual prompt and the output segmentation output for the same target are semantically different representations of the same entity. Existing methods ignore this inherent duality, leading to often separate or even conflicting internal feature representations when describing images or finding objects based on spoken instructions. This feature fragmentation makes it difficult for the model to utilize the geometric boundary information of the localization task to calibrate the accuracy of the generated description, and also prevents it from effectively utilizing the rich semantics of the description task to optimize localization accuracy. Consequently, semantic ambiguity or localization drift easily occurs when facing complex scenes or long text instructions.

[0004] Second, visual cue feature extraction methods are coarse, lacking detail or context. In region description tasks, the key is how to transform specific regions in an image into visual cue embeddings that the model can understand. Existing technologies typically employ two approaches: one is to directly crop local image regions and input them into the encoder. While this method preserves local details, it loses the connection with the surrounding environment, causing the model to fail to understand the relationship between objects and their environment. The other is to perform simple dot multiplication or pooling on the full-image features and the mask. While this method preserves context, it often blurs the edge pixel details of small objects. Existing feature extractors struggle to balance pixel-level details with global semantic context, limiting the model's ability to recognize small or blurred targets.

[0005] Chinese invention patent CN120849867A discloses a method for 3D scene understanding and instruction analysis based on a multimodal large model. The method includes: constructing a multimodal large model; acquiring, preprocessing, spatiotemporally aligning, semantically labeling, and organizing multimodal data; introducing the GOAT module for efficient parameter fine-tuning; and employing a sparse hybrid expert architecture and lightweight parameter fine-tuning to achieve multimodal feature fusion and temporal modeling, thereby improving the model's adaptability and efficiency. This invention significantly enhances the perception and understanding of complex 3D scenes, supports multiple downstream tasks, and possesses efficient and robust intelligent decision-making capabilities. However, it still suffers from problems such as fragmented multi-task features, easy loss of environmental or detailed information during feature extraction, and insufficient model inference accuracy in complex environments.

[0006] In summary, there is currently a lack of a multimodal visual understanding method based on consistency learning and hybrid feature extraction to solve or partially solve the above problems. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a multimodal visual understanding method based on consistency learning and hybrid feature extraction, so as to solve or partially solve the problems of feature fragmentation in multi-task, easy loss of environment or details in feature extraction, and insufficient model inference accuracy in complex environments.

[0008] The objective of this invention can be achieved through the following technical solutions: This invention provides a multimodal visual understanding method based on consistency learning and hybrid feature extraction, specifically including: S1. Acquire multimodal data and preprocess it to obtain preprocessed data, wherein the multimodal data includes image data and corresponding text instructions; S2. Construct an end-to-end fine-grained consistency learning framework, input preprocessed image data, use a hybrid region extractor to extract features through parallel processing of semantic and positional branches, obtain final semantic tokens and final positional tokens respectively, and combine the final semantic tokens and final positional tokens to generate a hybrid visual cue embedding; S3. The preprocessed image data, text instructions and the hybrid visual cues are embedded and spliced ​​together to construct a multimodal input sequence. The sequence is then input into a large multimodal model for joint reasoning to establish a semantic mapping relationship between the hybrid visual cues embedded at the input end and the segmented token at the output end. S4. Based on the semantic mapping relationship, generate output results according to the task type of the input instruction to achieve multimodal visual understanding; S5. Use the loss function of two-stage training to perform consistency learning and parameter updates on the multimodal large model.

[0009] As a preferred technical solution, the processing of the semantic branch includes: The target region of the preprocessed image data is cropped and enlarged, and its features are extracted by the pre-trained segmentation model encoder. Combined with the foreground selection operator and mask pooling operation, the dimensions are aligned through the projection layer to generate the initial mask token. The initial mask token is used as the query vector, and the features extracted by the segmentation model encoder are used as key-value pairs. Pixel enhancement is then performed on the initial mask token using a cross-attention mechanism. The semantic context features of a global visual encoder are introduced. Taking the center point of the target region as a reference, global semantic cues are fused through sampling operators and deformable attention mechanisms, and the final semantic token is generated through a multilayer perceptron.

[0010] As a preferred technical solution, the processing of the position branch includes: In geometric coordinate encoding, the spatial geometric information of the target region is extracted, and the spatial geometric information is encoded by a multilayer perceptron to obtain geometric features; In shape structure encoding, the binary mask of the target region is adjusted to a fixed resolution, and then dimensionality reduction encoding is performed by another independent multilayer perceptron to extract shape structure features; The geometric features and the shape and structure features are concatenated in the channel dimension and then passed through a linear projection layer to generate the final position token.

[0011] As a preferred technical solution, the task types of the input instructions include descriptive tasks and location tasks; When performing the description task, natural language text describing the details of the target region is generated autoregressively based on the image and the hybrid visual cue embedding. When performing the localization task, a segmentation token is generated in the output sequence based on the text generated by the description task. The segmentation token is then input into the mask decoder to obtain a binary pixel-level mask, thereby achieving the localization of the target object.

[0012] As a preferred technical solution, the two-stage training defines a total loss function, which includes cross-entropy loss, segmentation loss, and consistency learning loss. The definition of consistency learning loss differs in different training stages.

[0013] As a preferred technical solution, in the first stage, the self-reconstruction loss is used as the consistency learning loss. The self-reconstruction loss forces the mask decoder to decode the position token into a result consistent with the original true mask, expressed as: in, For the losses of self-reconstruction, To divide the loss, For mask decoder, For location tokens, This is the original, true mask.

[0014] As a preferred technical solution, in the second stage, the latent feature loss is used as the consistency learning loss. The latent feature loss is calculated by maximizing the cosine similarity between the output segmentation token and the input position token corresponding to the same region in the feature space.

[0015] As a preferred technical solution, in the second stage of the two-stage training, a consistency learning mechanism is used to freeze the parameters of the visual encoder and decoder, and only fine-tune the parameters of the multimodal large model, thereby forcing the segmentation tokens output by the localization task to be consistent with the hybrid visual cue embeddings describing the task input in the latent space distribution.

[0016] As a preferred technical solution, the method also introduces a detailed localization representation segmentation task, which specifically includes: selecting samples that include detailed local descriptions, reversing and reconstructing the descriptive text of the samples into referential instructions, inputting the image and the referential instructions into the multimodal large model for joint inference, and outputting a segmentation token.

[0017] As a preferred technical solution, the method further includes: performing a comprehensive performance evaluation of the method using a visual benchmark dataset that includes referential segmentation, region description, and referential dialogue.

[0018] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) This invention constructs a consistency learning mechanism based on latent space, introduces self-reconstruction loss and latent loss in the two-stage training respectively, explicitly narrows the distance between the description task and the localization task in the feature space, enables the model to use the geometric boundary constraints of the localization task to correct the illusion in description generation, and uses the semantic depth of the description task to assist in the localization of complex targets, effectively solving the problem of feature fragmentation in multi-tasks, and realizing the improvement of semantic consistency of target objects in multimodal scenarios.

[0019] (2) This invention combines the high-resolution advantage of local cropping, the pixel-level details of the SAM encoder, and the global context of the visual encoder through a hybrid region extractor that includes semantic branches and positional branches. This generates a hybrid visual cue embedding that combines minute details and environmental semantics. This solves the problem that existing technologies are prone to losing the environment or details when extracting features, thereby improving the accuracy of feature extraction and enhancing the perception of small targets and occluded targets.

[0020] (3) This invention constructs explicit geometric coding in the position branch and uses detailed positioning pointers to express the segmentation task and then performs fine-grained feature joint reasoning, which significantly enhances the model's ability to understand complex long text instructions and complex semantic information, including spatial relationships and attribute features. It solves the problem of insufficient model reasoning accuracy in complex environments and realizes enhanced model reasoning and generalization ability in complex scenarios. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the main steps of the present invention; Figure 2 This is a schematic diagram of the overall technical architecture of the consistency learning framework of the present invention; Figure 3 This is a detailed schematic diagram of the modular structure of the mixed region extractor of the present invention; Figure 4 This is a schematic diagram of the multimodal large model network structure and two-stage training strategy of the present invention; Figure 5 This is the visualization result of the present invention under various visual tasks. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] To address the problems existing in the prior art, this embodiment provides a multimodal visual understanding method based on consistency learning and hybrid feature extraction. The method flow is as follows: Figure 1 As shown.

[0024] This method proposes to construct an end-to-end fine-grained consistency learning framework, hybrid region extraction and latent alignment, and to construct a detailed localization index representation segmentation task, specifically including: The end-to-end Fine-Grained Consistency Learning (FCLM) framework unifies the representation of region-level description and pixel-level localization tasks by explicitly maintaining the semantic consistency between input visual cues and output segmentation labels. While utilizing the geometric boundary constraints of the localization task for description generation, it also leverages the semantic depth of the description task to optimize localization accuracy, achieving mutual promotion and symbiosis of cross-task features. This significantly improves the accuracy and generalization ability of multimodal large models in fine-grained perception tasks. Figure 2As shown, the overall structure comprises two main modules: localization and description. The localization task processes the information output by the large language model through a mask decoder, generating text descriptions and visual masks for the segmentation results. The description task, on the other hand, is responsible for generating image descriptions. Images are processed by a visual encoder to generate image embeddings, text instructions are processed by a word segmenter to obtain text embeddings, and visual cues are transformed into hybrid visual cue embeddings by a hybrid region extractor. These multimodal embeddings are input into the large language model for fusion processing, ultimately obtaining image segmentation and localization.

[0025] Hybrid Region Extraction and Potential Alignment: This technique innovatively integrates progressive feature extraction with bidirectional consistency loss. Through the collaborative work of semantic and positional branches, it preserves minute details by utilizing local focus cropping and pixel enhancement, while introducing global context by combining deformable attention mechanisms. Furthermore, it forces explicit alignment of input cue embedding and output segmentation labels in the feature space through self-reconstruction loss and latent spatial loss, thereby optimizing the accuracy of fine-grained semantic transmission and avoiding the problems of damaged visual cue information and fragmented task features in traditional methods.

[0026] The Detailed Localization-Instruction-Expression Segmentation (DL-RES) task fills the gap in existing benchmarks that only focus on brief instructions or coarse-grained localization. By reversing and reconstructing detailed local image descriptions into complex instructional terms, it conducts extreme tests on the model's pixel-level segmentation capabilities under long text logical reasoning. This achieves a deep quantification of the model's fine-grained understanding capabilities, better aligns with the perception needs of real-world intelligent systems in complex interactive scenarios, and provides key technical support for the refined application of multimodal large models.

[0027] The method of the present invention can be described by the following steps, such as... Figure 1 As shown: S1. Multimodal data acquisition.

[0028] First, the system performs multimodal data acquisition and preprocessing. This step receives the raw image to be processed. and the corresponding text instructions As input. The form of the input data varies depending on the specific task type: for region-level description tasks, the input includes the original image. Text instructions and user-specified visual cue mask It is used to explicitly indicate the region of interest in an image; for pixel-level localization tasks, the input includes the original image. Text instructions containing referential descriptions After acquiring the input, the system processes the original image. A unified resolution adjustment and normalization preprocessing is performed to obtain the image tensor adapted to the model. Simultaneously, a text segmentation tool is used to segment the text instructions. This is transformed into corresponding text embedding sequences, providing a standardized data foundation for subsequent multimodal joint inference. In the description task during the training phase, the true binary mask of the target object is obtained. Calculate the minimum bounding rectangle of the mask to obtain the pixel-level center coordinates. and width and height .

[0029] S2. Generation of hybrid visual cues.

[0030] To address the difficulty in simultaneously capturing local details and global context in existing technologies, this invention designs a hybrid region extractor that includes semantic and positional branches. This designed hybrid region extractor generates hybrid visual cue embeddings, aiming to solve the problem of insufficient semantic information in traditional visual cues. This step extracts features through parallel processing of semantic and positional branches, such as... Figure 3 As shown, it specifically includes: S2.1. In the semantic branch, a progressive generation strategy is adopted. First, local aggregation is performed on the input image. Perform focus clipping to magnify the region of interest. and will Input pre-trained segmentation model encoder Feature extraction. Combine foreground selection operator. With mask pooling operation and through the projection layer Align dimensions and generate initial mask tokens The calculation process can be expressed as follows: S2.2. Pixel enhancement is then performed using the initial mask token. As a query vector, Query is used to segment the model encoder on the original image. The extracted high-resolution features are used as keys and values, and then processed through a cross-attention mechanism. right The calculation process for refining the update can be expressed as follows: S2.3. Finally, semantic guidance is performed by introducing a global visual encoder. Semantic context features. Based on the region center point. For reference, through the sampling operator and deformable attention mechanism Global semantic cues are fused, and the final semantic mask is generated using a multilayer perceptron (MLP), represented as follows: S2.4. In the position branch, in order to make the generated mark contain both explicit coordinate priors and reconstructable shape and structure information, this invention designs a dual-path encoding fusion strategy.

[0031] In geometric coordinate encoding, the system extracts the spatial geometric information of the target area, including the coordinates of the center point, width, and height. These coordinate data, after being encoded by a multilayer perceptron, generate location tokens that explicitly contain spatial geometric features, denoted as... .

[0032] In shape structure encoding, the binary mask of the target region Adjust to a fixed resolution, such as 448×448, to get It is then flattened and dimensionality-reduced through another independent multilayer perceptron to extract shape and structural features.

[0033] Finally, the geometric features and shape structure features are concatenated along the channel dimension and then passed through a linear projection layer to generate the final location marker. This design ensures In the subsequent first-stage training, it can serve as a sufficient source of information to support the mask decoder in reconstructing the original mask.

[0034] Finally, the system combines the aforementioned semantic mask tokens with location tokens to form a hybrid visual cue rich in semantic and spatial information. The process of generating the above-mentioned positional branches can be uniformly represented by the following formula: in, This represents the generated output location token, which participates in subsequent reasoning as part of the hybrid visual cue embedding; and A multilayer perceptron network that processes coordinate data and mask data contains linear layers and nonlinear activation functions. This represents a vector concatenation operation; This means flattening a two-dimensional image into a one-dimensional vector. Represents the pixel coordinates of the center point of the target region; This represents the width and height of the target area, i.e., the pixel value. These represent the overall width and height of the input image, respectively, and are used to normalize the geometric parameters.

[0035] S3. Multimodal joint reasoning.

[0036] First, a global image embedding is extracted from the entire image using a global visual encoder. Then, the global image embedding, the text embedding generated in step S1, and the hybrid visual cue embedding generated in step S2 are combined. A complete multimodal input sequence is constructed by concatenating specific sequences. This sequence is then fed into the backbone network of a Large Language Model (LLM). The model utilizes a self-attention mechanism to capture deep semantic relationships between images, text instructions, and mixed visual cues within the same latent space. During this process, the model learns to establish the mixed visual cue embeddings at the input end and the segmentation tokens at the output end. The latent semantic mapping relationship between them, where the hybrid visual cue embedding represents the region input and the segmentation token represents the region output. .

[0037] S4. Task-adaptive output.

[0038] After inference, the model adaptively generates output based on the task type of the input instructions. For region-level description tasks, the model embeds image features and hybrid visual cues. The model autoregressively generates natural language text describing the details of the target region; if performing pixel-level localization tasks, the model generates special segmentation tokens in the output sequence based on the text description. The system extracts the generated segmentation token and inputs it into the mask decoder. The mask decoder combines image features to decode abstract segmentation tokens into specific binary pixel-level masks, i.e. This enables precise segmentation and positioning of the target object.

[0039] S5. Consistent Learning and Parameter Update.

[0040] To enhance the consistency of the model across description and localization tasks, this invention employs a two-stage training strategy involving a specific loss function design for parameter updates. The training process defines a total loss function. Its formula is: in, This is used to optimize the cross-entropy loss for text generation; The segmentation loss used to optimize mask quality is typically composed of a weighted average of the DICE loss and the binary cross-entropy loss. This is the consistency learning loss.

[0041] The training is divided into two phases, with different objectives for each phase. The definitions differ, such as Figure 4 As shown: S5.1. The first phase aims to establish general capabilities. The system performs joint training on all modules and enables self-reconstruction loss. This loss serves as a consistency loss. This loss forces the mask decoder... Enter the location token Decoded to match the original real mask The consistent result is calculated using the following formula: This mechanism enables the input end It can learn precise spatial shape information and initially realize the connection with the output end. Functional alignment.

[0042] S5.2. The second phase aims to enhance fine-grained reasoning and consistency. In this phase, the parameters of the visual encoder, hybrid region extractor, and mask decoder are frozen, and only the large language model is fine-tuned. The detailed localization and reference segmentation (DL-RES) task is also introduced in the second phase training to further enhance fine-grained understanding capabilities.

[0043] First, the system performs reverse reconstruction of the task data. Unlike short pronoun phrases in traditional datasets, this step is based on datasets containing dense region descriptions, such as the Describe Anything Dataset, selecting specific regions with rich attribute information in the image as target regions. The system extracts the long text description corresponding to this target region, denoted as... And reconstruct it from descriptive text into directive text. Specifically, it will... Embedded into a preset instruction template. For example, segment objects in an image that match the following description: Construct detailed referential instructions containing long text information. .

[0044] Next, joint inference of fine-grained features is performed. The system will process the original image... Compared with the above detailed reference instructions The input is fed into a multimodal large language model. During this process, the model is forced to understand the complex semantic information contained in the instructions, such as fine-grained attributes like color, texture, and relative position, rather than simply identifying the target category. After inference, the model predicts the corresponding segmentation tokens in the output sequence. .

[0045] Finally, supervised optimization based on the latent space is performed. To verify and enhance the model's localization accuracy for detailed descriptions, the system does not rely solely on pixel-level mask loss for supervision, but instead introduces latent feature consistency constraints. Utilizing the location branch from step S2, corresponding pseudo-label location tokens are generated based on the geometric coordinates of the target region. Subsequently, latent feature loss. This serves as a consistency loss. The loss is achieved by maximizing the output token segmentation corresponding to the same region. With input position token It is implemented using cosine similarity in the feature space, and its calculation formula is as follows: in, This represents the L2 norm.

[0046] By minimizing this loss, the segmentation tokens generated by the model are forced to be highly aligned with the location tokens representing precise spatial locations in the semantic space, thereby achieving a fine-grained semantic mapping from detailed textual descriptions to precise spatial positioning.

[0047] S6. Comprehensive performance evaluation of multimodal visual understanding methods.

[0048] To address the lack of fine-grained inference samples in existing datasets, this invention utilizes datasets containing detailed local descriptions, such as the Describe Anything Dataset, to invert and reconstruct detailed text into complex instruction expressions, requiring the model to output the corresponding mask. Training through this task enhances the model's ability to perform complex inferences from long text descriptions to pixel-level localization.

[0049] like Figure 5 As shown, Figure 5 It showcases visualization results for various visual tasks, including Reference Object Classification (ROC), Detailed Local Captioning (DLC), Detailed Local Reference Segmentation (DL-RES), Reference Segmentation (RES), Reasoning Segmentation (ReasonSeg), Multi-Objective Reasoning Segmentation (MUSE), and Dialogue-Based Localization (GCG).

[0050] If the above methods are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device, including a personal computer, server, or network device, to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0051] To verify the performance of the above method, the following experiment was designed in this embodiment.

[0052] This invention uses a constructed multi-source hybrid dataset as the benchmark dataset. This dataset covers three major categories of fine-grained visual understanding paradigms and four core visual perception tasks, containing millions of high-quality image-text pair samples. The training and evaluation datasets of this invention contain approximately 1.4 million samples, with a wide range of data sources and sufficient sample size. It integrates multiple authoritative public data sources, including RefCOCO / + / g, ReasonSeg, Visual Genome, LVIS-PACO, Flickr30K, and Describe AnythingDataset. Specifically, the samples in the dataset are finely divided into multiple categories according to task type, comprehensively covering the core needs from semantic understanding to pixel perception, including: Pixel-level visual localization: encompasses referential segmentation and inference segmentation, focusing on outputting precise binary masks based on brief or logical instructions; Region-level image description: encompasses object classification and detailed local description, focusing on generating fine-grained text descriptions for specific image regions; Dialogue-based localization: This includes multi-turn dialogue data interwoven with text and images, used to train the model's ability to simultaneously describe and localize during interactions; To verify the beneficial effects of the proposed multimodal visual understanding method (FCLM) based on consistency learning and hybrid feature extraction, extensive comparative experiments were conducted on several mainstream visual understanding benchmark datasets. The experimental results are shown in Tables 1 to 5. The following section provides a detailed analysis of the evaluation metrics and specific experimental data.

[0053] To verify the effectiveness of the method of the present invention, it was comprehensively compared with the most advanced methods in the current technical field. These comparison methods mainly include: (1) general multimodal large models, such as GPT-4o and Qwen2.5-VL, which represent the general visual understanding benchmarks trained on large-scale data; (2) pixel-level visual localization and segmentation models, such as LISA, PixelLM, SEEM and UniPixel, which focus on solving the tasks of referential segmentation and multi-objective reasoning; (3) fine-grained region perception and description models, such as Osprey, Ferret, DAM and SPHINX, which represent the latest technical level of feature extraction and description for local visual regions.

[0054] In the above experiments, the following metrics were used to quantitatively evaluate the model performance: IoU metrics are used to evaluate the degree of overlap between the predicted mask and the ground truth mask in segmentation tasks. cIoU refers to the cumulative intersection-union ratio, and gIoU refers to the generalized intersection-union ratio; higher values ​​indicate more accurate localization.

[0055] Semantic Similarity (SS): Used to evaluate the cosine similarity between predicted text and real labels in the semantic embedding space in classification or phrase generation tasks.

[0056] Semantic IoU (sIoU): Measures the degree of overlap between the predicted text and the real text at the lexical level.

[0057] Fine-grained description metrics (Pos, Neg, Avg): for detailed description tasks (DLC); Pos (Positive) represents the proportion of the generated description that correctly includes the target object's attributes; Neg (Negative) represents the proportion of non-target object attributes successfully excluded from the generated description, i.e., the proportion of hallucinations that did not occur. Avg is the average of the two; A higher Neg score indicates that the model is better able to suppress hallucinations and that the description is more faithful to the image content.

[0058] For the task of dialogue localization, which involves dual-modal interaction of vision and language, the following two types of indicators are used for comprehensive evaluation: Text generation quality metrics (M, C): METEOR (M) is based on single-word exact matching and synonym matching, used to evaluate the language fluency and semantic alignment of the generated response; CIDEr (C) is based on TF-IDF weighted n-gram consistency, used to quantitatively evaluate the semantic content consistency between the generated image description and the human reference text.

[0059] Comprehensive localization metrics (AP50, Recall): For the multi-target localization needs involved in the dialogue process, AP50 is used to evaluate the average detection accuracy when the Intersection over Union (IoU) threshold is 0.5; Mask Recall is used to evaluate the proportion of instances of the target referred to by the user that the model successfully retrieves and segments in multiple rounds of dialogue. This metric directly reflects the completeness of the model's response in interactive scenarios.

[0060] The specific experimental results are shown in Table 1-5.

[0061] Table 1. Comparison with state-of-the-art methods in image referential segmentation (RES) and reason segmentation (Reason Seg) (%) Table 1 shows the image referencing and reasoning segmentation task. As shown in Table 1, the accuracy of the method of this invention on the RefCOCO series datasets comprehensively surpasses the existing 7B parameter state-of-the-art models, such as UniPixel, proving that the consistency learning strategy can achieve better visual-language alignment with less computational cost.

[0062] Table 2 Comparison with state-of-the-art methods on Multi-Object Inference Segmentation (MUSE) (%) Table 2 shows the multi-objective reasoning segmentation task. As shown in Table 2, in the highly challenging multi-objective scenario, the gIoU and cIoU metrics of this invention are significantly better than existing methods such as PixelLM, indicating that the explicit positional encoding effectively enhances the model's ability to distinguish and locate multiple instances.

[0063] Table 3 Comparison with state-of-the-art methods in image detail description (DLC) (%) Table 3 shows the fine-grained detail description task. As shown in Table 3, the present invention achieves a high score of 83.6% on the Neg index, which measures the ability to suppress hallucinations, outperforming GPT-4o. This confirms that the self-reconstruction loss effectively ensures that the generated description is strictly faithful to the pixel details of the image by forcing visual cues to restore the mask.

[0064] Table 4 Comparison with state-of-the-art methods in Image Representation Object Classification (ROC) (%) Table 4 shows the semantic similarity (SS) of the present invention on the LVIS dataset, which is 90.3%, significantly outperforming methods such as Osprey, and verifying the significant advantage of the hybrid region extractor in preserving local high-fidelity semantic features.

[0065] Table 5 Comparison with state-of-the-art methods in image dialogue-based generation (GCG) (%) Table 5 shows the localization dialogue generation task. As shown in Table 5, the present invention significantly outperforms state-of-the-art methods such as GLaMM in key indicators such as CIDEr (which measures the overall generation quality) and Recall (the localization recall rate), proving that the model achieves bidirectional mutual promotion and synergistic improvement of descriptive semantics and localization accuracy in text-text interwoven dialogue scenarios.

[0066] The above comparison results fully verify the effectiveness and universality of the method proposed in this paper.

[0067] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1.A multi-modal visual understanding method based on consistent learning and hybrid feature extraction, characterized in that, The method specifically comprises: S1. Obtain multi-modal data and pre-process to obtain pre-processed data, the multi-modal data comprising image data and corresponding text instructions; S2. Construct an end-to-end fine-grained consistency learning framework, input the pre-processed image data, use a hybrid region extractor, process and extract features through a semantic branch and a position branch in parallel, respectively obtain final semantic tokens and final position tokens, and combine the final semantic tokens and the final position tokens to generate a hybrid visual prompt embedding; S3. Concatenate the pre-processed image data, the text instructions and the hybrid visual prompt embedding to construct a multi-modal input sequence, input the multi-modal large model for joint inference, and establish a semantic mapping relationship between the input hybrid visual prompt embedding and the output segmentation token; S4. Based on the semantic mapping relationship, generate an output result according to the task type of the input instruction to realize multi-modal visual understanding; S5. Use a two-stage training loss function to perform consistency learning and parameter updating on the multi-modal large model. 2.The multi-modal visual understanding method based on consistent learning and hybrid feature extraction according to claim 1, characterized in that, The processing of the semantic branch comprises: Crop and enlarge the target region of the pre-processed image data, input a pre-trained segmentation model encoder to extract features, combine a foreground selection operator and a mask pooling operation, align the dimensions through a projection layer, and generate initial mask tokens; Use the initial mask tokens as query vectors and the features extracted by the segmentation model encoder as key-value pairs to perform pixel enhancement on the initial mask tokens through a cross-attention mechanism, Introduce semantic context features of a global visual encoder, take the center point of the target region as a reference, fuse global semantic clues through a sampling operator and a deformable attention mechanism, and generate final semantic tokens through a multi-layer perception machine. 3.The multi-modal visual understanding method based on consistent learning and hybrid feature extraction of claim 1, characterized in that, The processing of the position branch comprises: In geometric coordinate coding, extract spatial geometric information of the target region, encode the spatial geometric information through a multi-layer perception machine to obtain geometric features; In shape structure coding, adjust the binary mask of the target region to a fixed resolution, perform dimension reduction coding through another independent multi-layer perception machine to extract shape structure features; Concatenate the geometric features and the shape structure features in the channel dimension, and generate final position tokens through a linear projection layer. 4.The method of claim 1, wherein, The task type of the input instruction comprises a description task and a positioning task; When the description task is executed, based on the image and the hybrid visual prompt embedding, a natural language text describing details of the target region is generated in an autoregressive manner; When the positioning task is executed, based on the text generated by the description task, a segmentation token generated in the output sequence is input into a mask decoder to obtain a binary pixel-level mask, and the positioning of the target object is realized. 5.The multi-modal visual understanding method based on consistent learning and hybrid feature extraction of claim 1, characterized in that, The two-stage training defines a total loss function, the total loss function comprising a cross-entropy loss, a segmentation loss and a consistency learning loss, and the definition of the consistency learning loss is different in different training stages. 6.The method of claim 5, wherein, In the first stage, a self-reconstruction loss is used as the consistency learning loss, the self-reconstruction loss forces the mask decoder to decode the position token into a result consistent with the original real mask, and is expressed as: wherein, is a self-reconstruction loss, is a segmentation loss, is a mask decoder, is a position token, is an original ground truth mask. 7.The multi-modal visual understanding method based on consistent learning and hybrid feature extraction of claim 5, wherein, In the second stage, the latent feature loss is used as the consistency learning loss, which is calculated by maximizing the cosine similarity between the output segmentation token and the input position token corresponding to the same region in the feature space. 8.The method of claim 1, wherein, In the second stage of the two-stage training, a consistency learning mechanism is used to freeze the parameters of the visual encoder and decoder and fine-tune only the parameters of the multimodal large model, thereby forcing the segmentation tokens output by the localization task to be consistent with the hybrid visual cue embeddings describing the task input in the latent space distribution. 9.The multi-modal visual understanding method based on consistent learning and hybrid feature extraction of claim 1, wherein, The method also introduces a detailed localization representation segmentation task, which specifically includes: selecting samples that include detailed local descriptions, reversing and reconstructing the descriptive text of the samples into referential instructions, inputting the image and the referential instructions into the multimodal large model for joint inference, and outputting a segmentation token. 10.The method of claim 1, wherein, The method further includes: performing a comprehensive performance evaluation of the method using a visual benchmark dataset that includes referential segmentation, region description, and referential dialogue.

Images