Method and system for attention weight reconstruction of visual language model facing open vocabulary semantic segmentation, terminal and storage medium

Abstract

The application discloses a visual language model attention weight reconstruction method and system for open vocabulary semantic segmentation, a terminal and a storage medium. The method reconstructs the object-level attention weight output by the model into an attention weight suitable for pixel-level dense prediction while keeping the visual language model parameters frozen by constructing a prior attention target that fuses local compactness and global connectivity and training a mapping module decoupled from the backbone. In the training stage, the prediction weight is constrained to approximate the prior target through an attention alignment loss, and a discardable cross-modal iterative enhancement module is introduced to maintain the alignment of the image and the text. In the inference stage, only the lightweight mapping module is retained to achieve plug-and-play. The method enhances the local consistency and long-range global connectivity while significantly improving the open vocabulary semantic segmentation accuracy and maintaining the lightweight deployment and easy integration characteristics.

Description

Technical Field

[0001] This invention relates to the field of computer vision and multimodal visual language model technology, and particularly to a method, system, terminal, and storage medium for reconstructing attention weights in a visual language model for open-vocabulary semantic segmentation. The method enhances pixel-level segmentation performance by mapping and recalibrating the Query / Key representations in the visual encoder's attention mechanism and combining multi-guidance information on local compactness and global connectivity, while freezing the parameters of the pre-trained visual language model. The invention also relates to a corresponding system, computer terminal, and computer-readable storage medium. Background Technology

[0002] Open-vocabulary semantic segmentation (OVSS) aims to assign a semantic label from an open set of categories (defined by text descriptions) to each pixel in an image. Visual language pre-trained models (such as CLIP) excel in open set recognition tasks due to their image-text alignment capabilities, but their training objectives are mainly based on image-level supervision, which is inherently mismatched with the pixel-level dense predictions required for semantic segmentation. Direct transfer often results in noisy and spatially inconsistent segmentation results.

[0003] Existing approaches to using CLIP and other models for OVSS can be broadly categorized into two types: 1) Training-based fine-tuning: This involves injecting segmentation-inductive bias by fine-tuning the CLIP backbone network or mask generator, either entirely or partially. While this approach improves performance, it typically requires extensive pixel-level annotation and high computational costs. Furthermore, reverse updates can disrupt the originally well-calibrated image-text embedding space, thus affecting the generalization ability of open vocabularies. 2) Training-free attention modification: This approach keeps the CLIP parameters frozen and improves the aggregation of value tokens by redesigning the interaction weights of the Query-Key in the attention mechanism. Typical examples include: SCLIP, which rewrites the qk weights as a combination of qq and kk to enhance local consistency; ProxyCLIP, which introduces the outputs of other basic vision models (such as DINO and MAE) to construct attention weights; and MaskCLIP, which constrains the qk attention to an identity matrix to directly guide feature aggregation. However, most of these methods rely on the "local compactness" heuristic (assuming that adjacent tokens are semantically similar), and do not adequately consider the global connectivity of tokens of the same type but far apart, which can easily lead to an incomplete understanding of the overall structure of the target and thus affect the segmentation quality in open scenarios.

[0004] Therefore, existing technologies have at least the following problems: how to simultaneously inject both "local compactness" and "global connectivity" information to obtain more pixel-aware attention weights and improve the semantic segmentation effect of open lexicon without destroying the cross-modal alignment capability of the frozen visual language model and without significantly increasing inference overhead remains a technical problem that urgently needs to be solved. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the present invention aims to solve at least the following technical problems: 1) It solves the problem that "object-aware attention" of visual language pre-trained models (such as CLIP) is difficult to directly adapt to pixel-level segmentation due to image-level pre-training, thereby reducing segmentation mask noise and spatial inconsistency.

[0006] 2) To address the issue that training attention-free modification methods tend to focus on local smoothing and neglect global semantic connectivity of similar long-distance tokens, in order to avoid semantic region fragmentation and incomplete structure.

[0007] 3) While freezing the parameters of the visual language model, it takes into account both the image-text alignment capability of open vocabulary and the pixel-level attention reconstruction capability, and keeps the inference stage lightweight and pluggable.

[0008] To achieve the above objectives, this invention provides a method for reconstructing attention weights in a visual language model for open-vocabulary semantic segmentation. The method is characterized by, while keeping all parameters of the pre-trained visual language model frozen, introducing a trainable attention mapping module decoupled from the backbone parameters, and constructing a prior supervision signal that integrates local compactness and global connectivity. This reconstructs the object-level attention weights output by the visual encoder into attention weights adapted for pixel-level dense prediction. The method includes the following steps: A method for reconstructing attention weights in a visual language model for open-vocabulary semantic segmentation is characterized by the following steps: S1: Extract the query vector Q, key vector K, and value vector V from the last layer output of the visual encoder in the visual language pre-trained model, as well as the category text embedding T from the text encoder output; S2: Construct local compactness prior attention matrices based on pixel-level annotation information. and global connectivity prior attention matrix Among them, the local compactness prior is determined based on the spatial distance attenuation between pixels of the same type, while the global connectivity prior is to establish a fully connected, equally weighted relationship between pixels of the same type; the two are then fused according to a preset fusion coefficient. Weighted fusion generates a unified prior attention target matrix. ; S3: The query vector Q and key vector K are nonlinearly mapped using a learnable attention mapping module to obtain the mapped query vector Q′ and key vector K′, and the predicted attention weight matrix is ​​calculated. By introducing KL divergence loss, the predicted attention weight matrix is ​​constrained. With the prior attention target matrix The distribution is aligned, thereby transforming object-level attention into attention weights suitable for pixel-level segmentation; S4: The mapped query vector Q' and key vector K', along with the original value vector V, constitute the image embedding F, which is beneficial for dense prediction tasks. A trainable semantic attention alignment module is constructed, which enhances the visual embedding F and text embedding T through multiple interactions via a cross-modal iterative attention mechanism, and finally outputs a prediction mask to align with the real segmentation results, thereby maintaining the semantic consistency of image F and text T, and implicitly restricting the learning of the attention mapping module. S5: During the inference phase, the trained attention mapping module is embedded as a plug-and-play component into the original visual language model process. The attention weights are reconstructed through forward propagation and aggregated with the value vector V. Then, the pixel-level similarity is calculated with the text embedding T, and the open-vocabulary semantic segmentation result is output.

[0009] Furthermore, the locally compact prior attention matrix The construction satisfies the following conditions: for any pixel position If the downsampled label satisfies ,but ,otherwise ,in For pixels and Euclidean distance.

[0010] Furthermore, the global connectivity prior attention matrix The construction satisfies the following conditions: for any pixel position If the downsampled label satisfies ,but ,otherwise .

[0011] Furthermore, the prior attention matrix The fusion formula is: Among them, the fusion coefficient These are preset or learnable hyperparameters, with values ​​ranging from 0.1 to 0.9.

[0012] Furthermore, the attention mapping module includes two structurally symmetrical query mapping submodules. Key mapping submodule The mapping relationship between the two is as follows: .

[0013] Furthermore, the query mapping submodule Key mapping submodule All of them consist of a stack of multiple layers of improved residual attention blocks, whose parameters are adjustable during training and are completely decoupled from the backbone parameters of the visual language model.

[0014] Furthermore, the KL divergence loss is used to constrain the predicted attention weight matrix. Fitting the prior attention target matrix : Furthermore, the training process combines Dice loss and binary cross-entropy loss for multi-task optimization (Dice + binary cross-entropy loss originates from the semantic attention alignment module).

[0015] Furthermore, the construction of the semantic attention alignment module specifically includes: (8.1) Based on the cosine similarity between image-level category labels and visual features and text embeddings, relevant semantic category subsets are selected. ,in and These represent "Image-level Category Labels (CLS-Text)" respectively. "and visual markers" -text "Similarity score between the two paths; based on..." Constructable category selection mask This allows cross-modal attention to be computed only on selected categories, thus suppressing irrelevant semantic interference; (8.2) Feature enhancement through three-stage iterative cross-modal attention: Phase 1: Using text features as keys, perform cross-modal attention enhancement on visual features: In the formula, This represents cross-modal attention. Represents the visual features output by the visual language model. Represents the text features output by the visual language model; Second stage: Update text features using the enhanced visual features as keys: ; The third stage: Using the updated text features as keys, the visual features are further enhanced to obtain aligned visual features for training supervision. .

[0016] (8.3) Based on aligned visual features Text features Given the input, we obtain the prediction mask. Finally, the Dice loss and binary cross-entropy loss are calculated together with the actual segmentation.

[0017] Furthermore, the cross-modal enhancement process includes a category selection step: based on the global classification output of the image and pixel-level similarity, a relevant semantic category is selected for each pixel. In the formula, This indicates the operation of calculating cosine distance. This represents the category token output by the visual encoder in the visual language model.

[0018] Second, the present invention also provides a system for implementing the above method, characterized in that it includes: The feature extraction unit is used to obtain the query vector Q, key vector K, and value vector V of the visual encoder in the frozen visual language model, as well as the text embedding T of the text encoder. Prior construction unit, used during the training phase to construct a prior attention target matrix that fuses local compactness and global connectivity based on pixel-level annotations; The attention mapping and reconstruction unit includes two trainable mapping submodules for mapping Q and K and generating a predicted attention weight matrix, which is optimized through loss alignment. Semantic alignment auxiliary unit is used to perform cross-modal iterative enhancements during the training phase to maintain text-image consistency; it is not enabled during the inference phase. The segmentation inference unit is a unit used in conjunction with the semantic alignment auxiliary unit to align the output of the semantic attention module during the training phase. The pixel-level classification score is calculated with the text embedding T to obtain the prediction mask, i.e., the semantic segmentation result, and then the DICE+ binary cross-entropy loss is calculated. This is also not enabled during the inference phase.

[0019] A computer terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method described above when executing the computer program.

[0020] A computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the method described above.

[0021] Compared with the prior art, the present invention has at least the following beneficial effects (all of which are supported by theoretical design and experimental data): 1) Simultaneously modeling local compactness and global connectivity enhances long-range associations between similar tokens. Experimental comparisons show that the average value of "long-range attention intensity" achieved by this invention reaches 0.373, significantly higher than training-free methods such as MaskCLIP (0.084) and SCLIP (0.142), which are mainly based on local compactness. This indicates that this invention can more fully capture the global connectivity relationships between similar, long-distance tokens.

[0022] 2) Improved performance of open-vocabulary semantic segmentation. Taking the baseline-free MaskCLIP training as an example, its average mIoU on 8 benchmarks is 27.89; after combining with the module of this invention, the average mIoU increases to 37.68, with an average gain of approximately +9.79; and significant gains are obtained on multiple datasets (e.g., VOC-21 +11.9, Context-59 +9.4, CityScapes +18.9, etc.). Simultaneously, this invention can also bring additional gains to training methods (e.g., improvements can still be obtained on frameworks such as SAN and SCAN).

[0023] 3) Lightweight, low inference overhead, and easy to deploy. The training alignment module (SAA) of this invention can be discarded after training, and only CAI is retained during the inference phase, thus maintaining pluggability and low additional computational overhead. At the same time, the module parameters of this invention are approximately 0.8M, the memory usage during training can be controlled to ≤2GB per batch, and offline training can be completed in a short training time. Attached Figure Description

[0024] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings; the drawings are only for assisting in understanding the technical solutions of the present invention and do not constitute a limitation on the scope of protection of the present invention. The accompanying drawings of the present invention include: Figure 1 This is a schematic diagram of the workflow and system module composition of the object-to-pixel attention mapping method for open vocabulary semantic segmentation in a preferred embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram illustrating the construction and fusion of local compactness prior attention and global connectivity prior attention in a preferred embodiment of the present invention, wherein (a) is the local attention matrix. (b) is the global attention matrix. (c) is the fusion weight matrix. .

[0026] Figure 3This is a schematic diagram of the structure of the context-aware attention injection module (CAI) in a preferred embodiment of the present invention, which maps the query vector / key vector and generates predicted attention weights. , , .

[0027] Figure 4 This is a schematic diagram of the iterative cross-modal fusion process of the Semantic Attention Alignment (SAA) module in a preferred embodiment of the present invention, including category selection, attention mask construction, and... , , Iterative updates and output .

[0028] The diagram shows the correspondence between icon numbers and component names: 100 - Image to be segmented; 110 - Visual encoder (frozen); 120 - Text encoder (frozen); 130 - Query vector ;140-key vector ;150-value vector ;160-Context-Aware Attention Injection Module (CAI); 161-Query Mapper ; 162-key mapper ; 170 - Prior attention construction module; 171 - Local compactness weight construction sub-unit; 172 - Global connectivity weight construction sub-unit; 180 - Attention alignment loss calculation module; 190 - Semantic attention alignment module SAA (for training); 200 - Segmentation mask output module (for training). Detailed Implementation

[0029] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes; any modifications and improvements made by those skilled in the art without departing from the concept of the present invention should fall within the protection scope of the present invention.

[0030] This invention targets "open-lexical semantic segmentation," a pixel-level semantic segmentation task where the category set is defined by text prompts and can be openly extended. For ease of explanation, the attention mechanism terminology in visual language pre-trained models is defined in Chinese as follows: Query vector. Key vector (Key) Value vector (Value) Each feature block in the visual feature grid is called a "token".

[0031] like Figure 1As shown in the figure, this embodiment provides an object-to-pixel attention mapping method for open-lexical semantic segmentation, which includes the following steps: S1 Feature Extraction and Basic Segmentation Calculation: Input the image to be segmented The frozen visual encoder is used to extract the corresponding attention module of the last layer. , , The text embedding is obtained by encoding the set of category text prompts using a frozen text encoder. Pixel-by-pixel classification is performed based on the visual features obtained after attention aggregation and the text embedding to obtain the segmentation mask: in This is a decision function that determines the maximum class or equivalent value for each pixel.

[0032] S2 Prior attention weight construction during training: During the training phase, images are acquired. Corresponding pixel-level annotation And downsampled to a resolution aligned with the visual feature grid: in Scaling factor This is the downsampling function. In an optional implementation, It can be an integer from 8 to 32 (e.g., consistent with the output grid step size of the visual encoder).

[0033] S3 Constructs a locally compact prior attention matrix : Based on the principle of "the closer the two elements are, the greater their weight," we construct local attention: in It can take the Euclidean distance in two-dimensional space; and it can normalize the matrix by row.

[0034] S4 Construct a global connectivity prior attention matrix : Based on the principle of "establishing global connectivity by labeling similar items", global attention is constructed: It can also perform mean normalization on each row, so that similar labels receive equal attention shares.

[0035] S5 fusion yields the prior attention target W_{qk}: Integrate local and global priors into a unified prior attention objective: , in The fusion coefficient; in a preferred setting It can also be adjusted within the range of 0.1 to 0.9.

[0036] S6 Context-Aware Attention Injection (CAI): for , Map and generate predictive attention : Without updating the parameters of the visual language model, an enhanced representation is obtained by introducing a mapper only for the Q and K layers of the last layer: And calculate the predicted attention weights: in , It can be composed of several attention blocks / cross-attention blocks. The final value vector is then compared with the frozen value vector. Pass together Form constitutes visual characteristics ; S7 Attention Alignment Supervision and Joint Loss: Predicting attention using KL divergence constraints Fitting the prior target : Simultaneously, to obtain final segmentation supervision, Dice loss and binary cross-entropy loss can be superimposed (Dice + binary cross-entropy loss comes from the semantic attention alignment module SAA) to obtain the total loss: In a preferred embodiment, it is possible to ( , , = (10, 5, 10).

[0037] S8 Semantic Attention Alignment (SAA): Iterative cross-modal fusion during training (discarded after training): To maintain the cross-modal alignment capability of text prompts, SAA is introduced during the training phase: First, the top-k categories are selected from the similarity scores of "image-level category label-text" and "visual label-text", and the union of these categories is taken. Construct an attention mask based on this. And perform iterative cross-attention updates: Output prediction: After training is complete, SAA does not participate in inference; during the inference phase, only CAI is retained as a plug-in enhancement module.

[0038] S9 Inference Phase Output: No annotation is required during reasoning. Compared to the prior construction steps, only the last layer is included. , Input CAI to obtain attention weights and compare them with the frozen ones. The interaction proceeds along the original computational flow to obtain the final visual features, which are then combined with text embeddings to obtain the segmentation result; thus achieving lightweight deployment and adaptability to different open vocabulary segmentation frameworks.

[0039] like Figures 1-4 As shown, the present invention also provides an object-to-pixel attention mapping system for open-lexical semantic segmentation, which may include: • Feature extraction module: Used to extract the feature from the last layer of the frozen visual encoder. , , and freeze text encoder output It includes a frozen visual encoder 110 and a frozen text encoder 120. The visual encoder (such as CLIP's ViT) is used to extract visual features from the input image 100 and outputs the query vector Q (130), key vector K (140), and value vector V (150) of the last attention module. The text encoder is used to encode the set of category text descriptions provided by the user and output the text feature embedding T.

[0040] • Prior attention construction module (training period) 170: used for annotation generate and construct , and integration .

[0041] • Context-aware attention injection module CAI 160: used to implement , and obtained .

[0042] • Loss calculation module: used for calculation And the weighted total loss including the loss from segmented supervision.

[0043] • Semantic attention alignment module SAA190 (training period): Used to filter by top-k categories and iteratively update across attention to achieve image-text alignment during training, and discarded after training.

[0044] • Segmentation mask output module 200: Used to output open vocabulary segmentation results during the training phase. This allows us to calculate the Dice loss and the binary cross-entropy loss, which are then discarded after training.

[0045] An embodiment of the present invention also provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can be used to execute the method of any of the above embodiments of the present invention, or to run the system of any of the above embodiments of the present invention.

[0046] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0047] A processor is used to execute computer programs stored in memory to implement the various steps of the methods or various modules of the systems involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method and system embodiments.

[0048] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0049] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the method of any of the above embodiments of the present invention, or to run the system of any of the above embodiments of the present invention.

[0050] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0051] Verification of Examples Integrating this invention as a plug-in into the training baseline-free method MaskCLIP can improve the average mIoU on 8 benchmarks from 27.89 to 37.68 (average gain +9.79). In another validation, the present invention achieved a score of 0.373 on the "long-range similar label attention intensity" index, which is higher than MaskCLIP (0.084) and SCLIP (0.142), indicating that it models global connectivity more fully.

[0052] Those skilled in the art will understand that various modifications can be made without departing from the core idea of ​​this invention: Prior construction: The distance metric d(i,j) can be Manhattan distance or position-encoded similarity. The weights of the global prior W_g do not have to be 1, but are adaptively set according to the number of pixels within the category.

[0053] Mapper structure: and Multilayer perceptron (MLP), lightweight Transformer layers, or other effective feature transformation networks can be used.

[0054] Integration Framework: The CAI module described in this invention can not only be applied to training frameworks such as MaskCLIP, but also be used as a plug-and-play enhancement module to be integrated into open vocabulary segmentation frameworks that require training, such as SAN and ZS3, to provide them with a better underlying attention mechanism.

[0055] Multimodal input: The method of this invention can be extended to process video sequences or point cloud data. By designing corresponding temporal or spatial priors, attention reconstruction with spatiotemporal context awareness can be achieved.

[0056] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for reconstructing attention weights in a visual language model for open-lexical semantic segmentation, characterized in that, Includes the following steps: S1: Extract the query vector Q, key vector K, and value vector V from the last layer output of the visual encoder in the visual language pre-trained model, as well as the category text embedding T from the text encoder output; S2: Construct local compactness prior attention matrices based on pixel-level annotation information. and global connectivity prior attention matrix Among them, the local compactness prior is determined based on the spatial distance attenuation between pixels of the same type, while the global connectivity prior is to establish a fully connected, equally weighted relationship between pixels of the same type; the two are then fused according to a preset fusion coefficient. Weighted fusion generates a unified prior attention target matrix. ; S3: The query vector Q and key vector K are nonlinearly mapped using a learnable attention mapping module to obtain the mapped query vector Q′ and key vector K′, and the predicted attention weight matrix is ​​calculated. By introducing KL divergence loss, the predicted attention weight matrix is ​​constrained. With the prior attention target matrix The distribution is aligned, thereby transforming object-level attention into attention weights suitable for pixel-level segmentation; S4: The mapped query vector Q' and key vector K', along with the original value vector V, constitute the image embedding F, which is beneficial for dense prediction tasks. A trainable semantic attention alignment module is constructed, which enhances the visual embedding F and text embedding T through multiple interactions via a cross-modal iterative attention mechanism, and finally outputs a prediction mask to align with the real segmentation results, thereby maintaining the semantic consistency of image F and text T, and implicitly restricting the learning of the attention mapping module. S5: During the inference phase, the trained attention mapping module is embedded as a plug-and-play component into the original visual language model process. The attention weights are reconstructed through forward propagation and aggregated with the value vector V. Then, the pixel-level similarity is calculated with the text embedding T, and the open-vocabulary semantic segmentation result is output.

2. The visual language model attention weight reconstruction method according to claim 1, characterized in that, The local compactness prior attention matrix The construction satisfies the following conditions: for any pixel position If the downsampled label satisfies ,but ,otherwise ,in For pixels and Euclidean distance.

3. The visual language model attention weight reconstruction method according to claim 1, characterized in that, The global connectivity prior attention matrix The construction satisfies the following conditions: for any pixel position If the downsampled label satisfies ,but ,otherwise .

4. The visual language model attention weight reconstruction method according to claim 1, characterized in that, The prior attention matrix The fusion formula is: Among them, the fusion coefficient These are preset or learnable hyperparameters, with values ​​ranging from 0.1 to 0.

9.

5. The visual language model attention weight reconstruction method according to claim 1, characterized in that, The attention mapping module includes two structurally symmetrical query mapping submodules. Key mapping submodule The mapping relationship between the two is as follows: 。 6. The visual language model attention weight reconstruction method according to claim 5, characterized in that, The query mapping submodule Key mapping submodule All of them consist of a stack of multiple layers of improved residual attention blocks, whose parameters are adjustable during training and are completely decoupled from the backbone parameters of the visual language model.

7. The visual language model attention weight reconstruction method according to claim 1, characterized in that, The KL divergence loss is used to constrain the predicted attention weight matrix. Fitting the prior attention target matrix : Furthermore, the training process combines Dice loss and binary cross-entropy loss for multi-task optimization, with the Dice+binary cross-entropy loss originating from the semantic attention alignment module.

8. The visual language model attention weight reconstruction method according to claim 1, characterized in that, The construction of the semantic attention alignment module specifically includes: (8.1) Based on the cosine similarity between image-level category labels and visual features and text embeddings, relevant semantic category subsets are selected. ,in and These represent "Image-level category label CLS-text" respectively. "and visual markers" -text "Similarity score between the two paths; based on..." Constructable category selection mask This allows cross-modal attention to be computed only on selected categories; (8.2) Feature enhancement through three-stage iterative cross-modal attention: Phase 1: Using text features as keys, perform cross-modal attention enhancement on visual features: In the formula, This represents cross-modal attention. Represents the visual features output by the visual language model. Represents the text features output by the visual language model; Second stage: Update text features using the enhanced visual features as keys: ； The third stage: Using the updated text features as keys, the visual features are further enhanced to obtain aligned visual features for training supervision. 。 (8.3) Based on aligned visual features Text features Given the input, we obtain the prediction mask. Finally, the Dice loss and binary cross-entropy loss are calculated together with the actual segmentation.

9. The visual language model attention weight reconstruction method according to claim 8, characterized in that, The cross-modal enhancement process includes a category selection step (8.1): based on the global classification output of the image and pixel-level similarity, a relevant semantic category is selected for each pixel. In the formula, This indicates the operation of calculating cosine distance. This represents the category token output by the visual encoder in the visual language model.

10. A system for implementing the method according to any one of claims 1-9, characterized in that, include: The feature extraction unit is used to obtain the query vector Q, key vector K, and value vector V of the visual encoder in the frozen visual language model, as well as the text embedding of the text encoder. ; Prior construction unit, used during the training phase to construct a prior attention target matrix that fuses local compactness and global connectivity based on pixel-level annotations; The attention mapping and reconstruction unit includes two trainable mapping submodules for mapping Q and K and generating a predicted attention weight matrix, which is optimized through loss alignment. Semantic alignment auxiliary unit is used to perform cross-modal iterative enhancements during the training phase to maintain image-text consistency; it is not enabled during the inference phase. The segmentation inference unit is a unit used in conjunction with the semantic alignment auxiliary unit to align the output of the semantic attention module during the training phase. With text embedding Pixel-level classification scores are calculated to obtain the prediction mask, i.e., the semantic segmentation result, and then the DICE+ binary cross-entropy loss is calculated. This is also not enabled during the inference phase.

11. A computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-9.

Images