Image segmentation method based on semantic alignment for open scene understanding

Abstract

This invention discloses a semantic alignment-based image segmentation method for open scene understanding. In the frozen visual encoder, an orthogonal alignment mechanism is introduced in the last self-attention module to correct the feature geometric relationships during the attention calculation stage. Based on the matching relationship between the aligned image patch features and the text prototype, pixel-by-pixel initial semantic prediction results are generated. A text-aware Laplacian propagation model is constructed on a compact grid, using text semantic relationships as the gating basis for neighborhood connections and image gradient information as boundary constraints to structurally refine the pixel-level classification results, resulting in semantically consistent and clearly defined segmentation results. By introducing semantic alignment and text-guided propagation mechanisms during the inference stage, this invention significantly improves the stability and consistency of pixel-level semantic responses in open vocabulary segmentation, significantly improves the segmentation effect of small target regions while maintaining complete coverage of large target regions.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and multimodal understanding technology, specifically a semantic alignment-based image segmentation method for understanding open scenes. Background Technology

[0002] Semantic segmentation is a fundamental task in computer vision, aiming to assign a semantic category label to each pixel in an image. Traditional semantic segmentation methods rely on large-scale pixel-level labeled datasets for supervised training, and can only handle a fixed set of categories. Once a new category is introduced, the model must be re-labeled and retrained, resulting in high cost and low efficiency. To address these issues, Open Lexical Semantic Segmentation (OVSS) emerged. This technique uses natural language to specify the category set during the inference phase, allowing the model to adapt to new categories without retraining. With the development of vision-language contrastive learning models (such as CLIP and ALIGN) and Transformer vision backbones, OVSS has made rapid progress. However, these models primarily emphasize global consistency between images and text during the training phase, rather than fine-grained pixel-level alignment. This often leads to problems such as high noise, spatial inaccuracy, and severe background bias in image patch features, thus affecting segmentation quality.

[0003] To improve performance, existing methods mainly adopt two approaches: one is to fine-tune the model with additional data, train the decoder or adapter, or introduce weak supervision information to improve dense prediction performance; the other is to keep the model frozen and make improvements only during the inference stage, such as feature cleansing, attention reconstruction, spatial smoothing, and graph structure propagation. While these methods improve segmentation accuracy to some extent, they often lead to high system complexity and large inference latency. Furthermore, text is often used only as a classifier and fails to participate in modeling semantic relationships between pixels, resulting in limited semantic alignment capabilities. Therefore, how to correct the visual feature structure from the attention geometry source without retraining or adding additional data, and introduce textual semantic relationships to guide the flow of pixel information during the propagation stage, in order to achieve a concise, efficient, and highly generalizable open-vocabulary semantic segmentation method, has become a pressing technical problem in this field. Summary of the Invention

[0004] This invention proposes an image segmentation method based on semantic alignment for understanding open scenes.

[0005] The technical solution to achieve the purpose of this invention is: an image segmentation method based on semantic alignment for open scene understanding, comprising:

[0006] The natural scene image to be segmented is input into the frozen visual encoder to obtain aligned image patch features;

[0007] Input the set of user input categories corresponding to the current image into the frozen text encoder to extract text prototype features;

[0008] Calculate the cosine similarity between the aligned image patch features and the text prototype features, and generate an initial segmentation score map;

[0009] The initial segmentation score map is downsampled to generate a compact grid; a category relationship graph is constructed based on the similarity between text prototype features, and propagation weights are further constructed; Laplacian propagation is performed on the compact grid to obtain the grid propagation result;

[0010] The grid propagation results are upsampled to the original image resolution using bilinear interpolation to obtain the final category score map.

[0011] Compared with the prior art, the significant advantages of this invention are:

[0012] This invention addresses the problems of unstable semantic matching, poor structural consistency, and reliance on complex post-processing in training-free open-vocabulary semantic segmentation by proposing a semantic alignment-based open-vocabulary segmentation technique. This method follows a "align first, then propagate" reasoning approach: by introducing an orthogonal alignment mechanism into the last layer of the frozen visual-language backbone network's self-attention layer, the inconsistency between visual features and the text semantic space is corrected; simultaneously, a text-guided semantic propagation strategy is combined to structurally refine the pixel-level prediction results, thereby achieving a stable improvement in open-vocabulary segmentation results without retraining, additional data, or auxiliary backbone networks.

[0013] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0014] Figure 1 This is a flowchart of the open vocabulary segmentation method based on semantic segmentation in Embodiment 1 of the present invention.

[0015] Figure 2 This is the attention response diagram at the selected query point in Embodiment 1 of the present invention. Detailed Implementation

[0016] The present invention will now be described in further detail with reference to the accompanying drawings.

[0017] like Figure 1As shown, this paper presents an image segmentation method based on semantic alignment for understanding open scenes. The method first inputs the image to be segmented, and then performs block segmentation, embedding, and encoding on the input image using a frozen visual encoder to extract features of the image blocks to be aligned. Simultaneously, it inputs a set of categories corresponding to the current image, and extracts text prototype features corresponding to each category using a frozen text encoder. Then, in the last layer of the frozen visual encoder's multi-head self-attention submodule, Proxistorical alignment is performed on the query vector and key vector corresponding to the image block features to be aligned, resulting in aligned image block features. An initial segmentation score map is generated using the cosine similarity between these image block features and the text prototype features. Next, in the text-aware propagation module, the initial segmentation score map is downsampled to generate a compact grid. Simultaneously, a category relationship graph is constructed based on the similarity between text prototype features, and propagation weights are further constructed. Laplacian propagation is then performed on the compact grid. Finally, the propagated result is upsampled to the original image resolution, and the category with the highest score at each pixel position is selected to output the final segmentation result. This method achieves high-precision image segmentation in open vocabulary scenarios without requiring retraining, additional data, or an auxiliary backbone network.

[0018] In the method of this invention, both the frozen visual encoder and the frozen text encoder employ a pre-trained visual-language model encoding network, and their parameters remain unchanged throughout the implementation of this invention. The frozen text encoder is used to map the input category set to the text prototype features corresponding to each category.

[0019] A semantic alignment-based image segmentation method for open scene understanding includes the following steps:

[0020] S1. Input the natural scene image to be segmented into the frozen visual encoder to obtain aligned image patch features:

[0021] First, a single natural scene image to be segmented is obtained as the input image. If its resolution is too high and inference cannot be completed in one go due to memory limitations, it is divided into multiple overlapping image windows as input images, and each image window is processed separately as an input image. Let the height of the input image be... Width is The number of channels is The input image is then fed into a frozen visual encoder for feature extraction.

[0022] The frozen visual encoder includes a pre-feature extraction section and a first transformer encoding module. The pre-feature extraction section includes an image segmentation unit, an image block embedding unit, and several second transformer encoding modules. The first transformer encoding module and each layer of the second transformer encoding module have the same structure, each including a multi-head self-attention submodule and a feedforward submodule. The multi-head self-attention submodule is used to model the global correlation between different image blocks; the feedforward submodule is used to perform nonlinear mapping and feature transformation on the attention-processed features to enhance the expressive power of the image block features. The backbone network of the entire frozen visual encoder maintains frozen parameters during the implementation of this invention; only the calculation process of the multi-head self-attention submodule in the first transformer encoding module is rewritten.

[0023] Specifically, the image segmentation unit divides the input image into multiple non-overlapping image blocks, each image block having a size of [size missing]. ,in This represents the side length of the image patch. After division, the image is obtained vertically. Image patches, obtained horizontally Each image block is divided into several image blocks to form an image block grid. Subsequently, the image block embedding unit maps each image block to a corresponding feature representation, and then sends them sequentially to several transformer encoding modules before the last layer transformer encoding module for encoding processing to obtain the image block feature matrix to be aligned.

[0024] The number of rows in the feature matrix of the image patches to be aligned represents the length of the feature sequence, and the number of columns represents the feature dimension; wherein, in addition to containing the local features corresponding to each image patch, the feature sequence also contains a global classification feature, therefore the total number of rows is equal to Add the total number of image blocks. Take the feature rows corresponding to the positions of each image block in the feature matrix of the image blocks to be aligned to form the features of the image blocks to be aligned.

[0025] The first transformer encoding module performs Proxistor orthogonal alignment on the feature matrix of the image blocks to be aligned to obtain the aligned image block features. The specific process is as follows:

[0026] After obtaining the image patch features to be aligned, in the last layer of the frozen visual encoder's multi-head self-attention submodule, i.e., the multi-head self-attention submodule of the first transformer encoding module, Proxistoric orthogonal alignment processing is performed on the query vector matrix and key vector matrix corresponding to the image patch features to be aligned, to obtain the aligned image patch features. The purpose of this step is to correct the geometric offset in the visual feature space without changing the local feature magnitude and angular relationship, thereby improving the matching stability between subsequent image patch features and text prototype features.

[0027] Specifically, for any attention head, its corresponding query vector matrix, key vector matrix, and numerical vector matrix are taken as the processing objects. To reduce the interference of high-response features of the background region and global classification features on local geometric relationships, the query vector and key vector are first weighted and centered. Let the first... The query vector for each position is The key vector is Then define non-negative weights. It is proportional to the magnitude of the query vector and satisfies the condition that the sum of all weights is equal to the sum of all weights. Simultaneously, the weights of the corresponding positions of the global classification features can be set to... Based on this, the weighted centroids of the query vector and key vector are calculated separately, and the corresponding centroids are subtracted from the original query vector matrix and key vector matrix to obtain the centered query vector matrix and centered key vector matrix. The calculation formula is as follows:

[0028]

[0029]

[0030] in, These represent the weighted centroids of the query vector and the key vector, respectively. and These represent the centered query vector matrix and key vector matrix, respectively. Indicates all Column vector, This indicates the row number of the feature matrix of the image patch to be aligned.

[0031] After obtaining the centered query vector matrix and key vector matrix, the Proxte orthogonal alignment problem is constructed, which involves finding an orthogonal matrix such that the centered key vector matrix, after rotation, is as close as possible to the centered query vector matrix. The solution takes the following form:

[0032]

[0033] in, Represents the optimal orthogonal transformation matrix. express A set of 3D orthogonal matrices Denotes the variables of the orthogonal transformation matrix to be solved. This represents the Frobenius norm. Furthermore, for the matrix... By decomposing, we can obtain:

[0034]

[0035]

[0036] in, and These are the left orthogonal matrix and the right orthogonal matrix obtained from the decomposition, respectively. It is a diagonal matrix. This optimal orthogonal matrix can also be obtained through extreme decomposition and a small number of Newton-Schulz iterations to avoid direct singular value decomposition.

[0037] Then, the optimal orthogonal matrix obtained from the above decomposition is... Acting on the original bond vector matrix This yields the aligned key vector matrix, and the attention output feature matrix of the attention head is recalculated. The process is represented as follows:

[0038] ;

[0039] ;

[0040] in, This represents the aligned key vector matrix. This represents the recalculated attention weight matrix. This represents the aligned attention output feature matrix. These represent the original query vector matrix, the original key vector matrix, and the original numeric vector matrix, respectively. This is a scaling factor used to balance the impact of changes in feature dimensions on the inner product result.

[0041] Repeat the above operation for all attention heads, then concatenate and linearly project the output feature matrices of each attention head to obtain the geometrically corrected aligned image patch features. Since this process only performs orthogonal rotation on the key vectors, it does not destroy the amplitude relationship of local features, but it can make the directional distribution of image patch features closer to the query subspace, thereby improving the matching stability between image patches and text semantics.

[0042] S2. Input the set of user input categories corresponding to the current image into the frozen text encoder to extract text prototype features:

[0043] While extracting image patch features, a user input category set corresponding to the current input image is obtained. This category set is a set of candidate semantic categories input by the user during the inference stage based on the content of the image to be segmented, used to indicate the target categories that may be contained in the input image. This category set consists of multiple natural language category names, such as "cat" and "dog". This category set is input into the frozen text encoder to obtain text prototype features corresponding to each category. To facilitate subsequent similarity calculation, each text prototype feature output by the text encoder is normalized, i.e., its modulus is [value missing]. The text prototype features of all categories are stacked row by row to obtain a text prototype feature matrix. The number of rows in this matrix represents the number of categories, and the number of columns represents the text feature dimension. This text feature dimension is consistent with the aforementioned image patch feature dimension to facilitate image-text feature matching.

[0044] S3. Calculate the cosine similarity between the aligned image patch features and the text prototype features, and generate an initial segmentation score map:

[0045] Using aligned image patch features and text prototype features as input, the cosine similarity between the aligned image patch features and the text prototype features is calculated, and an initial classification score is obtained by scaling based on this. The position of the aligned image patch features is similar to that obtained after the input image is segmented. Each position in the image patch grid corresponds one-to-one.

[0046] Specifically, let the image patch grid be the first... Line number The aligned features corresponding to the column positions are , No. The text prototype features corresponding to each category are: Then the cosine similarity between the two vectors is the ratio of the vector dot product to the product of the magnitudes of the two vectors, that is:

[0047] ;

[0048] Then the image patch belongs to the first... The initial scores for each category are represented as follows:

[0049] ;

[0050] in, Represents the image block grid. Line number Column position relative to the first Initial classification scores for each category; This is a scaling factor, which takes the reciprocal of the square root of the feature dimension, and is used to make similarity results under different dimensions comparable; This represents the L2 norm.

[0051] By repeating the above calculations on the aligned image patch features at all locations and the text prototype features for all categories, an initial segmentation score map can be generated. This initial segmentation score map reflects the matching strength of each location to each category and serves as the input for the subsequent text-aware propagation module.

[0052] As described in step S1, if the input image cannot complete inference in one go due to insufficient video memory or limited hardware resources, the input image is divided into multiple overlapping image windows, and the initial segmentation score map corresponding to each image window is calculated separately. Then, the windows are fused according to their weights to obtain the initial segmentation score map corresponding to the entire input image. Let the number of windows be... , No. The score chart corresponding to each window is Window weight is Then the initial segmentation score after fusion can be expressed as:

[0053]

[0054] in, In the overlapping areas of each window, the weight sum is satisfied. The value is taken in the non-corresponding window area. This ensures the continuity and integrity of the output results in high-resolution scenarios.

[0055] S4: In the text-aware propagation module, the initial segmentation score map is downsampled to generate a compact grid.

[0056] The initial segmentation score map obtained in step S3 is fed into the text-aware propagation module. First, adaptive pooling averaging is performed on the initial segmentation score map to obtain a compact grid representation with lower resolution but retaining the main semantic structure. Let the height of this compact grid be... Width is The grid score map obtained after pooling is denoted as Each node in the grid score map corresponds one-to-one with a local region in the initial segmentation score map, and each node represents the aggregated score of each category within the corresponding local region. For the first node in the compact grid... The class distribution of each node can be obtained by softmax normalization:

[0057]

[0058] in, This represents the category score vector at that node. This represents the category probability distribution corresponding to the node. Through this step, the original pixel-level or image block-level initial segmentation score map can be transformed into a compact grid structure representation that facilitates subsequent propagation calculations, thereby reducing the complexity of subsequent propagation calculations while preserving the main semantic information.

[0059] S5: Calculate text prototype similarity and construct a category relationship graph:

[0060] After generating a compact grid, the similarity between categories is calculated using text prototype features to construct a category relationship graph. Let the text prototype feature matrix be... Then the category relationship diagram can be derived from the category relationship matrix. Represented as:

[0061]

[0062] in, This means performing a softmax operation on each row of the matrix. For temperature parameters, satisfying This is used to control the smoothness of the category similarity distribution; This is the diagonal enhancement factor, used to enhance the class's ability to retain itself; express The identity matrix, This represents the number of categories. After adding diagonal terms, the matrix is ​​normalized so that the sum of the elements in each row equals 1. The category relationship graph constructed through this step can represent the semantic proximity relationships between categories in the text space, providing a basis for the subsequent construction of propagation weights.

[0063] S6: Construct propagation weights:

[0064] The propagation weights are constructed based on the compact grid node information generated in step S4, combined with the category relationship graph constructed in step S5 and the grayscale differences of the original input image. The propagation weights consist of two parts: node confidence and propagation edge weights.

[0065] First, calculate the confidence score for each compact grid node. Let the first node be... The class probability distribution of the nodes is as follows: Then the maximum class probability of that node is taken as And calculate the relationship matrix between this node and the categories in step S5. The consistency between the represented textual semantic priors is Then, the two are combined to obtain the node confidence score:

[0066]

[0067] in, It is a very small positive number that satisfies This is used to prevent node confidence from degrading to 0 in extreme cases; Indicates the first The probability of the most trustworthy category of each node; This indicates the degree of consistency between the category distribution of the i-th node and the text prior. Indicates the first The confidence score of each node is used to measure which grid nodes have more reliable original classification results.

[0068] Secondly, calculate the propagation edge weights between nodes in the compact grid. For any pair of adjacent nodes... First, calculate the image edge intensity based on the grayscale differences of the original input image:

[0069]

[0070] in, The image edge suppression parameters satisfy... ; This indicates that the original input image grayscale image is at the node. With nodes The difference in grayscale between corresponding regions. The greater the difference in grayscale, the more likely that the location is at a boundary, and therefore the more its propagation should be suppressed.

[0071] Then, based on the category relationship matrix obtained in step S5 Calculate the text consistency gating based on the category probability distributions of the two nodes:

[0072] in, This means limiting the result to between 0 and 1. Represents a node With nodes Textual semantic consistency gating For the first The class probability distribution of a compact grid node. For the first The class probability distribution of a compact grid node. This is a category relationship matrix.

[0073] Finally, by combining image edge strength and text consistency gating, the propagation edge weights between adjacent nodes are obtained:

[0074]

[0075] in, For text consistency enhancement parameters, satisfy The propagation edge weight indicates that when two nodes are relatively smooth on the image boundary and relatively consistent in text semantics, the propagation between them is stronger; conversely, the propagation is weaker.

[0076] S7: Perform Laplace propagation on a compact mesh:

[0077] After constructing the propagation weights in step S6, Laplace propagation is performed on the compact mesh. Let the propagation edge weights be... The constructed weighted graph Laplacian matrix is The propagation results on the compact grid This is obtained by minimizing the following objective function:

[0078]

[0079] in, These represent the numbers after propagation and before propagation, respectively. The category score vector of each compact grid node; Indicates the node confidence level; For smoothing weight parameters, satisfying ; Represents a compact set of grid nodes; This represents the set of edges between adjacent nodes in a compact grid. The first term of the objective function is used to constrain the propagation result to be as close as possible to the original classification score of the high-confidence nodes, and the second term is used to constrain adjacent nodes to maintain smoothness and consistency when the image boundaries are weak and the text semantics are consistent, thereby achieving a structured refinement of the initial segmentation result.

[0080] Solving the above objective function yields the following normal equation:

[0081]

[0082] in, This represents a diagonal matrix with the confidence scores of each node in the compact grid as its diagonal elements. When at least one node has a confidence level greater than 0, the above linear system is a symmetric positive definite system, which can be efficiently solved using the conjugate gradient method of fixed order. This step allows for structured propagation and refinement on a low-resolution compact grid, thereby reducing computational overhead and maintaining semantic structure stability.

[0083] S8: Upsample to the original image resolution and take the category with the highest score:

[0084] The mesh propagation result is obtained in step S7. Then, the image is upsampled to the original input image resolution using bilinear interpolation to obtain the final category score map. Then, for each pixel location in the original image, the category with the highest score is selected as the segmentation category for that pixel, yielding the final segmentation result. Its expression is:

[0085]

[0086] in, Indicates the input image In category collection Under the conditions, No. Line number The final segmentation category of column pixels, This indicates that the pixel position belongs to the first... The final score for each category is then calculated. This completes the entire processing flow from the input image and input category set to the output segmentation result.

[0087] In summary, this invention provides a semantic alignment-based image segmentation method for open scene understanding. First, a frozen visual encoder and a frozen text encoder are used to extract image patch features and text prototype features to be aligned, respectively. Then, Proxist orthogonal alignment is performed in the last layer of the frozen visual encoder's multi-head self-attention submodule to obtain aligned image patch features. Next, the cosine similarity between the aligned image patch features and the text prototype features is used to generate an initial segmentation score map. Furthermore, a structured Laplacian propagation is performed on a compact grid through a text-aware propagation module. Finally, the image is upsampled to the original resolution, and the segmentation result is output. This method expands linguistic information from simple category labels to structural priors involved in propagation modeling, thereby improving the semantic consistency and boundary integrity of open-vocabulary image segmentation without requiring retraining.

[0088] Example 1

[0089] To verify the effectiveness and rationality of this invention, comparative experiments were conducted on seven standard benchmark datasets according to the training-free open-vocabulary semantic segmentation evaluation protocol, covering both settings with and without explicit background classes. The datasets included: Pascal VOC 21 (V21, 21 classes), Pascal Context 60 (PC60, 60 classes), COCO-Object (Object, derived from MS-COCO with 80 object categories), Pascal VOC 20 (V20, 20 classes), PascalContext 59 (PC59, 59 classes), COCO-Stuff (Stuff, 171 classes), and Cityscapes (City, 19 classes). Specifically, the method of this invention strictly loads and applies only to the frozen visual-language backbone network during the testing phase, without introducing any auxiliary backbone network, performing any additional training, or using any external supervision. By default, both the visual encoder and text encoder use CLIP ViT-B / 16. This invention introduces text-aware Laplace propagation, performs category logit refinement by solving a small-scale symmetric positive definite linear system on a compact grid, and performs a fixed number of conjugate gradient iterations before final upsampling. The grid size... The default setting is: City uses The remaining datasets use To avoid over-smoothing of thin structures and fine-grained "stuff" details in high-resolution scenes. All experiments were conducted on a single NVIDIA V100 (32GB) image, with the input image uniformly resized to a shorter side of 336 pixels (560 pixels for the shorter side in City due to its higher base resolution). When the entire image could not fit into the video memory, sliding window inference was used: the cropping window size was... The step size is 112. All results are obtained using single-scale inference (without multi-scale or flips). To ensure fair comparison, no post-processing refinement modules such as DenseCRF or PAMR are used in any reported metrics. The performance metric primarily uses the mean intersection-union ratio (mIoU), supplemented by pixel accuracy (pAcc) as a secondary evaluation.

[0090] Figure 2 The results show a comparison of the attention response of the method of this invention with other representative methods (CLIP, NACLIP) at selected query points, where (1) is a 336×336 resolution input and (2) is a 512×512 resolution input. The attention response distribution of the CLIP method is relatively scattered and biased towards the background region. Although the NACLIP method enhances local focusing, it is easy to split the same target into multiple isolated regions. However, the method of this invention introduces an orthogonal alignment mechanism in the last layer of the self-attention module, making the attention response more concentrated, continuous and consistent with the target region, while effectively suppressing background interference and maintaining the integrity of fine structures. The above advantages can be maintained at different resolutions, which reflects the superiority of the method of this invention in terms of attention stability and semantic consistency.

Claims

1. A semantic alignment-based image segmentation method for open scene understanding, characterized in that, include: The natural scene image to be segmented is input into the frozen visual encoder to obtain aligned image patch features; Input the set of user input categories corresponding to the current image into the frozen text encoder to extract text prototype features; Calculate the cosine similarity between the aligned image patch features and the text prototype features, and generate an initial segmentation score map; The initial segmentation score map is downsampled to generate a compact grid; a category relationship graph is constructed based on the similarity between text prototype features, and propagation weights are further constructed; Laplacian propagation is performed on the compact grid to obtain the grid propagation result; The grid propagation results are upsampled to the original image resolution using bilinear interpolation to obtain the final category score map.

2. The image segmentation method based on semantic alignment for open scene understanding according to claim 1, characterized in that, The frozen visual encoder includes a pre-stage feature extraction part and a first transformer encoding module. The pre-stage feature extraction part includes an image segmentation unit, an image block embedding unit, and several second transformer encoding modules. The image segmentation unit is used to divide the input image into multiple non-overlapping image blocks. The image block embedding unit is used to map each image block to a corresponding feature representation, and then sequentially send them to several second transformer encoding modules for encoding processing to obtain the image block feature matrix to be aligned. The first transformer encoding module performs Proxistor orthogonal alignment on the feature matrix of the image block to be aligned to obtain the aligned image block features.

3. The image segmentation method based on semantic alignment for open scene understanding according to claim 2, characterized in that, The first transformer encoding module and each layer of the second transformer encoding module have the same structure, both including a multi-head self-attention submodule and a feedforward submodule. The multi-head self-attention submodule is used to model the global correlation between different image patches; the feedforward submodule is used to perform nonlinear mapping and feature transformation on the attention-processed features to enhance the expressive power of image patch features.

4. The image segmentation method based on semantic alignment for open scene understanding according to claim 3, characterized in that, The specific method used by the first transformer encoding module to obtain the aligned image block features by performing Proxistor orthogonal alignment on the feature matrix of the image block to be aligned is as follows: In the multi-head self-attention submodule, each attention head performs weighted centering on the query vector and key vector: Let the first The query vector for each position is The key vector is Then define non-negative weights. It is proportional to the magnitude of the query vector and satisfies the condition that the sum of all weights is equal to the sum of all weights. Simultaneously, the weights of the corresponding positions of the global classification features are set to... ; Calculate the weighted centroids of the query vector and key vector respectively, and subtract the corresponding centroids from the original query vector matrix and key vector matrix to obtain the centered query vector matrix and centered key vector matrix. The specific calculation formula is as follows: ； ； in, These represent the weighted centroids of the query vector and the key vector, respectively. and These represent the centered query vector matrix and key vector matrix, respectively. Indicates all Column vector, The original bond vector matrix, This represents the original query vector matrix. This indicates the row number of the feature matrix of the image patch to be aligned; Construct the Proxist orthogonal alignment problem, solve the Proxist orthogonal alignment problem, and make the centered key vector matrix approximate the centered query vector matrix after rotation. The solution is in the form of: ； in, Represents the optimal orthogonal transformation matrix. express A set of 3D orthogonal matrices Denotes the variables of the orthogonal transformation matrix to be solved. Represents the Frobenius norm; For matrix Decomposition yields: ； ； in, and These are the left orthogonal matrix and the right orthogonal matrix obtained from the decomposition, respectively. It is a diagonal matrix; The optimal orthogonal matrix is ​​obtained through extreme decomposition and a small number of Newton-Schultz iterations. ; The optimal orthogonal matrix obtained from the above decomposition Acting on the original bond vector matrix The aligned key vector matrix is ​​obtained, and the attention output feature matrix of the attention head is recalculated. The specific process is as follows: ； ； ； in, This represents the aligned key vector matrix. This represents the recalculated attention weight matrix. This indicates that the attention output feature matrix represents the features of the aligned image patches. Represents the original numerical vector matrix. This is the scaling factor; The output feature matrices of each attention head are concatenated and linearly projected to obtain the aligned image patch features after geometric correction.

5. The image segmentation method based on semantic alignment for open scene understanding according to claim 1, characterized in that, The frozen text encoder structure is the same as the frozen visual encoder structure. The frozen text encoder takes the user-input category set as input, obtains text prototype features corresponding to each category, performs unitization on each text prototype feature, and stacks the text prototype features of all categories row by row to obtain a text prototype feature matrix.

6. The image segmentation method based on semantic alignment for open scene understanding according to claim 1, characterized in that, Initial segmentation score image patch grid Line number Column position relative to the first Initial classification scores for each category Specifically: ； In the formula, For the image block grid, the first Line number Aligned image patch features corresponding to column positions For the first Text prototype features corresponding to each category This is the scaling factor. This represents the L2 norm.

7. The image segmentation method based on semantic alignment for open scene understanding according to claim 1, characterized in that, The category relationship diagram is composed of a category relationship matrix. Specifically, it means: ； in, Represents the text prototype feature matrix Perform a softmax operation on each row. For temperature parameters, satisfying ; This is the diagonal enhancement factor; express The identity matrix, Indicates the number of categories.

8. The image segmentation method based on semantic alignment for open scene understanding according to claim 1, characterized in that, The specific method for constructing propagation weights is as follows: Calculate the confidence level for each compact grid node: Let the first node be... The class probability distribution of the compact grid nodes is as follows: Then the maximum class probability of the compact grid is taken as Represent the category index and calculate the compact grid-to-category relationship matrix. The consistency between the prior semantics of the texts represented is specifically as follows: ; The node confidence scores are obtained by combining the results: ； in, It is a very small positive number that satisfies ; Indicates the first The probability of the most trustworthy category of each node; This indicates the degree of consistency between the category distribution of the i-th node and the text prior. Indicates the first The confidence level of each node; Calculate the propagation weights between nodes in the compact grid The specific formula is as follows: ； In the formula, Image edge intensity, For text consistency gating. For text consistency enhancement parameters, satisfy .

9. The image segmentation method based on semantic alignment for open scene understanding according to claim 8, characterized in that, The text consistency gating specifically refers to: ； in, This means limiting the result to between 0 and 1. Represents a node With nodes Textual semantic consistency gating For the first The class probability distribution of a compact grid node. For the first The class probability distribution of a compact grid node. This is a category relationship matrix.

10. The image segmentation method based on semantic alignment for open scene understanding according to claim 8, characterized in that, The specific process of performing Laplace propagation on the compact mesh to obtain the mesh propagation result is as follows: By propagation edge weight The weighted graph Laplacian matrix is ​​constructed. ; The propagation results on the compact grid are obtained by minimizing the following objective function. : in, These represent the numbers after propagation and before propagation, respectively. The category score vector of each compact grid node; Indicates the node confidence level; For smoothing weight parameters, satisfying ; Represents a compact set of grid nodes; This represents the set of adjacent node edges in a compact mesh.

Images