An open-vocabulary segmentation method based on parallel cost aggregation and expert perception
By using parallel cost aggregation and expert-aware learning modules, the problem of knowledge interference caused by serial aggregation is solved, the alignment capability of open vocabulary semantic segmentation and component segmentation is improved, and significant performance improvements are demonstrated on multiple benchmarks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV OF SCI & TECH
- Filing Date
- 2026-04-30
- Publication Date
- 2026-06-05
AI Technical Summary
In existing open-vocabulary semantic segmentation methods, serial cost aggregation leads to knowledge interference between category-level semantics and spatial structure information, affecting image-text alignment capabilities, and incurring significant computational overhead.
We employ a parallel cost aggregation and expert-driven perceptual learning module. By parallelizing spatial aggregation and category aggregation and combining feature orthogonality decoupling strategy, we reduce the redundancy of category semantics and spatial context features. We also introduce an expert-driven perceptual learning module to adaptively fuse multi-source information.
It improves the alignment learning capabilities of open-vocabulary semantic segmentation and component segmentation, reduces computational overhead, and demonstrates significant performance improvements on multiple benchmarks.
Smart Images
Figure CN122154691A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of open vocabulary segmentation technology, and in particular to an open vocabulary segmentation method based on parallel cost aggregation and expert perception. Background Technology
[0002] Traditional open-vocabulary semantic segmentation methods are usually limited by a predefined set of categories, making it difficult to adapt to category variations in open-world visual environments. In recent years, the visual-language foundation model CLIP has shown significant open-vocabulary recognition capabilities after being trained on large-scale image-text pair datasets. Based on this, existing research has adapted the visual foundation model, originally used for image-level recognition, to dense prediction tasks, thereby promoting the development of open-vocabulary semantic segmentation and open-vocabulary component segmentation, enabling the model to classify and segment arbitrary categories.
[0003] In open-vocabulary semantic segmentation, early methods typically froze CLIP parameters in a two-stage framework to maintain their strong visual-language alignment capabilities. However, such methods have high computational overhead, limiting scalability and hindering further research. In contrast, current mainstream methods adopt an end-to-end learning paradigm, achieving dense predictions by fine-tuning the attention layer of CLIP.
[0004] Existing end-to-end methods typically construct a similarity matrix from visual and text encoder features and perform two consecutive operations on it: spatial aggregation and category aggregation. Typically, spatial aggregation and category aggregation are performed sequentially in a serial structure to capture spatial context and category semantic information. Although the above design can achieve good performance, its serial architecture introduces knowledge interference between category-level semantics and spatial structure information, thereby affecting the further refinement of cost volume and weakening image-text alignment ability. When spatial aggregation is performed before category aggregation, the model can capture contextual structure information, but it may cause category semantics to shift. The subsequent category aggregation will further amplify this semantic bias, resulting in region misclassification.
[0005] Therefore, how to more effectively aggregate spatial context and category semantic information in terms of cost volume, reduce the interference between the two, and extract richer image-text alignment cues is an important problem in open lexical semantics and component segmentation. Summary of the Invention
[0006] To address the knowledge interference problem caused by serial cost aggregation in open vocabulary semantics and component segmentation tasks, this application provides an open vocabulary segmentation method based on parallel cost aggregation and expert perception. By parallelizing spatial aggregation and category aggregation, richer image-text alignment cues are extracted from the cost volume. Simultaneously, an expert-driven perception learning module achieves efficient fusion of image and text knowledge flows, and a feature orthogonal decoupling strategy reduces redundancy between category semantics and spatial context features, thereby improving the alignment learning capability and segmentation performance of the segmentation method.
[0007] This application provides an open vocabulary segmentation method based on parallel cost aggregation and expert perception, including:
[0008] Step S10, given the input image and its corresponding text description Image features and text embeddings are extracted using a visual encoder and a text encoder, respectively, and the matching degree between the two is calculated to generate the cost volume. The cost volume embedding is obtained through convolutional mapping. ; Step S20, for the cost body embedding of the first Input at each stage Spatial context features are obtained by simultaneously computing spatial aggregation operators and category aggregation operators using a parallel architecture. and category semantic features Furthermore, by employing a feature orthogonal decoupling strategy, the relationship between spatial contextual knowledge and category-level semantics is constrained. Step S30: The spatial context features obtained in step S20 are... and category semantic features Input to expert-driven perceptual learning module The module extracts diverse and complementary information through a multi-expert parser and adaptively fuses multiple features output by the multi-expert parser using coefficients learned by a coefficient mapper to produce a unified feature embedding. ; Step S40: Embed the unified feature The prediction results are generated by the decoder.
[0009] Furthermore, the visual encoder and text encoder are derived from a pre-trained visual-language model CLIP, with the visual encoder generating visual embeddings. Text encoder generates text embeddings ,in, and These represent the height and width of the feature map, respectively. Indicates the total number of categories. Indicates the feature dimension; To improve the accuracy of object localization, the final attention layer of the visual encoder is replaced with two convolutional kernels of size [missing value]. The convolutional layer.
[0010] Furthermore, to distinguish it from the sequential processing method of spatial aggregation followed by category aggregation in the serial structure, a parallel approach is adopted to process the cost body embedding, so that spatial aggregation and category aggregation are executed simultaneously for the same input, thereby forming two independent knowledge streams.
[0011] Furthermore, the category-level semantic and spatial context knowledge streams of the parallel architecture are designed to complement each other, thereby enhancing the alignment capabilities of images and text. To this end, an expert-driven perceptual learning module efficiently integrates the two knowledge streams. In the module, a multi-expert parser decouples and refines the category-level semantic and spatial context knowledge, extracting diverse and complementary information. Then, these features are adaptively fused using coefficients learned by a coefficient mapper to produce a unified and robust feature embedding.
[0012] Furthermore, the category-level semantics and spatial context knowledge represent information across two different dimensions, and the relationship between these two knowledge flows should remain independent. This is a key factor for the success of the expert-driven perceptual learning module. More specifically, the two unrelated feature representations ensure that the EPL can learn the diversity and complementary information required for segmentation from a broad feature space. Therefore, this invention proposes a feature orthogonality decoupling strategy to constrain the relationship between category-level semantics and spatial context knowledge, thereby promoting orthogonality.
[0013] This application discloses the following technical effects: Regarding open vocabulary segmentation capabilities, this invention employs a parallel cost aggregation design to enable category-level semantic learning and spatial structure context modeling to proceed independently. This alleviates the knowledge interference problem caused by cascading propagation in serial structures, promotes the extraction of richer image-text alignment cues from the cost body, and enhances the overall performance of open vocabulary semantic segmentation and open vocabulary component segmentation.
[0014] In terms of knowledge diversity and complementary information utilization, this invention introduces an expert-driven perceptual learning mechanism. By having multiple experts analyze and integrate semantic and contextual information from different perspectives, and using pixel-level adaptive weights for fusion, this invention fully utilizes multi-source complementary knowledge to obtain a more robust unified feature representation.
[0015] In terms of redundancy suppression and decoupling learning, this invention further adopts a feature orthogonal decoupling strategy to constrain the correlation between the two-way representation of category semantics and spatial context, reduce the redundancy of the two-way features, provide a more independent source of information for multi-expert modules, thereby enhancing the learning of diverse features and improving fine-grained alignment capabilities.
[0016] In terms of balancing computational overhead and performance gains, this invention maintains a relatively lightweight structural increment while introducing parallel aggregation and supporting modules, achieving significant performance improvements within an acceptable resource overhead range, and is suitable for efficient image-text alignment modeling in open vocabulary segmentation tasks. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating an open vocabulary segmentation method based on parallel cost aggregation and expert perception, provided in Embodiment 1 of this application.
[0018] Figure label explanation: EPL, Expert-Driven Perceptual Learning Module.
[0019] Figure 2 This is a schematic diagram of the parallel architecture in the open vocabulary segmentation method based on parallel cost aggregation and expert perception provided in Embodiment 1 of this application.
[0020] Figure 3 This provides an analysis of the knowledge coupling between category-level semantics and spatial context under different designs in Embodiment 1 of the present invention.
[0021] Figure 4 This provides a redundancy analysis of feature knowledge between different expert blocks as provided in Embodiment 1 of the present invention. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application will be provided in conjunction with the accompanying drawings. The described embodiments should not be considered as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0023] Example 1: This application provides an open vocabulary segmentation method based on parallel cost aggregation and expert perception, such as... Figure 1 As shown, the method includes: Step S10, given the input image and its corresponding text description Image features and text embeddings are extracted using a visual encoder and a text encoder, respectively, and the matching degree between the two is calculated to generate the cost volume. The cost volume embedding is obtained through convolutional mapping. .
[0024] In this embodiment, an input image is first given. and its corresponding text description (Including a set of category names), image features are extracted using the visual encoder of the pre-trained visual-language model CLIP, while text embeddings are extracted using the text encoder: Visual encoder generates visual embeddings Text encoder generates text embeddings ,in, and These represent the height and width of the feature map, respectively. Indicates the total number of categories. The feature dimension is represented; to improve the accuracy of object localization, this embodiment replaces the final attention layer of the visual encoder with two convolutional kernels of size [missing information]. Convolutional layers; Next, the matching degree between visual embeddings and text embeddings is calculated based on cosine similarity, the two embedding features are aligned, and the cost volume is generated. The specific calculation formula is as follows:
[0025] in, Represents visual embedding. 1 pixel With text embedding Categories Cosine similarity between them The L2 norm of the embedded features; Finally, the generated cost volume With convolution kernel size The convolution is performed to map it to a cost volume embedding. This provides input for subsequent parallel aggregation.
[0026] Step S20, for the cost body embedding of the first Input at each stage Spatial context features are obtained by simultaneously computing spatial aggregation operators and category aggregation operators using a parallel architecture. and category semantic features Furthermore, by employing a feature orthogonal decoupling strategy, the relationship between spatial contextual knowledge and category-level semantics is constrained.
[0027] In this embodiment, to distinguish it from the sequential processing method of "first spatial aggregation and then category aggregation" in the serial structure, as follows: Figure 2 As shown, this invention uses a parallel approach to process cost volume embedding, enabling spatial aggregation and category aggregation to be executed simultaneously for the same input, forming two independent knowledge flows; at the same time, feature orthogonality decoupling (dashed line part) is used to force the features of the two branches to be orthogonal (decoupling), thereby preserving more accurate image-text alignment information.
[0028] Specifically, for the cost volume embedding of the first Input at each stage A parallel architecture is adopted, using spatial aggregation operators respectively. With category aggregation operator Simultaneously, the following calculations were performed:
[0029] in, Indicates the first Spatial context features generated at each stage Indicates the first Each stage generates category semantic features, and each pair of spatial aggregation operators and category aggregation operators To form a parallel cost aggregation block, the spatial aggregation operator in this embodiment adopts the Swing Transformer structure, and the category aggregation adopts a linear variant to reduce complexity; At the same time, in spatial context features and category semantic features A feature orthogonality decoupling strategy is introduced to constrain the relationship between category-level semantics and spatial context knowledge, thereby promoting orthogonality. The correlation between the two features is measured using cosine similarity, expressed by the formula:
[0030] in, Representation of spatial context features and category semantic features The same position pixel features and Minimizing the similarity between two pieces of knowledge can reduce the correlation between them; however, the presence of negative values in the cosine similarity can cause gradient backpropagation to become inaccurate, leading the optimization in the wrong direction. To address this issue, the following feature orthogonality decoupling loss is introduced:
[0031] in, The orthogonality of the features decouples the loss, and the squared term ensures that the loss is positive and converges to zero, thereby promoting the orthogonality between category-level semantics and spatial context knowledge.
[0032] Figure 3 This is a knowledge coupling analysis between category-level semantics and spatial context under different designs. The horizontal axis represents the number of training iterations, and the vertical axis represents feature similarity. The similarity of the serial method (serial cost aggregation, dashed line) is greater than 88%, indicating that spatial and category information are entangled. However, the similarity of the parallel method of this invention (parallel cost aggregation, solid line) drops rapidly and approaches 0, proving that the feature orthogonal resolution strategy has successfully achieved feature decoupling.
[0033] Step S30: The spatial context features obtained in step S20 are... and category semantic features Input to expert-driven perceptual learning module The module extracts diverse and complementary information through a multi-expert parser and adaptively fuses multiple features output by the multi-expert parser using coefficients learned by a coefficient mapper to produce a unified feature embedding. .
[0034] In this embodiment, the expert-driven perceptual learning module decouples and refines category-level semantic and spatial context knowledge through a multi-expert parser: First, connect the spatial context features along the channel dimension. and category semantic features Obtain features Subsequently, multiple expert blocks deal with, Indicates the total number of expert blocks. For expert block indexing; to extract complementary knowledge representations from different perspectives. This can be expressed as a formula:
[0035] Each expert block uses the same structure, including two convolutional operations, a batch normalization layer in between, and a GeLU activation function following the output; the number of expert blocks in this embodiment... To ensure optimal performance; Figure 4 The analysis of feature knowledge redundancy among different expert blocks shows the similarity between features extracted by different expert modules. The similarity is generally less than 50%, indicating that each expert has indeed learned different and complementary knowledge, rather than repetitive information, thus verifying the necessity of the multi-expert design.
[0036] The next step is to fuse and weight the outputs of the multi-expert parsers to obtain fine-grained and robust feature embeddings. Unlike previous methods that applied fixed weights for feature aggregation, this embodiment uses a coefficient mapper to autonomously learn and assign pixel-level weights to the features of each expert block's parsing output in a lightweight and adaptive manner, ensuring effective integration of diverse knowledge while preserving complementary information. This includes the following detailed steps: Features Input coefficient mapper Generation coefficient This can be expressed as a formula:
[0037] The coefficient mapper consists of two convolutional layers with a ReLU activation function in between, followed by a softmax function to ensure that the sum of the coefficients is 1; its parameter count is... The coefficient mapper is used to assign pixel-level weights, adaptively learning and integrating complementary knowledge from multi-expert analysis. Finally, for Pixel-level weighted aggregation is performed to obtain a unified feature embedding. This can be expressed as a formula:
[0038] Step S40: Embed the unified feature The prediction results are generated by the decoder.
[0039] In this embodiment, unified feature embedding via decoder Generate prediction results relative to the true labels The monitoring loss is expressed by the formula:
[0040] in, Indicates monitoring losses, The cross-entropy operator is used to combine the supervised loss and the feature orthogonality decoupling loss to form the total loss for parameter optimization, expressed by the formula:
[0041] in, Represents the total loss function. To decouple the loss weights for feature orthogonality, this embodiment sets them to 0.01. During training, only the relevant parameters of the query matrix and value matrix in the visual and text encoders are updated, and the parameters of the external module of the encoder are updated synchronously to reduce training overhead and maintain the alignment capability of images and text.
[0042] Example 2: To verify the effectiveness of the method of the present invention in open vocabulary semantic segmentation tasks, this embodiment of the application was implemented and evaluated in a unified experimental environment: In this embodiment, an open vocabulary segmentation model based on parallel cost aggregation and expert perception was built using PyTorch and Detectron2. All experiments were completed on eight NVIDIA A6000 GPUs, with AdamW as the optimizer and a batch size of 8. For open-vocabulary semantic segmentation, the model was trained on the COCO-Stuff dataset, which contains 118k high-quality annotations covering 171 categories. After training, the model was evaluated on standard benchmarks such as ADE20K, PASCAL-Context, and PASCAL VOC, and organized by category as A-847, A-150, PC-459, PC-59, PAS-20, and PAS. The evaluation metric used is mIoU, which means that the intersection-union ratio (IoU) is calculated for each category that needs to be segmented, and then the average value is taken. The intersection-union ratio is the area of the intersection between the predicted region and the real region divided by the area of the union between the predicted region and the real region.
[0043] ViT-B / 16 (first 13 rows in Table 1) and ViT-L / 14 (rows 14-23 in Table 1) were used as backbone networks, with a learning rate of [missing information]. The method was trained for 80,000 iterations, and the comparison results are shown in Table 1. The method of this invention achieved better mIoU performance on six open vocabulary semantic segmentation benchmarks.
[0044] Table 1. Performance comparison of the method of the present invention with other state-of-the-art methods on multiple open-vocabulary semantic segmentation benchmarks in Example 2.
[0045] As shown in row 13 of Table 1, when ViT-B / 16 is used as the backbone, the method of the present invention applies to A-847, PC-459, A-150, PC-59, PAS-20, and PAS. The mIoU values on the above six benchmarks are 16.1, 22.3, 38.1, 62.2, 97.3 and 82.7, respectively. The method of the present invention also provides the improvement of mIoU relative to H-CLIP on the above six benchmarks as 3.6%, 2.9%, 5.7%, 4.3%, 2.1% and 4.5%, respectively, and the improvement of mIoU relative to DeCLIP as 0.8%, 0.9%, 1.8%, 1.6%, 0.7% and 1.4%, respectively.
[0046] As shown in row 23 of Table 1, when using the ViT-L / 14 backbone, the method of the present invention applies to A-847, PC-459, A-150, PC-59, PAS-20, and PAS. The mIoU values were 18.3, 26.7, 41.5, 64.7, 98.4 and 84.4, respectively.
[0047] Example 3: To verify the effectiveness of the method of the present invention in the open vocabulary component segmentation task, this embodiment of the application was implemented and evaluated in a unified experimental environment: In this embodiment, an open vocabulary segmentation model based on parallel cost aggregation and expert perception was built using PyTorch and Detectron2. All experiments were completed on eight NVIDIA A6000 GPUs, with AdamW as the optimizer and a batch size of 8. In terms of open vocabulary component segmentation, this invention evaluates its method on two benchmarks: Pascal-Part-116 and ADE20K-Part-234. It employs two protocols: Pred-All and Oracle-Obj. In addition to mIoU, the harmonic mean intersection-union ratio (h-IoU) between the seen and unseen categories is used to measure generalization ability.
[0048] The experimental setup used ViT-B / 16 as the backbone network, with a Pascal-Part-116 learning rate of [missing information]. It was trained for 30,000 iterations, with an ADE20K-Part-234 learning rate of [missing information]. And train it for 20,000 iterations.
[0049] The experimental results on Pascal-Part-116 are shown in Table 2. The seen, unseen, and h-IoU values of the method of this invention are 57.2, 43.5, and 49.3 in the Pred-All setting, and 60.5, 47.1, and 52.9 in the Oracle-Obj setting, respectively. Compared with PartCATSeg, the h-IoU is improved by 3.5% and 2.5% in the Pred-All and Oracle-Obj settings, respectively.
[0050] Table 2. Zero-shot performance comparison of the method of the present invention with other state-of-the-art methods on the Pascal-Part-116 dataset for open-vocabulary component segmentation in Example 3.
[0051] Note: Pas-Pr and Pas-Or represent the use of the Pred-All and Oracle-Obj protocols respectively on the Pascal-Part-116 dataset. The experimental results on ADE20K-Part-234 are shown in Table 3. The results of the method of this invention are 40.8, 19.2 and 26.1 under the Pred-All settings, and 56.3, 48.2 and 51.8 under the Oracle-Obj settings; which are 1.9% and 1.8% higher than PartCATSeg, respectively.
[0052] Table 3. Zero-shot performance comparison of the method of the present invention with other state-of-the-art methods on the ADE20K-Part-234 dataset for open-vocabulary component segmentation in Example 3.
[0053] Note: ADE-Pr and ADE-Or respectively represent the use of the Pred-All and Oracle-Obj protocols on the ADE20K-Part-234 dataset. Example 4: To further verify the contribution of the key structures and components of the present invention, this example conducts a comparative analysis of parallel and serial cost aggregation, and provides complexity calculation and ablation results.
[0054] As shown in Table 4, the complexity analysis of serial and parallel blocks indicates that the parameters, number of floating-point operations (FLOPs), memory, and time for the serial block are 0.685M, 53.67G, 0.60G, and 0.11s, respectively, while those for the parallel block are 1.03M, 88.07G, 1.56G, and 0.19s. The incremental difference between the parallel and serial blocks is 0.35M parameters, an additional 34.4G FLOPs, an additional 0.96G memory, and a single-graph latency of 0.08s.
[0055] Table 4. Complexity analysis of each serial and parallel block of the method of the present invention under the same experimental settings.
[0056] Regarding component ablation, this embodiment, based on the parallel architecture proposed in this invention, only introduces EPL to enable PAS. The improvement from 81.1 to 82.0 can be achieved by introducing only feature orthogonality decoupling to 82.1, and by introducing both features together to 82.7. At the same time, A-150 is improved from 36.4 to 38.1. In the serial architecture, the corresponding improvement is smaller. The A-150 of the serial baseline structure is 36.3. Introducing only EPL can improve it to 36.6, introducing only feature orthogonality decoupling can improve it to 36.6, and introducing both features together can improve it to 36.7.
[0057] Parameter and fusion strategy analysis shows that the number of expert blocks Performance continued to improve from 1 to 4 and When A-150 reaches 37.2, the characteristic orthogonality decoupling loss weights At a value of 0.010, A-150 is 38.1 and PC-59 is 62.2, which are optimal. In the multi-expert fusion strategy, the coefficient mapper is 61.2 on PC-59, which is higher than the convolution, averaging and accumulation methods.
[0058] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application. In some cases, the actions or steps described in this application can be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results.
Claims
1. An open vocabulary segmentation method based on parallel cost aggregation and expert perception, characterized in that, The method includes: Step S10, given the input image and its corresponding text description Image features and text embeddings are extracted using a visual encoder and a text encoder, respectively, and the matching degree between the two is calculated to generate the cost volume. The cost volume embedding is obtained through convolutional mapping. ; Step S20, for the cost body embedding of the first Input at each stage Spatial context features are obtained by simultaneously computing spatial aggregation operators and category aggregation operators using a parallel architecture. and category semantic features Furthermore, by employing a feature orthogonal decoupling strategy, the relationship between spatial contextual knowledge and category-level semantics is constrained. Step S30: The spatial context features obtained in step S20 are... and category semantic features Input to expert-driven perceptual learning module The module extracts diverse and complementary information through a multi-expert parser and adaptively fuses multiple features output by the multi-expert parser using coefficients learned by a coefficient mapper to produce a unified feature embedding. ; Step S40: Embed the unified feature The prediction results are generated by the decoder.
2. The open vocabulary segmentation method based on parallel cost aggregation and expert perception as described in claim 1, characterized in that, In step S10, the visual encoder and text encoder are derived from the pre-trained visual-language model CLIP, and the visual encoder generates visual embeddings. Text encoder generates text embeddings ,in, Represents a matrix tensor. and These represent the height and width of the feature map, respectively. Indicates the total number of categories. Indicates the feature dimension; Replace the final attention layer of the visual encoder with two convolutional kernels of size [size missing]. The convolutional layer.
3. The open vocabulary segmentation method based on parallel cost aggregation and expert perception as described in claim 2, characterized in that, The degree of matching between the visual embedding and the text embedding is represented by a cosine similarity metric, and the similarity calculation formula is as follows: in, Represents visual embedding. 1 pixel With text embedding Categories Cosine similarity between them is the L2 norm of the embedded features.
4. The open vocabulary segmentation method based on parallel cost aggregation and expert perception as described in claim 1, characterized in that, In step S20, in spatial context features and category semantic features A feature orthogonality decoupling strategy is introduced. The correlation between two features is measured using cosine similarity. Minimizing this similarity reduces the correlation between the two feature knowledge sets. The following feature orthogonality decoupling loss is introduced: in, and These represent the height and width of the feature map, respectively. Representation of spatial context features and category semantic features In the same position Similarity per pixel The characteristic orthogonality decoupling loss is represented by the squared term, which ensures that the loss is positive and converges to zero.
5. The open vocabulary segmentation method based on parallel cost aggregation and expert perception as described in claim 1, characterized in that, In step S30, the multi-expert parser decouples and refines category-level semantics and spatial context knowledge, including: First, connect the spatial context features along the channel dimension. and category semantic features Obtain features Subsequently, multiple expert blocks deal with, Indicates the total number of expert blocks. For expert block indexing, extract complementary knowledge representations from different perspectives. Each expert block uses the same structure, including two convolution operations, a batch normalization layer in between, and the output is followed by the GeLU activation function.
6. The open vocabulary segmentation method based on parallel cost aggregation and expert perception as described in claim 1, characterized in that, In step S30, the coefficient mapper adaptively learns and assigns pixel-level weights to the features parsed by each expert block, integrating diverse knowledge while retaining complementary information. Features Input coefficient mapper Generation coefficient , Indicates the total number of expert blocks. For expert block indexing, the coefficient mapper consists of two convolutional layers with a ReLU activation function in between, followed by a softmax function mapping to ensure that the sum of the coefficients is 1.
7. An open vocabulary segmentation method based on parallel cost aggregation and expert perception as described in claim 1 or 4, characterized in that, In step S40, unified feature embedding via decoder Generate prediction results relative to the true labels This results in a loss of oversight. The supervised loss and the feature orthogonality decoupling loss are combined to form the total loss for parameter optimization. During training, only the relevant parameters of the query matrix and value matrix in the visual and text encoders are updated, and the parameters of the external module of the encoder are updated synchronously to reduce training overhead and maintain the alignment ability of images and text.