Robust test-time knowledge transfer method for open-vocabulary semantic segmentation

Abstract

The application discloses a robust test-time knowledge transfer method and device for open vocabulary semantic segmentation, and relates to the technical field of computer vision and image semantic segmentation. The method comprises the following steps: a visual-language model is used to extract multi-modal features of a test image and an open category, a semantic response graph is generated through feature similarity, and an initial prediction of a source model is obtained; a high response area of the response graph is selected as a prompt input to segment a model, candidate areas are generated, and reliable indicators are calculated; a double threshold is dynamically configured according to historical data, the areas are divided into three categories of reliable, ambiguous and noise, and are respectively stored in three types of caches; ranking, sampling and prototype selection are used to update and simplify cache features; three types of prediction results are obtained by matching images and cache features, the reliable and ambiguous branch results are weighted and fused to obtain enhanced prediction, and finally, the initial prediction is fused and optimized to complete open vocabulary semantic segmentation reasoning. The application can improve the segmentation robustness under complex domain shift conditions.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image semantic segmentation technology, and in particular to a robust knowledge transfer method and apparatus for open-vocabulary semantic segmentation during testing. Background Technology

[0002] Open-vocabulary semantic segmentation aims to perform pixel-level semantic annotation on different regions of an input image based on text or natural language descriptions of any category. Unlike traditional semantic segmentation methods that rely on a fixed set of categories, open-vocabulary semantic segmentation can identify open categories beyond the training categories, making it more suitable for long-tail category recognition, unknown category perception, and complex scene understanding tasks in real-world open environments.

[0003] In recent years, visual-language models have mapped image and text features into a unified semantic space through large-scale image-text alignment learning, enabling them to perform zero-shot recognition based on text category cues. These models have demonstrated good semantic generalization capabilities in open-vocabulary classification, detection, and segmentation tasks. However, visual-language models typically rely primarily on similarity matching between image-level and text features, resulting in limited spatial localization capabilities and susceptibility to boundary blurring, region fragmentation, and local misclassification in dense prediction tasks. On the other hand, basic segmentation models can generate high-quality candidate regions based on point, bounding box, mask, or response region cues, exhibiting advantages in target boundary, region integrity, and spatial structure modeling. However, these models typically lack stable open-vocabulary semantic discrimination capabilities and struggle to independently perform pixel-level semantic recognition for arbitrary categories. Therefore, effectively integrating the open semantic understanding capabilities of visual-language models with the spatial region generation capabilities of basic segmentation models is a crucial technical challenge in open-vocabulary semantic segmentation. In practical applications, models also face the challenge of varying target domain distributions. For example, in scenarios such as autonomous driving, robot perception, security monitoring, or remote sensing image analysis, test images may be affected by weather changes, lighting changes, motion blur, sensor noise, or scene migration. These factors can cause the distribution of test data to deviate from the distribution of the model's pre-training data, making the model's semantic response unstable and thus affecting the results of open-vocabulary semantic segmentation.

[0004] Test-time adaptive techniques have been proposed to mitigate the performance degradation of models on unlabeled data in the target domain. Existing test-time adaptive methods can be broadly categorized into two types: one is parameter update-based methods, which dynamically adjust model parameters using unlabeled samples during the testing phase; the other is caching or memory-based methods, which store historical test sample features without updating model parameters and assist current predictions through similarity retrieval. However, these methods still have significant shortcomings in open-vocabulary semantic segmentation tasks. Parameter update-based methods are susceptible to noise from densely predicted pseudo-labels. When there are many local mispredictions in the test image, directly using unreliable predictions for parameter updates can easily contaminate model parameters with erroneous information, leading to confirmation bias and error accumulation. While caching-based methods avoid the risks of parameter updates, existing caching mechanisms are mostly designed for image classification tasks, typically using whole-image features as caching units, making them unsuitable for semantic segmentation tasks where an image contains multiple semantic regions or where different regions have significantly different reliability.

[0005] Therefore, existing technologies still lack a test-time knowledge transfer mechanism suitable for open-vocabulary semantic segmentation that can achieve regional-level knowledge construction, reliability screening, structured cache maintenance, and dynamic prediction fusion without accessing source domain data or updating model parameters, thereby improving the model's segmentation robustness under complex domain offset conditions. Summary of the Invention

[0006] To address the limitations of existing cached adaptive testing methods for open-vocabulary semantic segmentation tasks, and their inherent problems such as susceptibility of reliable knowledge to noise contamination, insufficient coverage of intra-class feature changes, and error accumulation due to repeated use of cached knowledge, this invention provides a robust test-time knowledge transfer method and apparatus for open-vocabulary semantic segmentation. The technical solution is as follows: On the one hand, a robust test-time knowledge transfer method for open-lexicon semantic segmentation is provided, which is implemented by a robust test-time knowledge transfer device for open-lexicon semantic segmentation. The method includes: S1. Obtain the target domain test image and open category text set, generate dense visual features of the test image and category text features of the open category text set through a visual-language model, calculate the similarity between the two to obtain semantic response maps for each category, and output the initial prediction results of the source model. S2. Based on the semantic response map, extract high-response regions as candidate cue regions, convert them into cue information that can be received by the basic segmentation model, and input them into the basic segmentation model to generate a candidate mask set. S3. Calculate the comprehensive reliability metric for the candidate region corresponding to each candidate mask; dynamically determine the first reliability threshold and the second reliability threshold according to the historical reliability statistics queue, and divide the candidate region into reliable region, ambiguous region and noisy region according to the comprehensive reliability metric. S4. Construct a multi-branch cache structure, including: reliable cache, ambiguous cache, and exclusion cache; store the features of the reliable region, the ambiguous region, and the noisy region into the reliable cache, the ambiguous cache, and the exclusion cache, respectively; S5. Compactly integrate the regional features within multiple caches, and update the regional features within each cache through reliability ranking, sampling, and geometric prototype selection. S6. Calculate the similarity between the current test image features and the region features of the multi-branch cache respectively, and generate reliable cache prediction results, ambiguous cache prediction results and exclusion cache prediction results; S7. The reliable cache prediction results and the ambiguous cache prediction results are weighted and fused to obtain the positive cache enhancement prediction results; the positive cache enhancement prediction results are adjusted and fused with the initial prediction results of the source model to obtain the final open vocabulary semantic segmentation results.

[0007] On the other hand, a robust test-time knowledge transfer apparatus for open-lexicon semantic segmentation is provided. This apparatus is applied to a robust test-time knowledge transfer method for open-lexicon semantic segmentation. The apparatus includes: The initial semantic response module is used to acquire the target domain test image and the open category text set, generate the dense visual features of the test image and the category text features of the open category text set through a visual-language model, calculate the similarity between the two to obtain the semantic response map of each category, and output the initial prediction results of the source model. The candidate region generation module is used to extract high-response regions as candidate cue regions based on the semantic response map, convert them into cue information that can be received by the basic segmentation model, and input them into the basic segmentation model to generate a candidate mask set. The reliability assessment module is used to calculate a comprehensive reliability metric for the candidate region corresponding to each candidate mask; dynamically determine a first reliability threshold and a second reliability threshold based on the historical reliability statistics queue; and divide the candidate region into a reliable region, an ambiguous region, and a noisy region according to the comprehensive reliability metric. The multi-branch cache module is used to build a multi-branch cache structure, including: reliable cache, ambiguous cache, and exclusion cache; the features of reliable regions, ambiguous regions, and noisy regions are stored in the reliable cache, ambiguous cache, and exclusion cache, respectively; The cache branch integration module is used to compactly integrate the regional features in multiple caches. It updates the regional features in each cache through reliability sorting, sampling and geometric prototype selection. The prediction result generation module is used to calculate the similarity between the features of the current test image and the regional features of the multi-branch cache, and generate reliable cache prediction results, ambiguous cache prediction results, and exclusion cache prediction results. The prediction result fusion module is used to weight and fuse the reliable cache prediction results and the ambiguous cache prediction results to obtain the positive cache enhanced prediction results; the positive cache enhanced prediction results are then adjusted and fused with the initial prediction results of the source model to obtain the final open vocabulary semantic segmentation results.

[0008] On the other hand, a robust test-time knowledge transfer device for open-vocabulary semantic segmentation is provided, the device comprising: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods described above for robust test-time knowledge transfer in open-vocabulary semantic segmentation.

[0009] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods for robust knowledge transfer during open-vocabulary semantic segmentation.

[0010] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: This invention utilizes a visual-language model to generate open-category semantic responses, and uses these responses to guide a basic segmentation model in generating candidate regions. This ensures that the candidate regions possess both category semantic directionality and spatial structural integrity, thereby alleviating the problem of the separation between semantic discrimination and spatial localization in existing methods. This invention does not directly use the prediction results from the testing phase as pseudo-labels; instead, it first performs a comprehensive reliability assessment on the candidate regions. Only the region knowledge that has passed reliability screening is written into the corresponding cache, thereby reducing the pollution of the historical knowledge base by low-quality pseudo-labels.

[0011] The embodiments of the present invention use region-level knowledge as the basic unit for caching and transfer. Compared with whole-image-level caching or image block-level caching, it can better adapt to the characteristics of multiple categories of regions coexisting in an image and significant differences in prediction quality between different regions.

[0012] This invention divides the region knowledge during testing into reliable knowledge, ambiguous knowledge, and noisy knowledge, and stores them in a reliable cache, an ambiguous cache, and a rejection cache, respectively. This mechanism allows reliable knowledge to be positively enhanced, ambiguous knowledge to be used cautiously, and noisy knowledge to be negatively suppressed, thereby reducing the risk of error propagation. This invention also compresses and filters the region features in the cache through reliability ranking sampling and geometric prototype selection, retaining more representative and diverse region knowledge within limited cache capacity, reducing duplicate information and redundant computation.

[0013] This invention employs an uncertainty-aware fusion mechanism to dynamically adjust fusion weights based on the uncertainty of different prediction sources, and introduces the initial predictions of the source model as stable priors. This results in more stable segmentation results under complex weather conditions, ambiguity, noise, lighting variations, and sensor degradation. This invention does not require access to source domain training data or updating of basic model parameters; it relies solely on forward inference, region knowledge caching, and similarity retrieval to complete knowledge transfer during testing. It is applicable to various visual-language models, basic segmentation models, and open-vocabulary semantic segmentation systems. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a robust testing method for knowledge transfer in semantic segmentation of open lexicons, provided by an embodiment of the present invention. Figure 2 This is an overall flowchart of a knowledge transfer method for robust testing provided by an embodiment of the present invention; Figure 3 This is a block diagram of a robust knowledge transfer device for open-vocabulary semantic segmentation during testing, provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of a knowledge transfer device for robust testing of open vocabulary semantic segmentation provided in an embodiment of the present invention. Detailed Implementation

[0016] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0017] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0018] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0019] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0020] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0021] This invention provides a robust test-time knowledge transfer method for open-vocabulary semantic segmentation. This method can be implemented by a robust test-time knowledge transfer device for open-vocabulary semantic segmentation, which can be a terminal or a server. Figure 1 The flowchart shown is for a robust, test-time knowledge transfer method for open-vocabulary semantic segmentation. The processing flow of this method may include the following steps: S1. Obtain autonomous driving test images and open-category text sets. Generate dense visual features of the test images and category text features of the open-category text sets through a visual-language model. Calculate the similarity between the two to obtain semantic response maps for each category and output the initial prediction results of the source model.

[0022] The open category text set includes: the name of the category to be identified, the category description text, the attribute text, or a text prompt template constructed from the category name.

[0023] In one feasible implementation, the specific implementation process of S1 includes: S11. Input the target domain test image into the visual encoder to obtain dense visual features; input the open category text set into the text encoder to obtain category text features; S12. Calculate the similarity between dense visual features and category text features, and generate a semantic response map for each category. The similarity can be cosine similarity, dot product similarity, normalized correlation, or other measures that can measure the degree of matching between image features and text features.

[0024] In one feasible implementation, based on the implementation process of S1 described above, the initial prediction result of the source model is obtained. This initial prediction result is used to provide semantic priors for open categories and serves as a stabilizing constraint in subsequent prediction fusion.

[0025] S2. Based on the semantic response map, extract high-response regions as candidate cue regions, convert them into cue information that can be received by the basic segmentation model, and input them into the basic segmentation model to generate a candidate mask set.

[0026] The basic segmentation model can be a large visual model with zero-shot generalization capability and support for prompting guidance, including but not limited to: SAM (Segment Anything Model), which can be adapted to different versions: SAM-ViT-B, SAM-ViT-L or SAM-ViT-H; and the lighter version FastSAM or MobileSAM.

[0027] The prompts include point prompts, box prompts, coarse mask prompts, response region prompts, or combinations thereof. Each candidate mask corresponds to a region in the image, which has corresponding spatial boundaries, regional visual features, and candidate semantic categories. This embodiment of the invention transforms the original pixel-level or image block-level open semantic responses into region-level knowledge units.

[0028] In this embodiment of the invention, a visual-language model is used to provide category semantic guidance, and a basic segmentation model is used to provide spatial structure constraints, thereby obtaining candidate regions that have both open category semantic orientation and good spatial integrity.

[0029] S3. Calculate the comprehensive reliability metric for the candidate region corresponding to each candidate mask; dynamically determine the first reliability threshold and the second reliability threshold based on the historical reliability statistics queue, and divide the candidate region into reliable region, ambiguous region and noisy region according to the comprehensive reliability metric.

[0030] Optionally, the specific implementation process of S3 includes S31-S32: S31. Based on the historical reliability statistics queue, dynamically determine the first reliability threshold and the second reliability threshold using quantile statistics. The historical reliability statistics queue is used to record the comprehensive reliability metric of candidate regions generated during the testing process. The first and second reliability thresholds can be determined using quantile statistics, moving averages, adaptive clustering, or other distribution estimation methods.

[0031] S32. Based on the comprehensive reliability metric of the candidate regions, the candidate regions are divided into three categories using the first reliability threshold and the second reliability threshold. Specifically, when the overall reliability metric of a candidate region is lower than the first reliability threshold, the region is determined to be a reliable region. Specifically, when the comprehensive reliability metric of a candidate region is greater than or equal to the first reliability threshold and less than the second reliability threshold, the region is identified as an ambiguous region. Specifically, when the overall reliability metric of a candidate region is greater than or equal to the second reliability threshold, the region is identified as a noise region.

[0032] Among them, the comprehensive reliability metric is used to measure whether the candidate region is suitable as knowledge that can be accumulated during the testing phase.

[0033] In one feasible implementation, the steps in S3 above can achieve hierarchical decoupling of regional knowledge during testing, avoiding the mixed storage of knowledge with different reliability levels, thereby reducing the risk of cache pollution.

[0034] Optionally, a comprehensive reliability metric may be used, including one or more of the following: semantic consistency index, regional purity index, spatial coverage index, local prediction uncertainty index, and regional semantic uncertainty index. The semantic consistency index is used to measure the degree of matching between the regional visual features of the candidate region and the corresponding category text features. The regional visual features are obtained by average pooling, weighted pooling, or attention aggregation of dense visual features within the candidate mask coverage area. Among them, the region purity index is used to measure whether the category responses within a candidate region are concentrated in the same semantic category. If the category responses of multiple pixels within a candidate region are consistent, the semantic purity of the region is high. If the responses within a region are scattered, the region contains multi-class hybridity or boundary errors. Among them, the spatial coverage index is used to measure the degree of spatial overlap between the candidate mask and the high-response region of the corresponding category; if the candidate mask can cover the high-response region of the category, it indicates that the candidate region and the semantic response are consistent. The local prediction uncertainty index is used to measure the degree of uncertainty of the pixel-level prediction distribution within the candidate region. The local prediction uncertainty index can be calculated by means of information entropy of pixel-level class probability distribution, maximum class confidence, or class interval.

[0035] Among them, the regional semantic uncertainty index is used to measure the degree of uncertainty in the overall predicted distribution of candidate regions.

[0036] The above indicators are weighted and combined to obtain a regional comprehensive reliability metric. The lower the comprehensive reliability metric, the more reliable the candidate region is. The higher the comprehensive reliability metric, the more likely the candidate region is to contain semantic confusion, boundary errors, or noise information.

[0037] S4. Construct a multi-branch cache structure, including: reliable cache, ambiguous cache and exclusion cache; store the features of the reliable region, the ambiguous region and the noisy region into the reliable cache, the ambiguous cache and the exclusion cache respectively.

[0038] Optionally, a reliable cache is used to store the regional features, prediction categories, and uncertainty information corresponding to reliable regions for positive semantic enhancement in inference; Among them, the ambiguity cache is used to store the regional features, candidate category distribution and uncertainty information corresponding to the ambiguous region, and to serve as an auxiliary reference in reasoning; Among them, the exclusion cache is used to store the regional features, unreliable category responses, or category information to be suppressed corresponding to the noisy region, and is used for negative constraints in inference.

[0039] In order to prevent the cache from growing indefinitely and accumulating redundant information, this embodiment of the invention performs compact integration of the regional features in the cache.

[0040] S5. Compactly integrate the regional features within multiple caches, and update the regional features within each cache through reliability sorting, sampling, and geometric prototype selection.

[0041] Optionally, the specific implementation process of S5 includes: The candidate region features are sorted from low to high based on the comprehensive reliability metric, and the region features with the highest reliability are selected. By performing interval sampling, hierarchical sampling, or reliability-weighted sampling on the regional features with the highest reliability, a set of candidate features is obtained, reducing spatial redundancy caused by adjacent or overlapping regions. Geometric prototype selection is performed on the candidate feature set. The center vector of the candidate feature set is calculated, and the distance between each candidate feature and the center vector is calculated. The feature that can represent the overall feature distribution is selected as the initial prototype. Taking into account the differences between candidate features and selected prototypes, as well as their supplementary role to the existing cache feature distribution, regional features that are both representative and diverse are gradually selected and written into the cache, and the regional feature representations in each cache are dynamically updated.

[0042] When a certain category or a certain cache branch reaches the preset capacity limit, the retention, replacement or deletion of cache entries is determined based on one or more of the following indicators: regional reliability, regional uncertainty, category balance, feature diversity and time freshness.

[0043] In one feasible implementation, through step S5, the present invention can maintain the representativeness, compactness and diversity of knowledge during testing under limited cache capacity, and avoid the cache being occupied by duplicate features or low-quality features.

[0044] S6. Calculate the similarity between the current test image features and the region features of the multi-branch cache, and generate reliable cache prediction results, ambiguous cache prediction results, and exclusionary cache prediction results.

[0045] Optionally, the specific implementation process of S6 includes S61-S62: S61. Calculate the similarity between the features of the current test image and the features in the reliable cache region, the features in the ambiguous cache region, and the features in the exclusion cache region, respectively; normalize the calculated similarities and output the normalized similarity; S62. Based on the normalized similarity, perform weighted aggregation on the category labels or category distributions in each cache to obtain reliable cache prediction results, ambiguous cache prediction results, and exclusion cache prediction results. Among them, reliable cached prediction results are used to enhance category responses consistent with historical reliable knowledge; ambiguous cached prediction results are used to provide auxiliary semantic references; and exclusionary cached prediction results are used to identify category responses in the current prediction that may be similar to historical noise patterns.

[0046] S7. The reliable cache prediction results and the ambiguous cache prediction results are weighted and fused to obtain the positive cache enhancement prediction results; the positive cache enhancement prediction results are adjusted and fused with the initial prediction results of the source model to obtain the final open vocabulary semantic segmentation results.

[0047] Optionally, the specific implementation process of S7 includes S71-S75: S71. Calculate the uncertainties of the reliable cache prediction result, the ambiguous cache prediction result, the rejected cache prediction result, and the initial prediction result of the source model, respectively. Wherein, for any prediction result, the corresponding Logits vector First, use temperature parameters. The temperature was calibrated, and the normalized class probability distribution was obtained through the Softmax function, which is expressed by the following formula (1): (1) in, Indicates the first The Logits values ​​corresponding to each category; This indicates the preset temperature calibration parameters; Indicates the total number of open categories; Indicates the first day after temperature calibration The predicted probabilities for each category.

[0048] Furthermore, the Shannon information entropy is calculated based on the normalized category probability distribution to characterize the uncertainty of the corresponding prediction result, expressed by the following formula (2): (2) in, This indicates the uncertainty of the prediction result. This is a very small positive number set to prevent the occurrence of the logarithm of zero in numerical calculations. The larger the information entropy, the more uncertain the prediction result; the smaller the information entropy, the more reliable the prediction result.

[0049] S72. Based on the uncertainty of the reliable cache prediction result and the uncertainty of the ambiguous cache prediction result, generate the fusion weight of the two. Among them, the information entropy based on the reliable cache prediction results Information entropy of ambiguous cache prediction results The dynamic fusion weights corresponding to the reliable cache prediction results and the ambiguous cache prediction results are calculated and expressed by the following formulas (3)-(4): (3) (4) in, The dynamic fusion weights represent the prediction results of reliable caching. This represents the dynamic fusion weight of the ambiguous cache prediction results; this weight allocation method gives higher fusion weights to prediction branches with lower uncertainty, thereby improving the stability of the cache prediction results.

[0050] S73. The reliable cache prediction result and the ambiguous cache prediction result are weighted and fused to obtain the positive cache enhancement prediction result; In one feasible implementation, the reliable cache prediction result is adjusted based on the obtained dynamic fusion weights. Ambiguous cache prediction results The weighted summation is performed to output the positive cache enhancement prediction result, which is expressed by the following formula (5): (5) in, This indicates that the output is positively cached to enhance the prediction results.

[0051] S74. The negative suppression of the positive cache enhancement prediction result is performed using the rejection cache prediction result, and the cache enhancement prediction result is output. In one feasible implementation, the rejection cache prediction result is used as a negative constraint to suppress the interference of noisy regions and confused categories on the semantic segmentation result, and the rejection cache prediction result is obtained by the following formula (6): (6) in, This indicates that the prediction result should not be cached. This represents the cache enhancement prediction result after negative suppression.

[0052] S75. Adjust and fuse the cache-enhanced prediction results with the initial prediction results of the source model to output the final open-vocabulary semantic segmentation results.

[0053] In one feasible implementation, the information entropy of the cache-enhanced prediction results is calculated separately. Information entropy of the initial prediction results of the source model Based on the uncertainty of both, a dynamic adjustment weight is generated, which is expressed by the following formulas (7)-(8): (7) (8) in, This represents the dynamic adjustment weights for cache-enhanced prediction results. This represents the dynamically adjusted weights of the initial prediction results from the source model.

[0054] Furthermore, the cache enhancement prediction results are adjusted based on dynamically adjusted weights. Compared with the initial prediction results of the source model The final open-vocabulary semantic segmentation result is obtained by fusion, and is expressed by the following formula (9): (9) in, This represents the final open-vocabulary semantic segmentation result. Through this dynamic adjustment mechanism, when the uncertainty of the cache-enhanced prediction result is high, the initial prediction result of the source model receives higher weight; when the cache-enhanced prediction result is more reliable, its contribution to the final prediction result increases accordingly, thereby reducing the interference of noisy cache on the segmentation result.

[0055] The initial prediction results of the source model serve as stable semantic priors to prevent cached predictions from excessively dominating the final results in cases of cache pollution or extreme domain offsets.

[0056] Among them, step S7 realizes the positive utilization of reliable knowledge, the effective utilization of ambiguous knowledge, the negative suppression of noisy knowledge, and the stable constraint of the source model prior, thereby improving the segmentation robustness in complex test environments.

[0057] In this embodiment of the invention, execution occurs during the testing phase, requiring no access to source domain training data or updating of network parameters of the visual-language model or the basic segmentation model. The problems that this embodiment can solve include: 1. Addressing the disconnect between semantic discrimination and spatial segmentation capabilities in open-vocabulary semantic segmentation, enabling the category semantic response provided by the visual-language model to effectively guide the basic segmentation model in generating semantically relevant candidate regions; 2. Resolving the issue of significant differences in the reliability of regions within test samples in dense prediction tasks, allowing for hierarchical management of region knowledge generated during the testing phase based on reliability; 3. Addressing the problem of existing caching methods easily introducing low-quality pseudo-labels, enabling the caching mechanism to suppress knowledge pollution caused by unreliable regions; 4. Resolving the lack of dynamic adjustment during the utilization of historical knowledge, allowing for adaptive fusion of reliable knowledge, ambiguous knowledge, noisy knowledge, and the initial predictions of the source model based on uncertainty; 5. Addressing the instability of open-vocabulary semantic segmentation results under complex domain offset conditions, improving the robustness and generalization ability of the model in the target domain testing phase without using source domain data or updating the basic model parameters.

[0058] This invention utilizes a visual-language model to generate open-category semantic responses, and uses these responses to guide a basic segmentation model in generating candidate regions. This ensures that the candidate regions possess both category semantic directionality and spatial structural integrity, thereby alleviating the problem of the separation between semantic discrimination and spatial localization in existing methods. This invention does not directly use the prediction results from the testing phase as pseudo-labels; instead, it first performs a comprehensive reliability assessment on the candidate regions. Only the region knowledge that has passed reliability screening is written into the corresponding cache, thereby reducing the pollution of the historical knowledge base by low-quality pseudo-labels.

[0059] The embodiments of the present invention use region-level knowledge as the basic unit for caching and transfer. Compared with whole-image-level caching or image block-level caching, it can better adapt to the characteristics of multiple categories of regions coexisting in an image and significant differences in prediction quality between different regions.

[0060] This invention divides the region knowledge during testing into reliable knowledge, ambiguous knowledge, and noisy knowledge, and stores them in a reliable cache, an ambiguous cache, and a rejection cache, respectively. This mechanism allows reliable knowledge to be positively enhanced, ambiguous knowledge to be used cautiously, and noisy knowledge to be negatively suppressed, thereby reducing the risk of error propagation. This invention also compresses and filters the region features in the cache through reliability ranking sampling and geometric prototype selection, retaining more representative and diverse region knowledge within limited cache capacity, reducing duplicate information and redundant computation.

[0061] This invention employs an uncertainty-aware fusion mechanism to dynamically adjust fusion weights based on the uncertainty of different prediction sources, and introduces the initial predictions of the source model as stable priors. This results in more stable segmentation results under complex weather conditions, ambiguity, noise, lighting variations, and sensor degradation. This invention does not require access to source domain training data or updating of basic model parameters; it relies solely on forward inference, region knowledge caching, and similarity retrieval to complete knowledge transfer during testing. It is applicable to various visual-language models, basic segmentation models, and open-vocabulary semantic segmentation systems.

[0062] Figure 3 This is a block diagram of a robust test-time knowledge transfer device for open-vocabulary semantic segmentation provided by an embodiment of the present invention. This device is used for a robust test-time knowledge transfer method for open-vocabulary semantic segmentation. (Refer to...) Figure 3 The device includes an initial semantic response module 310, a candidate region generation module 320, a reliability evaluation module 330, a multi-branch caching module 340, a cache branch integration module 350, a prediction result generation module 360, and a prediction result fusion module 370. Among them: The initial semantic response module 310 is used to acquire autonomous driving test images and open category text sets, generate dense visual features of the test images and category text features of the open category text sets through a visual-language model, calculate the similarity between the two to obtain semantic response maps for each category, and output the initial prediction results of the source model. The candidate region generation module 320 is used to extract high-response regions as candidate cue regions based on the semantic response map, convert them into cue information that can be received by the basic segmentation model, and input them into the basic segmentation model to generate a candidate mask set. The reliability assessment module 330 is used to calculate a comprehensive reliability metric for the candidate region corresponding to each candidate mask; dynamically determine a first reliability threshold and a second reliability threshold based on the historical reliability statistics queue; and divide the candidate region into a reliable region, an ambiguous region, and a noisy region according to the comprehensive reliability metric. The multi-branch cache module 340 is used to construct a multi-branch cache structure, including: a reliable cache, an ambiguous cache, and a rejection cache; the features of the reliable region, the features of the ambiguous region, and the features of the noisy region are stored in the reliable cache, the ambiguous cache, and the rejection cache, respectively; The cache branch integration module 350 is used to compactly integrate the regional features in multiple caches and update the regional features in each cache through reliability sorting, sampling and geometric prototype selection processing. The prediction result generation module 360 ​​is used to calculate the similarity between the features of the current test image and the regional features of the multi-branch cache, and generate reliable cache prediction results, ambiguous cache prediction results and exclusion cache prediction results. The prediction result fusion module 370 is used to perform weighted fusion of reliable cache prediction results and ambiguous cache prediction results to obtain positive cache enhancement prediction results; and to adjust and fuse the positive cache enhancement prediction results with the initial prediction results of the source model to obtain the final open vocabulary semantic segmentation results.

[0063] Optionally, the comprehensive reliability metric includes one or more of the following: semantic consistency index, regional purity index, spatial coverage index, local prediction uncertainty index, and regional semantic uncertainty index. The semantic consistency index is used to measure the degree of matching between the regional visual features of the candidate region and the corresponding category text features. The regional visual features are obtained by average pooling, weighted pooling, or attention aggregation of dense visual features within the candidate mask coverage area. Among them, the region purity index is used to measure whether the category responses within a candidate region are concentrated in the same semantic category. If the category responses of multiple pixels within a candidate region are consistent, the semantic purity of the region is high. If the responses within a region are scattered, the region contains multi-class hybridity or boundary errors. Among them, the spatial coverage index is used to measure the degree of spatial overlap between the candidate mask and the high-response region of the corresponding category; if the candidate mask can cover the high-response region of the category, it indicates that the candidate region and the semantic response are consistent. Among them, the local prediction uncertainty index is used to measure the degree of uncertainty of the pixel-level prediction distribution within the candidate region; Among them, the regional semantic uncertainty index is used to measure the degree of uncertainty in the overall predicted distribution of candidate regions.

[0064] Optionally, the reliability assessment module 330 is used for: Based on the historical reliability statistics queue, the first reliability threshold and the second reliability threshold are dynamically determined using quantile statistics. Based on the comprehensive reliability metric of the candidate regions, the candidate regions are divided into three categories using a first reliability threshold and a second reliability threshold. Specifically, when the overall reliability metric of a candidate region is lower than the first reliability threshold, the region is determined to be a reliable region. Specifically, when the comprehensive reliability metric of a candidate region is greater than or equal to the first reliability threshold and less than the second reliability threshold, the region is identified as an ambiguous region. Specifically, when the overall reliability metric of a candidate region is greater than or equal to the second reliability threshold, the region is identified as a noise region.

[0065] Optionally, the reliable cache is used to store the regional features, prediction categories, and uncertainty information corresponding to the reliable region, for positive semantic enhancement in inference; Among them, the ambiguity cache is used to store the regional features, candidate category distribution and uncertainty information corresponding to the ambiguous region, and to serve as an auxiliary reference in reasoning; Among them, the exclusion cache is used to store the regional features, unreliable category responses, or category information to be suppressed corresponding to the noisy region, and is used for negative constraints in inference.

[0066] Optionally, the cache branch integration module 350 is used for: The candidate region features are sorted from low to high based on the comprehensive reliability metric, and the region features with the highest reliability are selected. The region features with the highest reliability are sampled at intervals, hierarchically, or with reliability-weighted sampling to obtain a set of candidate features. Geometric prototype selection is performed on the candidate feature set. The center vector of the candidate feature set is calculated, and the distance between each candidate feature and the center vector is calculated. The feature that can represent the overall feature distribution is selected as the initial prototype. Taking into account the differences between candidate features and selected prototypes, as well as their supplementary role to the existing cache feature distribution, regional features that are both representative and diverse are gradually selected and written into the cache, and the regional feature representations in each cache are dynamically updated.

[0067] Optionally, the prediction result generation module 360 ​​is used to: Calculate the similarity between the features of the current test image and the features in the reliable cache region, the features in the ambiguous cache region, and the features in the exclusion cache region, respectively; normalize the calculated similarities and output the normalized similarity; Based on the normalized similarity, the category labels or category distributions in each cache are weighted and aggregated to obtain reliable cache prediction results, ambiguous cache prediction results, and exclusion cache prediction results. Among them, reliable cached prediction results are used to enhance category responses consistent with historical reliable knowledge; ambiguous cached prediction results are used to provide auxiliary semantic references; and exclusionary cached prediction results are used to identify category responses in the current prediction that may be similar to historical noise patterns.

[0068] Optionally, the prediction result fusion module 370 is used for: Calculate the uncertainties of the reliable cache prediction results, the ambiguous cache prediction results, the rejected cache prediction results, and the initial prediction results of the source model, respectively; Based on the uncertainty of reliable cache prediction results and the uncertainty of ambiguous cache prediction results, a fusion weight is generated for the two. The reliable cache prediction results and the ambiguous cache prediction results are weighted and fused to obtain the positive cache enhancement prediction results; The negative suppression of the positive cache enhancement prediction result is achieved by using the rejection cache prediction result, and the cache enhancement prediction result is output. The cache-enhanced prediction results are adjusted and fused with the initial prediction results of the source model to output the final open-vocabulary semantic segmentation results.

[0069] This invention utilizes a visual-language model to generate open-category semantic responses, and uses these responses to guide a basic segmentation model in generating candidate regions. This ensures that the candidate regions possess both category semantic directionality and spatial structural integrity, thereby alleviating the problem of the separation between semantic discrimination and spatial localization in existing methods. This invention does not directly use the prediction results from the testing phase as pseudo-labels; instead, it first performs a comprehensive reliability assessment on the candidate regions. Only the region knowledge that has passed reliability screening is written into the corresponding cache, thereby reducing the pollution of the historical knowledge base by low-quality pseudo-labels.

[0070] The embodiments of the present invention use region-level knowledge as the basic unit for caching and transfer. Compared with whole-image-level caching or image block-level caching, it can better adapt to the characteristics of multiple categories of regions coexisting in an image and significant differences in prediction quality between different regions.

[0071] This invention divides the region knowledge during testing into reliable knowledge, ambiguous knowledge, and noisy knowledge, and stores them in a reliable cache, an ambiguous cache, and a rejection cache, respectively. This mechanism allows reliable knowledge to be positively enhanced, ambiguous knowledge to be used cautiously, and noisy knowledge to be negatively suppressed, thereby reducing the risk of error propagation. This invention also compresses and filters the region features in the cache through reliability ranking sampling and geometric prototype selection, retaining more representative and diverse region knowledge within limited cache capacity, reducing duplicate information and redundant computation.

[0072] This invention employs an uncertainty-aware fusion mechanism to dynamically adjust fusion weights based on the uncertainty of different prediction sources, and introduces the initial predictions of the source model as stable priors. This results in more stable segmentation results under complex weather conditions, ambiguity, noise, lighting variations, and sensor degradation. This invention does not require access to source domain training data or updating of basic model parameters; it relies solely on forward inference, region knowledge caching, and similarity retrieval to complete knowledge transfer during testing. It is applicable to various visual-language models, basic segmentation models, and open-vocabulary semantic segmentation systems.

[0073] Figure 4 This is a schematic diagram of the structure of a knowledge transfer device for robust testing of open-vocabulary semantic segmentation provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the knowledge transfer device for robust testing of open-lexical semantic segmentation may include the above-mentioned Figure 3 The illustrated robust test-time knowledge transfer apparatus for open-lexical semantic segmentation. Optionally, the robust test-time knowledge transfer apparatus 410 for open-lexical semantic segmentation may include a first processor 2001.

[0074] Optionally, the knowledge transfer device 410 for robust testing of open-lexical semantic segmentation may also include a memory 2002 and a transceiver 2003.

[0075] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0076] The following is combined Figure 4 The various components of the knowledge transfer device 410 used for robust testing of open-vocabulary semantic segmentation are described in detail below: The first processor 2001 is the control center of the knowledge transfer device 410 used for robust testing of open-vocabulary semantic segmentation. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0077] Optionally, the first processor 2001 can perform various functions of the knowledge transfer device 410 for robust testing of open vocabulary semantic segmentation by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0078] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0079] In a specific implementation, as one example, the knowledge transfer device 410 for robust testing of open-lexical semantic segmentation may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0080] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0081] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the knowledge transfer device 410 for robust testing of open-lexical semantic segmentation. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0082] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0083] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0084] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the knowledge transfer device 410 for robust testing of open vocabulary semantic segmentation. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0085] It should be noted that, Figure 4 The structure of the robust test-time knowledge transfer device 410 for open vocabulary semantic segmentation shown in the figure does not constitute a limitation on the router. Actual robust test-time knowledge transfer devices for open vocabulary semantic segmentation may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0086] Furthermore, the technical effect of the robust test-time knowledge transfer device 410 for open vocabulary semantic segmentation can be referred to the technical effect of the robust test-time knowledge transfer method for open vocabulary semantic segmentation described in the above method embodiments, and will not be repeated here.

[0087] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0088] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0089] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0090] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0091] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0092] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0093] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0094] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0095] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0096] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0097] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0098] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A robust test-time knowledge transfer method for open-lexical semantic segmentation, characterized in that, The method includes: S1. Obtain autonomous driving test images and open category text sets, generate dense visual features of the test images and category text features of the open category text sets through a visual-language model, calculate the similarity between the two to obtain semantic response maps for each category, and output the initial prediction results of the source model. S2. Based on the semantic response map, extract high-response regions as candidate cue regions, convert them into cue information that can be received by the basic segmentation model, and input them into the basic segmentation model to generate a candidate mask set. S3. Calculate the comprehensive reliability metric for the candidate region corresponding to each candidate mask; dynamically determine the first reliability threshold and the second reliability threshold according to the historical reliability statistics queue, and divide the candidate region into reliable region, ambiguous region and noisy region according to the comprehensive reliability metric. S4. Construct a multi-branch cache structure, including: reliable cache, ambiguous cache, and exclusion cache; store the features of the reliable region, the ambiguous region, and the noisy region into the reliable cache, the ambiguous cache, and the exclusion cache, respectively; S5. Compactly integrate the regional features within multiple caches, and update the regional features within each cache through reliability ranking, sampling, and geometric prototype selection. S6. Calculate the similarity between the current test image features and the region features of the multi-branch cache respectively, and generate reliable cache prediction results, ambiguous cache prediction results and exclusion cache prediction results; S7. The reliable cache prediction results and the ambiguous cache prediction results are weighted and fused to obtain the positive cache enhancement prediction results; the positive cache enhancement prediction results are adjusted and fused with the initial prediction results of the source model to obtain the final open vocabulary semantic segmentation results.

2. The robust knowledge transfer method for open-lexical semantic segmentation during testing according to claim 1, characterized in that, The comprehensive reliability metric includes one or more of the following: semantic consistency index, regional purity index, spatial coverage index, local prediction uncertainty index, and regional semantic uncertainty index. The semantic consistency index is used to measure the degree of matching between the regional visual features of the candidate region and the corresponding category text features. The regional visual features are obtained by average pooling, weighted pooling, or attention aggregation of dense visual features within the candidate mask coverage area. Among them, the region purity index is used to measure whether the category responses within a candidate region are concentrated in the same semantic category. If the category responses of multiple pixels within a candidate region are consistent, the semantic purity of the region is high. If the responses within a region are scattered, the region contains multi-class hybridity or boundary errors. Among them, the spatial coverage index is used to measure the degree of spatial overlap between the candidate mask and the high-response region of the corresponding category; if the candidate mask can cover the high-response region of the category, it indicates that the candidate region and the semantic response are consistent. Among them, the local prediction uncertainty index is used to measure the degree of uncertainty of the pixel-level prediction distribution within the candidate region; Among them, the regional semantic uncertainty index is used to measure the degree of uncertainty in the overall predicted distribution of candidate regions.

3. The robust knowledge transfer method for open-lexical semantic segmentation during testing according to claim 1, characterized in that, S3 dynamically determines the first and second reliability thresholds based on the historical reliability statistics queue, and divides the candidate region into reliable, ambiguous, and noisy regions according to the comprehensive reliability metric, including: S31. Based on the historical reliability statistics queue, dynamically determine the first reliability threshold and the second reliability threshold using quantile statistics. S32. Based on the comprehensive reliability metric of the candidate regions, the candidate regions are divided into three categories using the first reliability threshold and the second reliability threshold. Specifically, when the overall reliability metric of a candidate region is lower than the first reliability threshold, the region is determined to be a reliable region. Specifically, when the comprehensive reliability metric of a candidate region is greater than or equal to the first reliability threshold and less than the second reliability threshold, the region is identified as an ambiguous region. Specifically, when the overall reliability metric of a candidate region is greater than or equal to the second reliability threshold, the region is identified as a noise region.

4. The robust knowledge transfer method for open-lexical semantic segmentation during testing according to claim 1, characterized in that, The reliable cache is used to store the regional features, prediction categories, and uncertainty information corresponding to the reliable region, and is used for positive semantic enhancement in inference; Among them, the ambiguity cache is used to store the regional features, candidate category distribution and uncertainty information corresponding to the ambiguous region, and to serve as an auxiliary reference in reasoning; Among them, the exclusion cache is used to store the regional features, unreliable category responses, or category information to be suppressed corresponding to the noisy region, and is used for negative constraints in inference.

5. The robust knowledge transfer method for open-lexical semantic segmentation during testing according to claim 1, characterized in that, The S5 process compactly integrates the regional features within the cache, updating the regional features of each cache through reliability ranking, sampling, and geometric prototype selection, including: The candidate region features are sorted from low to high based on the comprehensive reliability metric, and the region features with the highest reliability are selected. The region features with the highest reliability are sampled at intervals, hierarchically, or with reliability-weighted sampling to obtain a set of candidate features. Geometric prototype selection is performed on the candidate feature set. The center vector of the candidate feature set is calculated, and the distance between each candidate feature and the center vector is calculated. The feature that can represent the overall feature distribution is selected as the initial prototype. Taking into account the differences between candidate features and selected prototypes, as well as their supplementary role to the existing cache feature distribution, regional features that are both representative and diverse are gradually selected and written into the cache, and the regional feature representations in each cache are dynamically updated.

6. The robust knowledge transfer method for open-lexical semantic segmentation during testing according to claim 1, characterized in that, S6 calculates the similarity between the current test image features and the region features of the multi-branch cache, respectively, and generates reliable cache prediction results, ambiguous cache prediction results, and exclusionary cache prediction results, including: S61. Calculate the similarity between the features of the current test image and the features in the reliable cache region, the features in the ambiguous cache region, and the features in the exclusion cache region, respectively; normalize the calculated similarities and output the normalized similarity; S62. Based on the normalized similarity, perform weighted aggregation on the category labels or category distributions in each cache to obtain reliable cache prediction results, ambiguous cache prediction results, and exclusion cache prediction results. Among them, reliable cached prediction results are used to enhance category responses consistent with historical reliable knowledge; ambiguous cached prediction results are used to provide auxiliary semantic references; and exclusionary cached prediction results are used to identify category responses in the current prediction that may be similar to historical noise patterns.

7. The robust test-time knowledge transfer method for open-lexical semantic segmentation according to claim 1, characterized in that, S7 performs a weighted fusion of the reliable cache prediction result and the ambiguous cache prediction result to obtain the positive cache enhancement prediction result; the positive cache enhancement prediction result is then adjusted and fused with the initial prediction result of the source model to obtain the final open vocabulary semantic segmentation result, including: S71. Calculate the uncertainties of the reliable cache prediction result, the ambiguous cache prediction result, the rejected cache prediction result, and the initial prediction result of the source model, respectively. S72. Based on the uncertainty of the reliable cache prediction result and the uncertainty of the ambiguous cache prediction result, generate the fusion weight of the two. S73. The reliable cache prediction result and the ambiguous cache prediction result are weighted and fused to obtain the positive cache enhancement prediction result; S74. The negative suppression of the positive cache enhancement prediction result is performed using the rejection cache prediction result, and the cache enhancement prediction result is output. S75. Adjust and fuse the cache-enhanced prediction results with the initial prediction results of the source model to output the final open-vocabulary semantic segmentation results.

8. A robust test-time knowledge transfer apparatus for open-vocabulary semantic segmentation, wherein the robust test-time knowledge transfer apparatus for open-vocabulary semantic segmentation is used to implement the robust test-time knowledge transfer method for open-vocabulary semantic segmentation as described in any one of claims 1-7, characterized in that, The device includes: The initial semantic response module is used to acquire the target domain test image and the open category text set, generate the dense visual features of the test image and the category text features of the open category text set through a visual-language model, calculate the similarity between the two to obtain the semantic response map of each category, and output the initial prediction results of the source model. The candidate region generation module is used to extract high-response regions as candidate cue regions based on the semantic response map, convert them into cue information that can be received by the basic segmentation model, and input them into the basic segmentation model to generate a candidate mask set. The reliability assessment module is used to calculate a comprehensive reliability metric for the candidate region corresponding to each candidate mask; dynamically determine a first reliability threshold and a second reliability threshold based on the historical reliability statistics queue; and divide the candidate region into a reliable region, an ambiguous region, and a noisy region according to the comprehensive reliability metric. The multi-branch cache module is used to build a multi-branch cache structure, including: reliable cache, ambiguous cache and exclusion cache; the features of reliable region, ambiguous region and noisy region are stored in reliable cache, ambiguous cache and exclusion cache respectively; The cache branch integration module is used to compactly integrate the regional features in multiple caches. It updates the regional features in each cache through reliability sorting, sampling and geometric prototype selection. The prediction result generation module is used to calculate the similarity between the features of the current test image and the regional features of the multi-branch cache, and generate reliable cache prediction results, ambiguous cache prediction results, and exclusion cache prediction results. The prediction result fusion module is used to weight and fuse the reliable cache prediction results and the ambiguous cache prediction results to obtain the positive cache enhanced prediction results; the positive cache enhanced prediction results are then adjusted and fused with the initial prediction results of the source model to obtain the final open vocabulary semantic segmentation results.

9. A robust test-time knowledge transfer device for open-lexical semantic segmentation, characterized in that, The robust test-time knowledge transfer device for open-vocabulary semantic segmentation includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images