Open scene multimodal image-text zero-shot semantic segmentation method and system

Abstract

The application discloses an open scene multi-modal image-text zero sample semantic segmentation method and system, belongs to the technical field of image semantic segmentation, is suitable for processing thermal and infrared images, and improves the zero sample generalization performance of RGB-T semantic segmentation in multiple scenes. Including two stages: an RGB-T open vocabulary target detection stage and a semantic segmentation stage. The RGB-T open vocabulary target detection stage takes pixel-level paired RGB images and thermal images and text as multi-modal input, uses text information and visual cues as conditions, generates initial detection proposals for the target region of the fused image, corrects ambiguous classes for the initial detection proposals in a semantic consistent manner, and obtains corrected predicted classes. The semantic segmentation stage takes the corrected detection classes as a segmentation base model prompt, performs semantic segmentation on the fused image, and generates a textual description of the target.

Description

Technical Field

[0001] The present invention belongs to the technical field of image semantic segmentation, and specifically relates to an open scene multimodal image-text zero-sample semantic segmentation method and system. Background Art

[0002] Semantic segmentation plays a key role in achieving scene understanding in unmanned systems, especially in areas such as autonomous driving, robotics, and remote sensing. Although previous studies have achieved significant segmentation results on standard datasets based on RGB images, these methods often encounter difficulties in accurately segmenting the target object from the background in low-visibility environments caused by inclement weather or low-light conditions. To overcome these limitations, researchers have begun to use thermal / infrared imagery to improve the performance of visual perception tasks. Despite some progress, traditional RGB-T semantic segmentation methods are mostly limited to training on predefined categories, which limits their potential to generalize to more scenarios.

[0003] In recent years, the rapid development of visual language models (VLMs) and open vocabulary learning has paved the way for expanding the capabilities of semantic segmentation tasks in open environments. Compared to traditional semantic segmentation methods, this approach, which combines VLMs and open vocabulary learning, can support text input of any category and achieve instance-level semantic segmentation, hence the name open vocabulary semantic segmentation.

[0004] Most existing open vocabulary semantic segmentation methods adopt a two-stage approach, which mainly includes two ideas.

[0005] The first category of methods first uses a visual foundation model to generate category-independent mask proposals, and then applies a pre-trained image-text matching model to classify these mask proposals. A major challenge with this approach is that its performance is highly dependent on the accurate generation of the initial mask and the subsequent classification step. In addition, since image-text matching models are typically trained on large-scale RGB image datasets, they may lead to classification errors when there are large differences between the image to be detected (target domain) and the training dataset (source domain).

[0006] To reduce potential classification errors, the second type of method first uses an open vocabulary object detector to generate detection boxes for the category to be detected, and then uses a visual base model to segment the object. However, this method may still encounter false detections or missed detections during the object detection stage because the open vocabulary object detector is also trained based on a fixed text description library. When the input text description has semantically ambiguous overlap with the categories in the training text library, an incorrect confidence score may be generated, leading to false detections or missed detections.

[0007] Furthermore, existing open-vocabulary semantic segmentation models are mostly limited to using a single modality, RGB. Due to the heterogeneity between different modal data, these models encounter difficulties in class understanding when processing thermal or infrared images. Summary of the Invention

[0008] In view of this, the present invention provides an open-scene multimodal image-text zero-shot semantic segmentation method and system, which is suitable for processing thermal and infrared images and improves the zero-shot generalization performance of RGB-T semantic segmentation in multiple scenarios.

[0009] To achieve the above objectives, the technical solution of the present invention is: an open scene multimodal image-text zero-shot semantic segmentation method, which includes two stages: RGB-T open vocabulary object detection stage and semantic segmentation stage.

[0010] The RGB-T open vocabulary object detection stage takes pixel-level paired RGB images, thermal images, and text as multimodal inputs. It uses text information and visual cues as conditions to generate initial detection proposals for the target area in the fused image. It then performs semantic consistency correction on the initial detection proposals to correct ambiguous categories and obtain the corrected predicted categories.

[0011] In the semantic segmentation stage, the corrected detection categories are used as hints for the segmentation base model to perform semantic segmentation on the fused image and generate text descriptions of the targets.

[0012] Furthermore, in the RGB-T open vocabulary object detection stage, multimodal input includes RGB image I RGB , Thermal Image I T and user-defined text T s , where the RGB image and the thermal image are paired images collected from the same scene and the same perspective, and the two are paired at the pixel level.

[0013] Furthermore, we use text information and visual cues as conditions to generate initial detection proposals for the target area of ​​the fused image. The specific steps are as follows:

[0014] Step S1: Use the attention-based fusion module to obtain the dynamic fusion weights of the RGB image and the thermal image, perform adaptive fusion on them, and obtain the fused image represented as I f .

[0015] Step S2: User-defined text T s Perform category extraction and obtain predefined semantic categories T = {t1, t2, ..., t K}, where K represents the number of semantic categories. Secondly, the fused image and the predefined semantic categories are input into the open vocabulary object detector Grounding DINO to obtain the detection proposal v with text as the hint. gd =Det gd (T,I f ), where Det gd (·,·) denotes the detection process with textual hints.

[0016] Step S3: Select J semantic target areas in the scene image and crop them to obtain J visual cue image detection frames b j =(x j ,y j ,w j ,h j ), where j = 1, 2, ..., J; x j 、y j Respectively represent the horizontal and vertical coordinates of the upper left corner of the detection box, w j 、h j Represent the width and height of the detection box respectively; the visual embedding V is obtained by the visual encoder: V=En(b1,b2,...,b J ); where En represents the visual encoder.

[0017] Step S4: The fused image passes through the image encoder and is input into the decoder together with the visual embedding V, which outputs the detection proposal v based on the visual prompt vp =De(V,f); where De represents the decoder and f represents the feature map output by the image encoder.

[0018] Step S5: The target detection proposal predicted by the entire multi-hint detection stage is expressed as: v N =v gd ∪v vp ; where N represents the number of initial detection proposals.

[0019] Furthermore, semantic consistency correction is performed on the initial detection proposal to correct the ambiguous categories and obtain the corrected predicted categories, specifically:

[0020] Initial detection proposals v1,v2,...,v N The corresponding initial category ids are

[0021] The pre-trained image-text matching model CLIP is used to detect image proposals and predefined semantic categories T = {t1, t2, ..., t K} perform independent encoding to obtain the corresponding visual embedding and text embedding Recorded as and K is the number of text categories; the similarity score between visual embedding and text embedding is calculated as follows:

[0022]

[0023] in represents the visual embedding of the n-th detection proposal, represents the text embedding of the k-th category, F nk The prediction confidence corresponding to the n-th detection proposal belongs to the k-th category, and <·,·> represents the dot product operation.

[0024] For each detection proposal, select the class with the highest prediction score As the predicted class label Where .index() is the index calling function, pointing to the confidence score The corresponding category k; if the predicted category With the initial detection category Matching, i.e. judging Is it true? If so, it means that the semantic understanding is consistent and no further correction is required.

[0025] If the predicted category is different from the initial detection category, additional judgment needs to be performed based on the following conditions:

[0026] The confidence score corresponding to the initial detection category is retrieved based on the similarity score:

[0027]

[0028] Then, the following two conditions are used for judgment:

[0029]

[0030] Where th1 and th2 are two constant thresholds. If both conditions are met, it is determined that the detection proposal should be updated to the predicted category.

[0031] Furthermore, in the semantic segmentation stage, by integrating the segmentation basic model TAP, the targets in the detection proposal are semantically segmented, thereby achieving zero-shot semantic segmentation of images in open scenes.

[0032] The present invention also provides another open-scene multimodal image-text zero-shot semantic segmentation system, including an RGB-T open vocabulary object detection module and a semantic segmentation module.

[0033] The RGB-T open vocabulary object detection module takes pixel-level paired RGB images, thermal images, and text as multimodal input. It uses text information and visual cues as conditions to generate initial detection proposals for the target area in the fused image. It then performs semantic consistency correction on the initial detection proposals to correct ambiguous categories, and obtains the corrected predicted categories, which are then fed into the semantic segmentation module.

[0034] The semantic segmentation module uses the corrected detection categories as hints for the segmentation base model, performs semantic segmentation on the fused image and generates a text description of the target.

[0035] Furthermore, the multimodal input of the RGB-T open vocabulary object detection module includes the RGB image I RGB , Thermal Image I T and user-defined text T s , where the RGB image and the thermal image are paired images collected from the same scene and the same perspective, and the two are paired at the pixel level.

[0036] Furthermore, the RGB-T open vocabulary object detection module includes: an image fusion unit, a category extraction unit, an open vocabulary object detector Grounding DINO, a visual cue input unit, a visual encoder, an image encoder, a decoder, and a merging unit.

[0037] The image fusion unit is used to obtain the dynamic fusion weights of the RGB image and the thermal image using the attention-based fusion module, and perform adaptive fusion on them to obtain the fused image represented as I f .

[0038] Category extraction unit, used for user-defined text T s Perform category extraction and obtain predefined semantic categories T = {t1, t2, ..., t K}, where K represents the number of semantic categories.

[0039] Open vocabulary object detector Grounding DINO, which takes a fused image and predefined semantic categories as input and obtains detection proposals v with text as hints gd =Det gd (T,I f ), where Det gd (·,·) denotes the detection process with textual hints.

[0040] The visual cue input unit is used to receive visual cue input, select J semantic target areas in the scene image, and crop them to obtain J visual cue image detection frames b j =(x j ,y j ,wj ,h j ), where j = 1, 2, ..., J; x j 、y j Respectively represent the horizontal and vertical coordinates of the upper left corner of the detection box, w j 、h j Represent the width and height of the detection box respectively.

[0041] Visual encoder, through the visual encoder to get the visual embedding V: V = En(b1,b2,...,b J ); where En represents the visual encoder.

[0042] The image encoder takes the fused image as input and outputs the encoded result of the fused image to the decoder.

[0043] Decoder, the fused image passes through the image encoder and is input into the decoder together with the visual embedding V. The decoder outputs the detection proposal v based on the visual prompt vp =De(V,f); where De represents the decoder and f represents the feature map output by the image encoder.

[0044] The merging unit is used to merge the detection proposals based on text and vision to obtain the target detection proposal predicted by the entire multi-prompt detection stage, which is expressed as: v N =v gd ∪v vp ; where N represents the number of initial detection proposals.

[0045] Furthermore, the RGB-T open vocabulary object detection module also includes an initial detection category mapping unit, a pre-trained image-text matching model CLIP, a similarity score calculation unit, and a semantic discrimination unit:

[0046] Initial detection category mapping unit, used to determine the initial detection proposal v1,v2,...,v N The corresponding initial category ids are

[0047] The pre-trained image-text matching model CLIP detects image proposals and predefined semantic categories T = {t1, t2, ..., t K} perform independent encoding to obtain the corresponding visual embedding and text embedding Recorded as and K is the number of text categories.

[0048] The similarity score calculation unit calculates the similarity score between the visual embedding and the text embedding as follows:

[0049]

[0050] in represents the visual embedding of the n-th detection proposal, represents the text embedding of the k-th category, F nk The prediction confidence corresponding to the n-th detection proposal belongs to the k-th category, and <·,·> represents the dot product operation.

[0051] Semantic discrimination unit, specifically:

[0052] For each detection proposal, select the class with the highest prediction score As the predicted class label Where .index() is the index calling function, pointing to the confidence score The corresponding category k; if the predicted category With the initial detection category Matching, i.e. judging Is it true? If so, it means that the semantic understanding is consistent and no further correction is required.

[0053] If the predicted category is different from the initial detection category, additional judgment needs to be performed based on the following conditions:

[0054] The confidence score corresponding to the initial detection category is retrieved based on the similarity score:

[0055]

[0056] Then, the following two conditions are used for judgment:

[0057]

[0058] Where th1 and th2 are two constant thresholds. If both conditions are met, it is determined that the detection proposal should be updated to the predicted category.

[0059] Furthermore, the semantic segmentation module performs semantic segmentation on the targets in the detection proposal by integrating the segmentation basic model TAP, thereby achieving zero-shot semantic segmentation of images in open scenes.

[0060] Beneficial effects:

[0061] 1; The open scene multimodal image-text zero-sample semantic segmentation method and system provided by the present invention improve the generalization ability and segmentation performance of the RGB-T semantic segmentation model in an open environment. The present invention introduces open vocabulary learning, so that the semantic segmentation model is no longer limited by tedious model training and predefined semantic categories, and can process any input category in any scenario, with zero-sample generalization capability. Compared with the existing open vocabulary semantic segmentation model, the present invention can better adapt to complex and changing open environments by utilizing multimodal information, thereby providing more stable and reliable segmentation results. In addition, the multi-cue detection proposed in the present invention can not only enhance the model's understanding of known categories, but also improve its adaptability to new categories or changing scenarios.

[0062] 2: By combining visual cues and textual cues, the present invention can improve the model's adaptability to new categories or changing scenarios, thereby performing more accurate detection.

[0063] 3: The semantic consistency correction proposed in this invention can effectively deal with the semantic ambiguity problem between categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a diagram of the open scene multimodal image-text zero-shot semantic segmentation method and system framework;

[0065] Figure 2 This is the principle diagram of the semantic consistency correction module. DETAILED DESCRIPTION

[0066] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0067] This paper provides an open scene multimodal image-text zero-shot semantic segmentation method and system to improve the zero-shot generalization performance of RGB-T semantic segmentation in multiple scenes. The specific method framework is as follows Figure 1 shown.

[0068] The open scene multimodal image-text zero-shot semantic segmentation method of the present invention mainly includes two stages: the RGB-T open vocabulary target detection stage and the semantic segmentation stage. The first stage uses pixel-level paired RGB images and thermal images and text as multimodal inputs, uses text information and visual cues as conditions, generates an initial detection box for the target area of ​​the fused image, and corrects ambiguous categories through the semantic consistency correction module to obtain the final predicted category. The second stage uses the detection box proposal and the corrected detection category as prompts for the segmentation base model, performs semantic segmentation on the fused image and generates a text description of the target. By executing these two stages sequentially, the method obtains a semantic understanding of the scene, which can be further applied to downstream perception tasks.

[0069] RGB-T Open Vocabulary Object Detection:

[0070] This stage mainly consists of three parts: multimodal input, multi-hint detection and semantic consistency correction. The multimodal input is represented as follows: RGB image I RGB , Thermal Image I T and user-defined text T s , where the RGB image and the thermal image are paired images collected from the same scene and the same perspective, and the two are paired at the pixel level.

[0071] In multi-cue detection, we first consider the richness and information content of different modalities in the input image, and use the attention-based fusion module to obtain the dynamic fusion weights of the two modalities, perform adaptive fusion on them, and obtain the fused image represented as I f Specifically, the fusion module adopts a global-local attention mechanism to focus on global context and local details according to the input image, adaptively updating the fusion weights to maintain effective fusion even when the quality of one of the modalities is impaired or suboptimal.

[0072] The attention-based fusion module used in the embodiment of the present invention adopts an existing module. Specifically, the attention-based fusion module involved in Tang L, Zhang H, Xu H, et al. Rethinking the necessity of image fusion in high-level vision tasks: A practical infrared and visible image fusion networkbased on progressive semantic injection and scene fidelity [J]. Information Fusion, 2023, 99: 101870 can be used.

[0073] Secondly, the target area of ​​the fused image is detected by combining textual cues and visual cues. For open vocabulary object detection with text as cues, the user-defined text T s Perform category extraction to obtain predefined semantic categories T = {t1, t2, ..., t K}, where K represents the number of semantic categories. Secondly, the fused image and the predefined semantic categories are input into the open vocabulary object detector Grounding DINO to obtain the detection proposal v with text as the hint. gd =Det gd (T,I f ), where Det gd(·,·) denotes the detection process with textual hints.

[0074] However, since directly using the open vocabulary object detector to fusion image I f Detection based on semantic category T can lead to false or missed detections. Therefore, the present invention introduces visual cues to enhance target understanding. When detecting a new category or a category that deviates significantly from predefined semantic categories, only a few visual examples from the scene are needed to understand the category in context.

[0075] Specifically, taking a category in the predefined categories as an example, J semantic target areas are selected in the scene image and cropped to obtain J visual cue image detection frames b j =(x j ,y j ,w j ,h j ), where j = 1, 2, ..., J. x j 、y j Respectively represent the horizontal and vertical coordinates of the upper left corner of the detection box, w j 、h j Represent the width and height of the detection box respectively. Then, the visual embedding V is obtained through the visual encoder, which can be expressed by the following formula:

[0076] V=En(b1,b2,...,b J )

[0077] Wherein En represents a visual encoder, which can be specifically a visual encoder designed in the literature “Jiang Q, Li F, Zeng Z, et al. T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergy [J]. arXiv preprint arXiv: 2403.14610, 2024.”

[0078] At the same time, the fused image passes through the image encoder, which can be specifically designed using the image encoder designed in the literature "Jiang Q, Li F, Zeng Z, et al. T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergy [J]. arXiv preprint arXiv: 2403.14610, 2024.", and is input into the decoder together with the visual embedding to output the detection proposal v based on the visual prompt. vp =De(V,f).

[0079] Where De represents the decoder (specifically, a decoder designed in the literature "Zhu X, Su W, Lu L, et al. Deformabledetr: Deformable transformers for end-to-end object detection[J]. arXiv preprint arXiv:2010.04159, 2020." can be used), and f represents the feature map output by the image encoder.

[0080] In summary, the target detection proposal predicted by the entire multi-hint detection stage can be expressed as follows:

[0081] v N =v gd ∪v vp

[0082] Where N represents the number of initial detection proposals. By combining text and visual cues, the accuracy of the detection model can be significantly improved.

[0083] Since the open vocabulary object detector has potential detection category errors in open scenes, the semantic information is further corrected by the semantic consistency correction module, which is as follows: Figure 2 .

[0084] Remember the initial detection proposal v1,v2,...,v N The corresponding initial category ids are (For example, detection proposals v1, v2, ..., v N The corresponding initial categories are pedestrian, traffic cone, vehicle, ..., pedestrian, and their initial category ids are 2, 7, 1, ..., 2 respectively). Then, the pre-trained image-text matching model CLIP is used to detect the image proposal and the pre-defined semantic category T = {t1, t2, ..., t K} perform independent encoding to obtain the corresponding visual embedding and text embedding Recorded as and K is the number of text categories. The similarity score between visual embedding and text embedding is calculated as follows:

[0085]

[0086] in represents the visual embedding of the n-th detection proposal, represents the text embedding of the k-th category, F nk The prediction confidence corresponding to the n-th detection proposal belongs to the k-th category, and <·,·> represents the dot product operation.

[0087] For each detection proposal, select the class with the highest prediction score As the predicted class label Where .index() is the index calling function, pointing to the confidence score The corresponding category k. If the predicted category With the initial detection category Matching, i.e. judging Is it true? If so, it means that the semantic understanding is consistent and no further correction is required. However, if the predicted category is different from the initial detection category (for example, the predicted category of detection proposal v1 is roadblock, and the predicted category id is 5, which is different from the id of the initial detection category of pedestrian), additional judgment is required based on the following conditions. First, we can retrieve the confidence score corresponding to the initial detection category based on the similarity score:

[0088]

[0089] Then, use the following formula to make a judgment:

[0090]

[0091] Where th1 and th2 are two constant thresholds. If both conditions are met, it is jointly determined that the detection proposal should be updated to the predicted category (for example, the corresponding category id after the detection proposal v1 is updated is 5).

[0092] Finally, after correction, the output detection proposals v1, v2, ..., v N The corresponding categories are roadblocks, traffic cones, vehicles, ..., pedestrians, and the corresponding category IDs are 5, 7, 1, ..., 2).

[0093] Semantic Segmentation:

[0094] In this phase, by integrating the segmentation foundation model TAP, we can semantically segment the objects in the detection proposals, thereby achieving zero-shot semantic segmentation of images in open scenes. Compared with existing open-lexicon semantic segmentation methods, our method can achieve instance-level object segmentation with high accuracy and precision.

[0095] Another embodiment of the present invention further provides an open scene multimodal image-text zero-shot semantic segmentation system, characterized by comprising an RGB-T open vocabulary object detection module and a semantic segmentation module;

[0096] The RGB-T open vocabulary object detection module takes pixel-level paired RGB images, thermal images, and text as multimodal inputs. It uses text information and visual cues as conditions to generate initial detection proposals for the target area in the fused image. It then performs semantic consistency correction on the initial detection proposals to correct ambiguous categories, and then obtains the corrected predicted categories, which are then fed into the semantic segmentation module.

[0097] The semantic segmentation module uses the corrected detection categories as hints for the segmentation base model, performs semantic segmentation on the fused image and generates a text description of the target.

[0098] The multimodal input of the RGB-T open vocabulary object detection module includes RGB image I RGB , Thermal Image I T and user-defined text T s , where the RGB image and the thermal image are paired images collected from the same scene and the same perspective, and the two are paired at the pixel level.

[0099] The RGB-T open vocabulary object detection module includes: image fusion unit, category extraction unit, open vocabulary object detector Grounding DINO, visual cue input unit, visual encoder, image encoder, decoder and merging unit.

[0100] The image fusion unit is used to obtain the dynamic fusion weights of the RGB image and the thermal image using the attention-based fusion module, and perform adaptive fusion on them to obtain the fused image represented as I f .

[0101] Category extraction unit, used for user-defined text T s Perform category extraction and obtain predefined semantic categories T = {t1, t2, ..., t K}, where K represents the number of semantic categories.

[0102] Open vocabulary object detector Grounding DINO, which takes a fused image and predefined semantic categories as input and obtains detection proposals v with text as hints gd =Det gd (T,I f ), where Det gd (·,·) denotes the detection process with textual hints.

[0103] The visual cue input unit is used to receive visual cue input, select J semantic target areas in the scene image, and crop them to obtain J visual cue image detection frames b j =(x j ,y j ,w j ,hj ), where j = 1, 2, ..., J; x j 、y j Respectively represent the horizontal and vertical coordinates of the upper left corner of the detection box, w j 、h j Represent the width and height of the detection box respectively.

[0104] Visual encoder, through the visual encoder to get the visual embedding V: V = En(b1,b2,...,b J ); where En represents the visual encoder.

[0105] The image encoder takes the fused image as input and outputs the encoded result of the fused image to the decoder.

[0106] Decoder, the fused image passes through the image encoder and is input into the decoder together with the visual embedding V. The decoder outputs the detection proposal v based on the visual prompt vp =De(V,f); where De represents the decoder and f represents the feature map output by the image encoder.

[0107] The merging unit is used to merge the detection proposals based on text and vision to obtain the target detection proposal predicted by the entire multi-prompt detection stage, which is expressed as: v N =v gd ∪v vp ; where N represents the number of initial detection proposals.

[0108] The RGB-T open vocabulary object detection module also includes an initial detection category mapping unit, a pre-trained image-text matching model CLIP, a similarity score calculation unit, and a semantic discrimination unit:

[0109] Initial detection category mapping unit, used to determine the initial detection proposal v1,v2,...,v N The corresponding initial category ids are

[0110] The pre-trained image-text matching model CLIP detects image proposals and predefined semantic categories T = {t1, t2, ..., t K} perform independent encoding to obtain the corresponding visual embedding and text embedding Recorded as and K is the number of text categories.

[0111] The similarity score calculation unit calculates the similarity score between the visual embedding and the text embedding as follows:

[0112]

[0113] in represents the visual embedding of the n-th detection proposal, represents the text embedding of the k-th category, F nk The prediction confidence corresponding to the n-th detection proposal belongs to the k-th category, and <·,·> represents the dot product operation.

[0114] Semantic discrimination unit, specifically:

[0115] For each detection proposal, select the class with the highest prediction score As the predicted class label Where .index() is the index calling function, pointing to the confidence score The corresponding category k; if the predicted category With the initial detection category Matching, i.e. judging Is it true? If so, it means that the semantic understanding is consistent and no further correction is required.

[0116] If the predicted category is different from the initial detection category, additional judgment needs to be performed based on the following conditions:

[0117] The confidence score corresponding to the initial detection category is retrieved based on the similarity score:

[0118]

[0119] Then, the following two conditions are used for judgment:

[0120]

[0121] Where th1 and th2 are two constant thresholds. If both conditions are met, it is determined that the detection proposal should be updated to the predicted category.

[0122] The semantic segmentation module performs semantic segmentation on the targets in the detection proposal by integrating the segmentation base model TAP, thereby achieving zero-shot semantic segmentation of images in open scenes.

[0123] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. Open scene multimodal image-text zero-shot semantic segmentation method, characterized by: It consists of two stages: RGB-T open vocabulary object detection stage and semantic segmentation stage; The RGB-T open vocabulary object detection stage uses pixel-level paired RGB images, thermal images, and text as multimodal inputs, uses text information and visual cues as conditions, generates initial detection proposals for the target area of ​​the fused image, performs semantic consistency correction on the initial detection proposals to correct ambiguous categories, and obtains corrected predicted categories. The initial detection proposals are generated for the target area of ​​the fused image using text information and visual cues as conditions, specifically using the following steps: Step S1: Use the attention-based fusion module to obtain the dynamic fusion weights of the RGB image and the thermal image, perform adaptive fusion on them, and obtain the fused image represented as I f ; Step S2: User-defined text T s Perform category extraction and obtain predefined semantic categories T = {t1, t2, ..., t K }, where K represents the number of semantic categories. Secondly, the fused image and the predefined semantic categories are input into the open vocabulary object detector Grounding DINO to obtain the detection proposal v with text as the hint. gd =Det gd (T,I f ), where Det gd (·,·) indicates the detection process with text prompts; Step S3: Select J semantic target areas in the scene image and crop them to obtain J visual cue image detection frames b j =(x j ,y j ,w j ,h j ), where j = 1, 2, ..., J; x j 、y j Respectively represent the horizontal and vertical coordinates of the upper left corner of the detection box, w j 、h j Represent the width and height of the detection box respectively; the visual embedding V is obtained by the visual encoder: V=En(b1,b2,...,b J );in En represents the visual encoder; Step S4: The fused image passes through the image encoder and is input into the decoder together with the visual embedding V, which outputs the detection proposal v based on the visual prompt vp =De(V,f); where De represents the decoder and f represents the feature map output by the image encoder; Step S5: The target detection proposal predicted by the entire multi-hint detection stage is expressed as: v N =v gd ∪v vp ; Where N represents the number of initial detection proposals; The semantic consistency correction is performed on the initial detection proposal to correct the ambiguous category and obtain the corrected predicted category, specifically: Initial detection proposals v1,v2,...,v N The corresponding initial category ids are The pre-trained image-text matching model CLIP is used to detect image proposals and predefined semantic categories T = {t1, t2, ..., t K } perform independent encoding to obtain the corresponding visual embedding and text embedding Recorded as and K is the number of text categories; the similarity score between visual embedding and text embedding is calculated as follows: in represents the visual embedding of the n-th detection proposal, represents the text embedding of the k-th category, F nk The prediction confidence corresponding to the nth detection proposal belongs to the kth category, <·,·> represents the dot product operation; For each detection proposal, select the class with the highest prediction score As the predicted class label Where .index() is the index calling function, pointing to the confidence score The corresponding category k; if the predicted category With the initial detection category Matching, i.e. judging Is it true? If so, it means that the semantic understanding is consistent and no further correction is needed; If the predicted category is different from the initial detection category, additional judgment needs to be performed based on the following conditions: The confidence score corresponding to the initial detection category is retrieved based on the similarity score: Then, the following two conditions are used for judgment: Where th1 and th2 are two constant thresholds. If both conditions are met, it is determined that the detection proposal should be updated to the predicted category. The semantic segmentation stage uses the corrected detection category as a hint for the segmentation base model, performs semantic segmentation on the fused image and generates a text description of the target.

2. The open scene multimodal image-text zero-shot semantic segmentation method according to claim 1, characterized in that: In the RGB-T open vocabulary object detection stage, the multimodal input includes an RGB image I RGB , Thermal Image I T and user-defined text T s , where the RGB image and the thermal image are paired images collected from the same scene and the same perspective, and the two are paired at the pixel level.

3. The open scene multimodal image-text zero-shot semantic segmentation method according to claim 1 or 2, characterized in that: In the semantic segmentation stage, the target in the detection proposal is semantically segmented by integrating the segmentation basic model TAP, thereby achieving zero-shot semantic segmentation of images in open scenes.

4. An open-scene multimodal image-text zero-shot semantic segmentation system, characterized by including an RGB-T open-vocabulary object detection module and a semantic segmentation module; The RGB-T open vocabulary object detection module takes pixel-level paired RGB images, thermal images, and text as multimodal inputs, and uses text information and visual cues as conditions to generate initial detection proposals for the target area of ​​the fused image. The initial detection proposals are then semantically corrected to correct ambiguous categories, and the corrected predicted categories are fed into the semantic segmentation module. The semantic segmentation module uses the corrected detection category as a hint for the segmentation base model, performs semantic segmentation on the fused image and generates a text description of the target; The RGB-T open vocabulary object detection module includes: an image fusion unit, a category extraction unit, an open vocabulary object detector Grounding DINO, a visual cue input unit, a visual encoder, an image encoder, a decoder, and a merging unit; The image fusion unit is used to obtain the dynamic fusion weights of the RGB image and the thermal image using the attention-based fusion module, and perform adaptive fusion on them to obtain the fused image represented as I f ; Category extraction unit, used for user-defined text T s Perform category extraction and obtain predefined semantic categories T = {t1, t2, ..., t K }, where K represents the number of semantic categories; Open vocabulary object detector Grounding DINO, which takes a fused image and predefined semantic categories as input and obtains detection proposals v with text as hints gd =Det gd (T,I f ), where Det gd (·,·) indicates the detection process with text prompts; The visual cue input unit is used to receive visual cue input, select J semantic target areas in the scene image, and crop them to obtain J visual cue image detection frames b j =(x j ,y j ,w j ,h j ), where j = 1, 2, ..., J; x j 、y j Respectively represent the horizontal and vertical coordinates of the upper left corner of the detection box, w j 、h j Respectively represent the width and height of the detection box; Visual encoder, through the visual encoder to get the visual embedding V: V = En(b1,b2,...,b J ); where En represents the visual encoder; An image encoder, which takes the fused image as input and outputs an encoding result of the fused image to the decoder; Decoder, the fused image passes through the image encoder and is input into the decoder together with the visual embedding V. The decoder outputs the detection proposal v based on the visual prompt vp =De(V,f); where De represents the decoder and f represents the feature map output by the image encoder; The merging unit is used to merge the detection proposals based on text and vision to obtain the target detection proposal predicted by the entire multi-prompt detection stage, which is expressed as: v N =v gd ∪v vp ; Where N represents the number of initial detection proposals; The RGB-T open vocabulary object detection module also includes an initial detection category mapping unit, a pre-trained image-text matching model CLIP, a similarity score calculation unit, and a semantic discrimination unit: The initial detection category mapping unit is used to determine the initial detection proposals v1, v2, ..., v N The corresponding initial category ids are The pre-trained image-text matching model CLIP detects image proposals and predefined semantic categories T = {t1, t2, ..., t K } perform independent encoding to obtain the corresponding visual embedding and text embedding Recorded as and K is the number of text categories; The similarity score calculation unit calculates the similarity score between the visual embedding and the text embedding as follows: in represents the visual embedding of the n-th detection proposal, represents the text embedding of the k-th category, F nk The prediction confidence corresponding to the nth detection proposal belongs to the kth category, <·,·> represents the dot product operation; The semantic discrimination unit is specifically: For each detection proposal, select the class with the highest prediction score As the predicted class label Where .index() is the index calling function, pointing to the confidence score The corresponding category k; if the predicted category With the initial detection category Matching, i.e. judging Is it true? If so, it means that the semantic understanding is consistent and no further correction is needed; If the predicted category is different from the initial detection category, additional judgment needs to be performed based on the following conditions: The confidence score corresponding to the initial detection category is retrieved based on the similarity score: Then, the following two conditions are used for judgment: Where th1 and th2 are two constant thresholds. If both conditions are met, it is determined that the detection proposal should be updated to the predicted category.

5. The open scene multimodal image-text zero-shot semantic segmentation system according to claim 1, characterized in that: The multimodal input of the RGB-T open vocabulary object detection module includes an RGB image I RGB , Thermal Image I T and user-defined text T s , where the RGB image and the thermal image are paired images collected from the same scene and the same perspective, and the two are paired at the pixel level.

6. The open scene multimodal image-text zero-shot semantic segmentation method according to claim 4 or 5, characterized in that: The semantic segmentation module performs semantic segmentation on the targets in the detection proposal by integrating the segmentation basic model TAP, thereby achieving zero-shot semantic segmentation of images in open scenes.

Images