Open scene multi-modal image-text zero sample semantic segmentation method and system

Abstract

The invention discloses an open scene multi-modal image-text zero sample semantic segmentation method and system, belongs to the technical field of image semantic segmentation, is suitable for processing thermal inductance and infrared images, and improves the zero sample generalization performance of RGB-T semantic segmentation in multiple scenes. Comprising two stages: an RGB-T open vocabulary target detection stage and a semantic segmentation stage. In the RGB-T open vocabulary target detection stage, pixel-level paired RGB images, thermal images and texts are used as multi-modal input, text information and visual prompts are used as conditions, an initial detection proposal is generated for a target area of a fused image, semantic consistency correction is carried out on the initial detection proposal, ambiguous categories are corrected, and the target area of the fused image is obtained. And obtaining a corrected prediction category. In the semantic segmentation stage, the corrected detection category is used as a prompt for segmenting the basic model, semantic segmentation is performed on the fused image, and text description of the target is generated.

Description

Technical Field

[0001] The present invention belongs to the technical field of image semantic segmentation, and in particular relates to an open scene multimodal image-text zero-sample semantic segmentation method and system. Background Art

[0002] Semantic segmentation plays a key role in achieving scene understanding for unmanned systems, especially in autonomous driving, robotic manipulation, and remote sensing. Although previous studies have achieved significant segmentation results on standard datasets based on RGB images, these methods often encounter difficulties in accurately segmenting the target object from the background in low visibility environments caused by bad weather or low light conditions. To overcome these limitations, researchers have begun to use thermal / infrared images to improve the performance of visual perception tasks. Despite some progress, traditional RGB-T semantic segmentation methods are mostly limited to training on predefined categories, which limits their potential to generalize to more scenarios.

[0003] In recent years, the rapid development of visual language models (VLMs) and open vocabulary learning has paved the way for expanding the capabilities of semantic segmentation tasks in open environments. Compared with traditional semantic segmentation methods, this method that combines VLMs and open vocabulary learning can support text input of any category and achieve instance-level semantic segmentation, so it is also called open vocabulary semantic segmentation.

[0004] Most existing open vocabulary semantic segmentation methods are implemented in a two-stage approach, which mainly includes two ideas.

[0005] The first category of methods first uses a visual base model to generate class-independent mask proposals, and then applies a pre-trained image-text matching model to classify these mask proposals. A major challenge of this approach is that its performance is highly dependent on the accurate generation of the initial mask and the subsequent classification step. In addition, since image-text matching models are usually trained on large-scale RGB image datasets, they may lead to classification errors when there are large differences between the image to be detected (target domain) and the training dataset (source domain).

[0006] In order to reduce potential classification errors, the second type of method first uses an open vocabulary object detector to generate a detection box for the category to be detected, and then uses a visual base model to segment the object. However, this method may still encounter the problem of false detection or missed detection in the object detection stage, because the open vocabulary object detector is also trained based on a fixed text description library. When the input text description has semantically ambiguous overlap with the categories in the training text library, an incorrect confidence score may be generated, which may lead to false detection or missed detection.

[0007] In addition, most existing open vocabulary semantic segmentation models are limited to the use of RGB single modality information. Due to the heterogeneity between different modal data, these models encounter difficulties in category understanding when they need to process thermal or infrared images. Summary of the invention

[0008] In view of this, the present invention provides an open scene multimodal image-text zero-sample semantic segmentation method and system, which is suitable for processing thermal and infrared images and improving the zero-sample generalization performance of RGB-T semantic segmentation in multiple scenes.

[0009] To achieve the above-mentioned object, the technical solution of the present invention is: an open scene multimodal image-text zero-sample semantic segmentation method, which includes two stages: an RGB-T open vocabulary target detection stage and a semantic segmentation stage.

[0010] The RGB-T open vocabulary object detection stage takes pixel-level paired RGB images and thermal images as well as text as multimodal inputs, and uses text information and visual cues as conditions to generate initial detection proposals for the target area of ​​the fused image. The initial detection proposals are semantically consistent corrected to correct ambiguous categories and obtain the corrected predicted categories.

[0011] In the semantic segmentation stage, the corrected detection categories are used as hints for the segmentation base model to perform semantic segmentation on the fused image and generate text descriptions of the targets.

[0012] Furthermore, in the RGB-T open vocabulary object detection stage, multimodal input includes RGB image I RGB 、Thermal Image I T and user-defined text T s , where the RGB image and the thermal image are paired images collected from the same scene and the same perspective, and the two are paired at the pixel level.

[0013] Furthermore, using text information and visual cues as conditions, an initial detection proposal is generated for the target area of ​​the fused image. The specific steps are as follows:

[0014] Step S1: Use the attention-based fusion module to obtain the dynamic fusion weights of the RGB image and the thermal image, perform adaptive fusion on them, and obtain the fused image represented as I f .

[0015] Step S2: User-defined text T s Perform category extraction to obtain the predefined semantic category T = {t1, t2, ..., t K}, where K represents the number of semantic categories. Secondly, the fused image and the predefined semantic categories are input into the open vocabulary object detector Grounding DINO to obtain the detection proposal v with text as the hint. gd =Det gd (T,I f ), where Det gd (·,·) denotes the detection process with textual hints.

[0016] Step S3: Select J semantic target areas in the scene image and crop them to obtain J visual cue image detection frames b j =(x j ,y j ,w j ,h j ), where j = 1, 2, ..., J; x j ,y j Respectively represent the horizontal and vertical coordinates of the upper left corner of the detection box, w j 、h j Represent the width and height of the detection box respectively; the visual embedding V is obtained through the visual encoder: V=En(b1,b2,...,b J ), where En represents the visual encoder.

[0017] Step S4: The fused image passes through the image encoder and is input into the decoder together with the visual embedding V, which outputs the detection proposal v with visual cues vp =De(V,f); where De represents the decoder and f represents the feature map output by the image encoder.

[0018] Step S5: The object detection proposal predicted by the entire multi-hint detection stage is represented as: N =v gd ∪v vp ; where N represents the number of initial detection proposals.

[0019] Furthermore, semantic consistency correction is performed on the initial detection proposal to correct the ambiguous categories and obtain the corrected predicted categories, specifically:

[0020] Initial detection proposal v1,v2,...,v N The corresponding initial category ids are

[0021] The pre-trained image-text matching model CLIP is used to detect image proposals and predefined semantic categories T = {t1, t2, ..., t K} perform independent encoding to obtain the corresponding visual embedding and text embedding Recorded as and K is the number of text categories; the similarity score between visual embedding and text embedding is calculated as follows:

[0022]

[0023] in represents the visual embedding of the nth detection proposal, represents the text embedding of the kth category, F nk The prediction confidence corresponding to the nth detection proposal belonging to the kth category, <·,·> represents the dot product operation.

[0024] For each detection proposal, select the class with the highest prediction score As predicted class label Where .index() is the index calling function, pointing to the confidence score The corresponding category k; if the predicted category With the initial detection category Matching, i.e. judging Is it true? If so, it means that the semantic understanding is consistent and no further correction is required.

[0025] If the predicted category is different from the initial detection category, additional judgment needs to be performed based on the following conditions:

[0026] The confidence score corresponding to the initial detection category is retrieved based on the similarity score:

[0027]

[0028] Then, the following two conditions are used for judgment:

[0029]

[0030] Where th1 and th2 are two constant thresholds. If both conditions are met, it is jointly determined that the detection proposal should be updated to the predicted category.

[0031] Furthermore, in the semantic segmentation stage, by integrating the segmentation basic model TAP, the targets in the detection proposal are semantically segmented, thereby achieving zero-shot semantic segmentation of images in open scenes.

[0032] The present invention also provides another open scene multimodal image-text zero-sample semantic segmentation system, including an RGB-T open vocabulary object detection module and a semantic segmentation module.

[0033] The RGB-T open vocabulary object detection module takes pixel-level paired RGB images and thermal images as well as text as multimodal inputs, and uses text information and visual cues as conditions to generate initial detection proposals for the target area of ​​the fused image. It then performs semantic consistency correction on the initial detection proposals to correct ambiguous categories, obtains the corrected predicted categories, and sends them to the semantic segmentation module.

[0034] The semantic segmentation module uses the corrected detection categories as hints for the segmentation base model, performs semantic segmentation on the fused image and generates text descriptions of the targets.

[0035] Furthermore, the multimodal input of the RGB-T open vocabulary object detection module includes the RGB image I RGB 、Thermal Image I T and user-defined text T s , where the RGB image and the thermal image are paired images collected from the same scene and the same perspective, and the two are paired at the pixel level.

[0036] Furthermore, the RGB-T open vocabulary object detection module includes: an image fusion unit, a category extraction unit, an open vocabulary object detector Grounding DINO, a visual cue input unit, a visual encoder, an image encoder, a decoder and a merging unit.

[0037] The image fusion unit is used to obtain the dynamic fusion weights of the RGB image and the thermal image by using the attention-based fusion module, and perform adaptive fusion on them to obtain the fused image represented as I f .

[0038] Category extraction unit, used to extract user-defined text T s Perform category extraction to obtain the predefined semantic category T = {t1, t2, ..., t K}, where K represents the number of semantic categories.

[0039] Open vocabulary object detector Grounding DINO, which takes a fused image and predefined semantic categories as input and obtains detection proposals v with text as hints gd =Det gd (T,I f ), where Det gd (·,·) denotes the detection process with textual hints.

[0040] The visual cue input unit is used to receive the visual cue input, select J semantic target areas in the scene image, and crop them to obtain J visual cue image detection frames b j =(x j ,y j ,wj ,h j ), where j = 1, 2, ..., J; x j ,y j Respectively represent the horizontal and vertical coordinates of the upper left corner of the detection box, w j 、h j Represent the width and height of the detection box respectively.

[0041] Visual encoder, through the visual encoder to get the visual embedding V: V = En(b1, b2, ..., b J ), where En represents the visual encoder.

[0042] The image encoder takes the fused image as input and outputs the encoded result of the fused image to the decoder.

[0043] Decoder: The fused image passes through the image encoder and is input into the decoder together with the visual embedding V. The decoder outputs a detection proposal v based on the visual prompt. vp =De(V,f); where De represents the decoder and f represents the feature map output by the image encoder.

[0044] The merging unit is used to merge the detection proposals with text as the prompt and the detection proposals with vision as the prompt to obtain the target detection proposal predicted by the entire multi-prompt detection stage, expressed as: v N =v gd ∪v vp ; where N represents the number of initial detection proposals.

[0045] Furthermore, the RGB-T open vocabulary object detection module also includes an initial detection category mapping unit, a pre-trained image-text matching model CLIP, a similarity score calculation unit, and a semantic discrimination unit:

[0046] Initial detection category mapping unit, used to determine the initial detection proposal v1,v2,...,v N The corresponding initial category ids are

[0047] The pre-trained image-text matching model CLIP detects image proposals and predefined semantic categories T = {t1, t2, ..., t K} perform independent encoding to obtain the corresponding visual embedding and text embedding Recorded as and K is the number of text categories.

[0048] The similarity score calculation unit calculates the similarity score between the visual embedding and the text embedding as follows:

[0049]

[0050] in represents the visual embedding of the nth detection proposal, represents the text embedding of the kth category, F nk The prediction confidence corresponding to the nth detection proposal belonging to the kth category, <·,·> represents the dot product operation.

[0051] Semantic discrimination unit, specifically:

[0052] For each detection proposal, select the class with the highest prediction score As predicted class label Where .index() is the index calling function, pointing to the confidence score The corresponding category k; if the predicted category With the initial detection category Matching, i.e. judging Is it true? If so, it means that the semantic understanding is consistent and no further correction is required.

[0053] If the predicted category is different from the initial detection category, additional judgment needs to be performed based on the following conditions:

[0054] The confidence score corresponding to the initial detection category is retrieved based on the similarity score:

[0055]

[0056] Then, the following two conditions are used for judgment:

[0057]

[0058] Where th1 and th2 are two constant thresholds. If both conditions are met, it is jointly determined that the detection proposal should be updated to the predicted category.

[0059] Furthermore, the semantic segmentation module, by integrating the segmentation basic model TAP, performs semantic segmentation on the targets in the detection proposal, thereby achieving zero-shot semantic segmentation of images in open scenes.

[0060] Beneficial effects:

[0061] 1; The open scene multimodal image-text zero-sample semantic segmentation method and system provided by the present invention improve the generalization ability and segmentation performance of the RGB-T semantic segmentation model in an open environment. The present invention introduces open vocabulary learning, so that the semantic segmentation model is no longer limited by cumbersome model training and predefined semantic categories, and can process any input category in any scene, with zero-sample generalization capability. Compared with the existing open vocabulary semantic segmentation model, the present invention can better adapt to complex and changing open environments by utilizing multimodal information, thereby providing more stable and reliable segmentation results. In addition, the multi-cue detection proposed in the present invention can not only enhance the model's understanding of known categories, but also improve its adaptability to new categories or changing scenes.

[0062] 2: By combining visual cues and text cues, the present invention can improve the model's adaptability to new categories or changing scenarios, thereby performing more accurate detection.

[0063] 3: The semantic consistency correction proposed in the present invention can effectively deal with the semantic ambiguity problem between categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is an open scene multimodal image-text zero-shot semantic segmentation method and system framework diagram;

[0065] Figure 2 This is the schematic diagram of the semantic consistency correction module. DETAILED DESCRIPTION

[0066] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0067] The present invention provides an open scene multimodal image-text zero-shot semantic segmentation method and system to improve the zero-shot generalization performance of RGB-T semantic segmentation in multiple scenes. The specific method framework is as follows Figure 1 shown.

[0068] The open scene multimodal image-text zero-sample semantic segmentation method of the present invention mainly includes two stages: the RGB-T open vocabulary target detection stage and the semantic segmentation stage. In the first stage, pixel-level paired RGB images and thermal images and text are used as multimodal inputs, and text information and visual cues are used as conditions to generate an initial detection box for the target area of ​​the fused image, and the ambiguous categories are corrected through the semantic consistency correction module to obtain the final predicted category. In the second stage, the detection box proposal and the corrected detection category are used as prompts for the segmentation base model to perform semantic segmentation on the fused image and generate a text description of the target. By executing these two stages sequentially, the method obtains a semantic understanding of the scene, which can be further applied to downstream perception tasks.

[0069] RGB-T Open Vocabulary Object Detection:

[0070] This stage mainly consists of three parts: multimodal input, multi-hint detection and semantic consistency correction. The multimodal input is represented as follows: RGB image I RGB 、Thermal Image I T and user-defined text T s , where the RGB image and the thermal image are paired images collected from the same scene and the same perspective, and the two are paired at the pixel level.

[0071] In multi-cue detection, we first consider the richness and information content of different modalities in the input image, and use the attention-based fusion module to obtain the dynamic fusion weights of the two modalities, and then perform adaptive fusion on them to obtain the fused image represented as I f Specifically, the fusion module adopts a global-local attention mechanism to focus on the global context and local details according to the input image, and adaptively updates the fusion weights to maintain effective fusion even when the quality of one of the modalities is impaired or suboptimal.

[0072] The attention-based fusion module used in the embodiment of the present invention adopts an existing module, and specifically can use the attention-based fusion module involved in Tang L, Zhang H, Xu H, et al. Rethinking the necessity of image fusion in high-level vision tasks: A practical infrared and visible image fusion networkbased on progressive semantic injection and scene fidelity [J]. Information Fusion, 2023, 99: 101870.

[0073] Secondly, the target area of ​​the fused image is detected by using textual cues and visual cues. For open vocabulary object detection with text as cues, the user-defined text T s Perform category extraction to obtain the predefined semantic category T = {t1, t2, ..., t K}, where K represents the number of semantic categories. Secondly, the fused image and the predefined semantic categories are input into the open vocabulary object detector Grounding DINO to obtain the detection proposal v with text as the hint. gd =Det gd (T,I f ), where Det gd(·,·) denotes the detection process with textual hints.

[0074] However, since directly using the open vocabulary object detector for the fused image I f If the detection is performed on a semantic category T, there will be problems of false detection or missed detection, so the present invention introduces visual cues to enhance the understanding of the target. When detecting a new category or a category with a large semantic deviation from a predefined category, only a few visual examples in the scene are needed to understand the category in context.

[0075] Specifically, taking a category in the predefined categories as an example, J semantic target areas are selected in the scene image and cropped to obtain J visual cue image detection boxes b j =(x j ,y j ,w j ,h j ), where j = 1, 2, ..., J. x j ,y j Respectively represent the horizontal and vertical coordinates of the upper left corner of the detection box, w j 、h j Respectively represent the width and height of the detection box. Next, the visual embedding V is obtained through the visual encoder, which can be expressed by the following formula:

[0076] V=En(b1,b2,...,b J )

[0077] Wherein En represents a visual encoder, which can specifically use a visual encoder designed in the literature “Jiang Q, Li F, Zeng Z, et al. T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergy [J]. arXiv preprint arXiv: 2403.14610, 2024.”

[0078] At the same time, the fused image passes through the image encoder, which can be specifically designed using the image encoder designed in the literature "Jiang Q, Li F, Zeng Z, et al. T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergy [J]. arXiv preprint arXiv: 2403.14610, 2024.", and is input into the decoder together with the visual embedding to output the detection proposal v with visual prompts. vp =De(V,f).

[0079] Where De represents a decoder (specifically, a decoder designed in the literature “Zhu X, Su W, Lu L, et al. Deformabledetr: Deformable transformers for end-to-end object detection[J]. arXiv preprint arXiv:2010.04159, 2020.” can be used), and f represents a feature map output by the image encoder.

[0080] In summary, the target detection proposal predicted by the entire multi-hint detection stage can be expressed as follows:

[0081] v N =v gd ∪v vp

[0082] Where N represents the number of initial detection proposals. By combining text and visual cues, the accuracy of the detection model can be significantly improved.

[0083] Since the open vocabulary object detector has potential detection category errors in open scenes, the semantic information is further corrected by the semantic consistency correction module, which is as follows Figure 2 .

[0084] Remember the initial detection proposal v1,v2,...,v N The corresponding initial category ids are (For example, detection proposals v1,v2,...,v N The corresponding initial categories are pedestrian, traffic cone, vehicle, ..., pedestrian, and their initial category ids are 2, 7, 1, ..., 2 respectively). Then, the pre-trained image-text matching model CLIP is used to detect the image proposal and the pre-defined semantic category T = {t1, t2, ..., t K} perform independent encoding to obtain the corresponding visual embedding and text embedding Recorded as and K is the number of text categories. The similarity score between visual embedding and text embedding is calculated as follows:

[0085]

[0086] in represents the visual embedding of the nth detection proposal, represents the text embedding of the kth category, F nk The prediction confidence corresponding to the nth detection proposal belonging to the kth category, <·,·> represents the dot product operation.

[0087] For each detection proposal, select the class with the highest prediction score As predicted class label Where .index() is the index calling function, pointing to the confidence score The corresponding category k. If the predicted category With the initial detection category Matching, i.e. judging Is it true? If it is true, it means that the semantic understanding is consistent and no further correction is required. However, if the predicted category is different from the initial detection category (for example, the predicted category of the detection proposal v1 is roadblock, and the predicted category id is 5, which is different from the id of the initial detection category of pedestrian), additional judgments need to be performed based on the following conditions. First, we can retrieve the confidence score corresponding to the initial detection category based on the similarity score:

[0088]

[0089] Then, use the following formula to make a judgment:

[0090]

[0091] Where th1 and th2 are two constant thresholds. If both conditions are met, it is jointly determined that the detection proposal should be updated to the predicted category (for example, the category id corresponding to the detection proposal v1 after the update is 5).

[0092] Finally, after correction, the output detection proposals v1,v2,...,v N The corresponding categories are roadblocks, traffic cones, vehicles, ..., pedestrians, and the corresponding category IDs are 5, 7, 1, ..., 2).

[0093] Semantic Segmentation:

[0094] In this stage, by integrating the segmentation basic model TAP, the objects in the detection proposal can be semantically segmented, thereby achieving zero-shot semantic segmentation of images in open scenes. Compared with the existing open vocabulary semantic segmentation methods, the present invention can achieve instance-level object segmentation and achieve high accuracy and precision performance.

[0095] Another embodiment of the present invention further provides an open scene multimodal image-text zero-shot semantic segmentation system, characterized in that it includes an RGB-T open vocabulary object detection module and a semantic segmentation module;

[0096] The RGB-T open vocabulary object detection module takes pixel-level paired RGB images and thermal images as well as text as multimodal inputs, and uses text information and visual cues as conditions to generate initial detection proposals for the target area of ​​the fused image. It then performs semantic consistency correction on the initial detection proposals to correct ambiguous categories, obtains the corrected predicted categories, and feeds them into the semantic segmentation module.

[0097] The semantic segmentation module uses the corrected detection categories as hints for the segmentation base model, performs semantic segmentation on the fused image and generates text descriptions of the targets.

[0098] The multimodal input of the RGB-T open vocabulary object detection module includes an RGB image I RGB 、Thermal Image I T and user-defined text T s , where the RGB image and the thermal image are paired images collected from the same scene and the same perspective, and the two are paired at the pixel level.

[0099] The RGB-T open vocabulary object detection module includes: an image fusion unit, a category extraction unit, an open vocabulary object detector Grounding DINO, a visual cue input unit, a visual encoder, an image encoder, a decoder, and a merging unit.

[0100] The image fusion unit is used to obtain the dynamic fusion weights of the RGB image and the thermal image by using the attention-based fusion module, and perform adaptive fusion on them to obtain the fused image represented as I f .

[0101] Category extraction unit, used to extract user-defined text T s Perform category extraction to obtain the predefined semantic category T = {t1, t2, ..., t K}, where K represents the number of semantic categories.

[0102] Open vocabulary object detector Grounding DINO, which takes a fused image and predefined semantic categories as input and obtains detection proposals v with text as hints gd =Det gd (T,I f ), where Det gd (·,·) denotes the detection process with textual hints.

[0103] The visual cue input unit is used to receive the visual cue input, select J semantic target areas in the scene image, and crop them to obtain J visual cue image detection frames b j =(x j ,y j ,w j ,hj ), where j = 1, 2, ..., J; x j ,y j Respectively represent the horizontal and vertical coordinates of the upper left corner of the detection box, w j 、h j Represent the width and height of the detection box respectively.

[0104] Visual encoder, through the visual encoder to get the visual embedding V: V = En(b1, b2, ..., b J ), where En represents the visual encoder.

[0105] The image encoder takes the fused image as input and outputs the encoded result of the fused image to the decoder.

[0106] Decoder: The fused image passes through the image encoder and is input into the decoder together with the visual embedding V. The decoder outputs a detection proposal v based on the visual prompt. vp =De(V,f); where De represents the decoder and f represents the feature map output by the image encoder.

[0107] The merging unit is used to merge the detection proposals with text as the prompt and the detection proposals with vision as the prompt to obtain the target detection proposal predicted by the entire multi-prompt detection stage, expressed as: v N =v gd ∪v vp ; where N represents the number of initial detection proposals.

[0108] The RGB-T open vocabulary object detection module also includes an initial detection category mapping unit, a pre-trained image-text matching model CLIP, a similarity score calculation unit, and a semantic discrimination unit:

[0109] Initial detection category mapping unit, used to determine the initial detection proposal v1,v2,...,v N The corresponding initial category ids are

[0110] The pre-trained image-text matching model CLIP detects image proposals and predefined semantic categories T = {t1, t2, ..., t K} perform independent encoding to obtain the corresponding visual embedding and text embedding Recorded as and K is the number of text categories.

[0111] The similarity score calculation unit calculates the similarity score between the visual embedding and the text embedding as follows:

[0112]

[0113] in represents the visual embedding of the nth detection proposal, represents the text embedding of the kth category, F nk The prediction confidence corresponding to the nth detection proposal belonging to the kth category, <·,·> represents the dot product operation.

[0114] Semantic discrimination unit, specifically:

[0115] For each detection proposal, select the class with the highest prediction score As predicted class label Where .index() is the index calling function, pointing to the confidence score The corresponding category k; if the predicted category With the initial detection category Matching, i.e. judging Is it true? If so, it means that the semantic understanding is consistent and no further correction is required.

[0116] If the predicted category is different from the initial detection category, additional judgment needs to be performed based on the following conditions:

[0117] The confidence score corresponding to the initial detection category is retrieved based on the similarity score:

[0118]

[0119] Then, the following two conditions are used for judgment:

[0120]

[0121] Where th1 and th2 are two constant thresholds. If both conditions are met, it is jointly determined that the detection proposal should be updated to the predicted category.

[0122] The semantic segmentation module performs semantic segmentation on the targets in the detection proposal by integrating the segmentation base model TAP, thereby achieving zero-shot semantic segmentation of images in open scenes.

[0123] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An open scene multimodal image-text zero-shot semantic segmentation method, characterized by: It consists of two stages: RGB-T open vocabulary object detection stage and semantic segmentation stage; The RGB-T open vocabulary object detection stage uses pixel-level paired RGB images and thermal images and text as multimodal inputs, uses text information and visual cues as conditions, generates initial detection proposals for the target area of ​​the fused image, performs semantic consistency correction on the initial detection proposals to correct ambiguous categories, and obtains corrected predicted categories; The semantic segmentation stage uses the corrected detection category as a hint for the segmentation base model, performs semantic segmentation on the fused image and generates a text description of the target.

2. The open scene multimodal image-text zero-shot semantic segmentation method according to claim 1, characterized in that: The RGB-T open vocabulary object detection stage, the multimodal input includes an RGB image I RGB 、Thermal Image I T and user-defined text T s , where the RGB image and the thermal image are paired images collected from the same scene and the same perspective, and the two are paired at the pixel level.

3. The open scene multimodal image-text zero-shot semantic segmentation method according to claim 1 or 2, characterized in that: The text information and visual cues are used as conditions to generate an initial detection proposal for the target area of ​​the fused image, specifically using the following steps: Step S1: Use the attention-based fusion module to obtain the dynamic fusion weights of the RGB image and the thermal image, perform adaptive fusion on them, and obtain the fused image represented as I f ; Step S2: User-defined text T s Perform category extraction to obtain the predefined semantic category T = {t1, t2, ..., t K }, where K represents the number of semantic categories. Secondly, the fused image and the predefined semantic categories are input into the open vocabulary object detector Grounding DINO to obtain the detection proposal v with text as the hint. gd =Det gd (T,I f ), where Det gd (·,·) indicates the detection process with textual hints; Step S3: Select J semantic target areas in the scene image and crop them to obtain J visual cue image detection frames b j =(x j ,y j ,w j ,h j ), where j = 1, 2, ..., J; x j ,y j Respectively represent the horizontal and vertical coordinates of the upper left corner of the detection box, w j 、h j Represent the width and height of the detection box respectively; the visual embedding V is obtained through the visual encoder: V=En(b1,b2,...,b J );in En represents the visual encoder; Step S4: The fused image passes through the image encoder and is input into the decoder together with the visual embedding V, which outputs the detection proposal v with visual cues vp =De(V,f); where De represents the decoder, and f represents the feature map output by the image encoder; Step S5: The object detection proposal predicted by the entire multi-hint detection stage is represented as: N =v gd ∪v vp ; where N represents the number of initial detection proposals.

4. The open scene multimodal image-text zero-shot semantic segmentation method according to claim 3, characterized in that: The semantic consistency correction of the initial detection proposal corrects the ambiguous category to obtain the corrected prediction category, which is specifically: Initial detection proposal v1,v2,...,v N The corresponding initial category ids are The pre-trained image-text matching model CLIP is used to detect image proposals and predefined semantic categories T = {t1, t2, ..., t K } perform independent encoding to obtain the corresponding visual embedding and text embedding Recorded as n=1,2,...,N and k=1,2,...,K, K is the number of text categories; the similarity score between visual embedding and text embedding is calculated as follows: in represents the visual embedding of the nth detection proposal, represents the text embedding of the kth category, F nk The prediction confidence corresponding to the nth detection proposal belonging to the kth category, <·,·> represents the dot product operation; For each detection proposal, select the class with the highest prediction score As predicted class label Where .index() is the index calling function, pointing to the confidence score The corresponding category k; if the predicted category With the initial detection category Matching, i.e. judging Is it true? If so, it means that the semantic understanding is consistent and no further correction is needed; If the predicted category is different from the initial detection category, additional judgment needs to be performed based on the following conditions: The confidence score corresponding to the initial detection category is retrieved based on the similarity score: Then, the following two conditions are used for judgment: Where th1 and th2 are two constant thresholds. If both conditions are met, it is jointly determined that the detection proposal should be updated to the predicted category.

5. The open scene multimodal image-text zero-shot semantic segmentation method according to any one of claims 1 to 4, characterized in that: In the semantic segmentation stage, the target in the detection proposal is semantically segmented by integrating the segmentation basic model TAP, thereby achieving zero-sample semantic segmentation of images in open scenes.

6. An open scene multimodal image-text zero-shot semantic segmentation system, characterized by including an RGB-T open vocabulary object detection module and a semantic segmentation module; The RGB-T open vocabulary object detection module uses pixel-level paired RGB images and thermal images and text as multimodal inputs, uses text information and visual cues as conditions, generates initial detection proposals for the target area of ​​the fused image, performs semantic consistency correction on the initial detection proposals to correct ambiguous categories, obtains the corrected predicted categories, and sends them to the semantic segmentation module; The semantic segmentation module uses the corrected detection category as a hint for the segmentation base model, performs semantic segmentation on the fused image and generates a text description of the target.

7. The open scene multimodal image-text zero-shot semantic segmentation system according to claim 1, characterized in that: The multimodal input of the RGB-T open vocabulary object detection module includes an RGB image I RGB 、Thermal Image I T and user-defined text T s , where the RGB image and the thermal image are paired images collected from the same scene and the same perspective, and the two are paired at the pixel level.

8. The open scene multimodal image-text zero-shot semantic segmentation system according to claim 1 or 2, characterized in that: The RGB-T open vocabulary object detection module includes: an image fusion unit, a category extraction unit, an open vocabulary object detector Grounding DINO, a visual cue input unit, a visual encoder, an image encoder, a decoder and a merging unit; The image fusion unit is used to obtain the dynamic fusion weights of the RGB image and the thermal image by using the attention-based fusion module, and perform adaptive fusion on them to obtain the fused image represented as I f ; Category extraction unit, used to extract user-defined text T s Perform category extraction to obtain the predefined semantic category T = {t1, t2, ..., t K }, where K represents the number of semantic categories; Open vocabulary object detector Grounding DINO, which takes a fused image and predefined semantic categories as input and obtains detection proposals v with text as hints gd =Det gd (T,I f ), where Det gd (·,·) indicates the detection process with textual hints; The visual cue input unit is used to receive the visual cue input, select J semantic target areas in the scene image, and crop them to obtain J visual cue image detection frames b j =(x j ,y j ,w j ,h j ), where j = 1, 2, ..., J; x j ,y j Respectively represent the horizontal and vertical coordinates of the upper left corner of the detection box, w j 、h j Respectively represent the width and height of the detection box; Visual encoder, through the visual encoder to get the visual embedding V: V = En(b1, b2, ..., b J ), where En represents the visual encoder; An image encoder, which takes the fused image as input and outputs the encoding result of the fused image to the decoder; Decoder: The fused image passes through the image encoder and is input into the decoder together with the visual embedding V. The decoder outputs a detection proposal v based on the visual prompt. vp =De(V,f); where De represents the decoder, and f represents the feature map output by the image encoder; The merging unit is used to merge the detection proposals with text as the prompt and the detection proposals with vision as the prompt to obtain the target detection proposal predicted by the entire multi-prompt detection stage, expressed as: v N =v gd ∪v vp ; where N represents the number of initial detection proposals.

9. The open scene multimodal image-text zero-shot semantic segmentation system according to claim 8, characterized in that: The RGB-T open vocabulary object detection module also includes an initial detection category mapping unit, a pre-trained image-text matching model CLIP, a similarity score calculation unit, and a semantic discrimination unit: The initial detection category mapping unit is used to determine the initial detection proposals v1, v2, ..., v N The corresponding initial category ids are The pre-trained image-text matching model CLIP detects image proposals and predefined semantic categories T = {t1, t2, ..., t K } perform independent encoding to obtain the corresponding visual embedding and text embedding Recorded as n=1,2,...,N and k = 1, 2, ..., K, K is the number of text categories; The similarity score calculation unit calculates the similarity score between the visual embedding and the text embedding as follows: in represents the visual embedding of the nth detection proposal, represents the text embedding of the kth category, F nk The prediction confidence corresponding to the nth detection proposal belonging to the kth category, <·,·> represents the dot product operation; The semantic discrimination unit is specifically: For each detection proposal, select the class with the highest prediction score As predicted class label Where .index() is the index calling function, pointing to the confidence score The corresponding category k; if the predicted category With the initial detection category Matching, i.e. judging Is it true? If so, it means that the semantic understanding is consistent and no further correction is needed; If the predicted category is different from the initial detection category, additional judgment needs to be performed based on the following conditions: The confidence score corresponding to the initial detection category is retrieved based on the similarity score: Then, the following two conditions are used for judgment: Where th1 and th2 are two constant thresholds. If both conditions are met, it is jointly determined that the detection proposal should be updated to the predicted category.

10. The open scene multimodal image-text zero-shot semantic segmentation method according to any one of claims 6 to 9, characterized in that: The semantic segmentation module performs semantic segmentation on the target in the detection proposal by integrating the segmentation basic model TAP, thereby realizing zero-shot semantic segmentation of images in open scenes.

Images