A first-view scene parsing method based on text supervision

Abstract

The application discloses a first-view scene analysis method based on text supervision, and belongs to the field of image processing, in particular to the field of first-view scene analysis. The application proposes a cross-modal prompt learning module, introduces visual and text learnable prompt vectors, fine-tunes the visual language model pre-trained on third-person data, so that the visual language model can be applied to first-view images with complex target relationships. In addition, the application also proposes a representation knowledge transfer module, which distills the feature-level knowledge of the visual language pre-training model into the fine-tuned first-view encoder, improves the cross-modal correlation ability of the first-view model, and accurately segments the target in the first-view scene based on the category text. The application innovatively proposes a first-view scene analysis method based on text supervision, which generates corresponding segmentation pseudo masks by using the category text of the first-view image to alleviate the problem of pixel-level annotation scarcity.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing, in particular, the field of first-person visual scene parsing, that is, based on first-person image input for analysis, and pixel-level segmentation extraction of instances of interest in the scene. BACKGROUND

[0002] With the continuous development of the field of computer vision and the concept of "embodied intelligence", more and more researchers have begun to focus on the research of first-person visual scene segmentation algorithms. Unlike traditional third-person images, first-person images are usually collected through wearable devices, focusing on the interaction between the hands of the wearer of the wearable device and the objects in the scene. At present, in the field of computer vision, most models are trained based on third-person images, but with the development of artificial intelligence, it is increasingly necessary to endow machines with the ability to immerse themselves in the world with a "self-centered" first-person perspective. Therefore, the first-person visual scene segmentation algorithm is well worth studying, which can naturally adapt to application scenarios such as humanoid robots and intelligent wearable devices.

[0003] First-person visual scene segmentation has the following two challenges: on the one hand, fine-grained pixel-level annotation of first-person images or videos is very time-consuming and laborious, and as the scale of first-person image or video data continues to grow, there will inevitably be a problem of scarcity of pixel-level annotation, which is a problem that cannot be ignored for the training of first-person visual scene segmentation models; on the other hand, unlike third-person scenes, first-person scenes have complex inter-object relationships, and current large-scale visual language pre-training models are usually trained based on third-person data, which cannot be directly applied to first-person related downstream tasks, which also limits the segmentation performance of the model.

[0004] In the field of third-person weakly supervised visual scene segmentation, there are many excellent methods. The current mainstream method is to use class labels or their texts as supervision information, generate pixel-level segmentation pseudo labels and train the network to perform weakly supervised visual scene segmentation. The corresponding methods can be divided into two parts: two-stage methods and end-to-end methods. The two-stage method first trains the model using class labels, extracts its class activation spectrum and refines it to obtain a pseudo segmentation mask, and then uses the generated pseudo segmentation mask to train a segmentation network. For example: MCTFormer V1, MCTFormer V2, CLIMS, etc. are all two-stage methods. The end-to-end method is to generate pseudo labels and perform image segmentation at the same time. Representative methods include AFA, ToCo, etc. In the first perspective field, there are also many related works on first-person hand-object interaction understanding. In order to realize first-person hand-object interaction understanding, researchers have tried various network frameworks based on CNN, RNN and Transformer. In addition, there are many works in the first perspective field based on cross-modal pre-training, open vocabulary behavior recognition, prompt learning, continuous learning, etc. The above works provide a solid foundation for the present invention.

[0005] The method is further improved based on the text-based third-person weakly supervised semantic segmentation method CLIP-ES proposed in CVPR2023. The original CLIP-ES method is based on a frozen CLIP visual language model for third-person weakly supervised semantic segmentation, but the CLIP model pre-trained on third-person data cannot be directly applied to first-person scenes with complex inter-target relationships. To solve the above problem, the present invention first introduces a learnable visual and text prompt to fine-tune the model to learn the first-person specific hand-object relationship. In order to maintain the cross-modal correlation ability of the visual language model during fine-tuning, the present invention further introduces the idea of knowledge transfer, distilling the feature-level knowledge of a large-scale visual language model to the first-person model, thereby accurately segmenting the first-person visual target based on class labels.

[0006] The present solution has not been published on domestic and foreign publications, nor has it been publicly used or known to the public in other ways. SUMMARY

[0007] The purpose of the present invention is to solve the following technical problems:

[0008] One: existing weakly supervised segmentation methods are usually applicable to third-view scenes, and previous weakly supervised segmentation methods only rely on a single visual modality, resulting in insufficient suppression of background regions and low-quality segmentation pseudo-labels. The latest weakly supervised method uses a pre-trained visual language model such as CLIP and uses class text as supervision information, but it cannot be directly applied to first-view scenes with complex target relationships. To solve this problem, the present application designs a cross-modal prompt learning module, which uses learnable visual and text prompts to fine-tune the pre-trained visual language model, enabling it to learn complex target relationships and apply to first-view scenes. Specifically, the present application inputs learnable visual and text prompts into visual and text encoders, respectively, and introduces a cross-modal mapping layer between the visual and text prompt vectors. The learnable prompts after cross-modal mapping are also inputted as inputs, thereby enhancing the cross-modal ability of the learnable prompts.

[0009] Secondly, after fine-tuning the pre-trained visual language model, the original visual language cross-modal correlation ability of the model will degrade, resulting in the model's inability to recognize various visual semantic targets in the first-view scene based on text, leading to possible false positive regions. To solve this problem, the present application designs a representation knowledge transfer module. Specifically, the present application uses a larger-scale, frozen pre-trained visual language model as a teacher model and a fine-tuned model in the first-view scene as a student model. Based on the same visual and text input, the visual and text features extracted by the encoders of the teacher and student models are averaged and pooled, and L2 loss is applied to the pooled visual and text features, respectively, as a constraint. This way, the feature-level knowledge of the original visual language pre-trained model is distilled into the first-view model, enabling the fine-tuned model to maintain the original cross-modal correlation ability and facilitating the generation of accurate first-view pseudo-labels.

[0010] To address the problem of the scarcity of high-quality pixel-level annotations in the first-view scene, the present application uses class text corresponding to image-level classification labels to supervise the first-view model for multi-label classification and generates first-view image segmentation pseudo-masks based on class activation maps. To address the problem that large-scale pre-trained visual language models cannot handle complex target relationships in the first-view scene, the present application designs a cross-modal prompt learning module that fine-tunes existing visual language models by introducing learnable visual and text prompts to learn complex hand-object relationships in the first-view visual scene. To address the problem of degradation of the cross-modal correlation ability of the visual language model during fine-tuning, the present application designs a representation knowledge transfer module that distills the feature-level knowledge of the frozen teacher visual language model to the student first-view model, enabling the fine-tuned model to maintain the original visual language correlation ability and facilitating the generation of accurate first-view pseudo-labels. Figure 1 The specific network architecture of the present method is shown.

[0011] The present application proposes a first-view scene parsing method based on text supervision, which includes three stages:

[0012] The first stage is a prompt learning and knowledge transfer stage,

[0013] The second stage is a pseudo-label generation stage,

[0014] The third stage is a segmentation model training stage;

[0015] The first stage includes:

[0016] Step 1: The pre-trained visual and text model of CLIP ViT-B / 16 version is used as the first-view visual encoder ψ V (·) and the first-view text encoder ψ T (·), and the visual and language model in the CLIP ViT-L / 14 version is used as the teacher visual encoder ψ' V (·) and the teacher text encoder ψ' T (·);

[0017] The first-view image I is input into the teacher visual encoder ψ' V (·) and the first-view visual encoder ψ V (·), and the corresponding first-view category text T CLS is input into the teacher text encoder ψ' T (·) and the first-view text encoder ψ T (·);

[0018] Step 2: A set of learnable visual prompt vectors P V and a set of learnable text prompt vectors P T are randomly initialized as additional inputs of the first-view visual encoder ψ V (·) and the text encoder ψ T (·), respectively; learnable visual-to-text prompt mapping layer φ V→T and text-to-visual prompt mapping layer φ T→V are introduced, and cross-modal mapping is performed on the visual prompt vectors P V and the text prompt vectors P T , respectively, to obtain the visual-to-text prompt vectors P V→T and the text-to-visual prompt vectors P T→V ;

[0019] P V→T is regarded as a prompt vector in the text modality, and P T→V is regarded as a prompt vector in the visual modality;

[0020] Step 3: input the first-view image I and the first-view category text T CLS respectively into the teacher visual encoder ψ' V (·) and the teacher text encoder ψ' T (·) to extract the teacher visual feature F' V and the teacher text feature F' T , whose specific formula is shown as follows:

[0021] F' V = ψ' V (I)

[0022] F' T = ψ' T (T CLS )

[0023] input the first-view image I, the visual learnable prompt vector P V , and the mapped P T→V into the visual encoder ψ V (·) for fine-tuning; similarly, input the first-view category text T CLS , the text learnable prompt vector P T , and the mapped P V→T into the text encoder ψ T (·);

[0024] Step 4: perform global average pooling operation on the teacher visual feature F' V , the teacher text feature F' T , the first-view visual feature F V , and the first-view text feature F T to obtain the corresponding global features g' V , g' T , g V , and g T , and then set the visual knowledge distillation loss L VKD and the text knowledge distillation loss L TKD for the features of the two modalities of vision and text, respectively;

[0025] Step 5: obtain the mapped visual feature by the cross-modal mapping layer of the CLIP model from the first-view visual global feature g V , and calculate the cosine similarity with each text feature vector in the first-view text feature F T , taking the calculation result as the multi-label classification prediction value and calculating the binary cross-entropy loss with the multi-label classification true label y, denoted as L CLS ; the optimization loss L of the first stage is shown as follows:

[0026] L = L CLS + λ1L VKD + λ2L TKD

[0027] In the above formula, λ1 and λ2 are loss balance coefficients;

[0028] The second stage is:

[0029] Step 6: An additional background text set T is constructed bg , and T bg is input into the trained text encoder ψ C (·) together with all the first-view category text T LS ′ T , and the first-view image I is input into the trained visual encoder ψ V (·), the cross-modal similarity is calculated based on the visual and text features to obtain the final classification prediction value, and based on the calculated classification prediction value, the gradient weighted class activation map method is used to obtain the class activation map corresponding to each class and to refine, and finally the first-view segmentation pseudo mask is obtained;

[0030] The third stage is:

[0031] Step 7: The generated first-view segmentation pseudo mask is used as the label for training the segmentation model, and the general semantic segmentation model DeepLab V3 is trained to obtain the final deployable first-view visual scene segmentation model.

[0032] Further, the specific formula of step 2 is as follows:

[0033] P V→T = φ V→T (P V )

[0034] P T→V = φ T→V (P T )

[0035] Further, the specific formula of the prompt vector set input in step 3 is as follows:

[0036] [E′ V ; F V ] = ψ V ([E V ; Flatten(Conv(I))]) = ψ V ([P V ; P T→V ; Flatten(Conv(I))])

[0037] [E′T ; F T ] = ψ T ([E T ; Embed(T CLS )]) = ψ T ([P T ; P V→T ; Embed(T CLS )])

[0038] In the above formula, Conv(·) represents an input image convolution layer, Flatten(·) represents a flattening operation on the feature, Embed(·) represents a text word embedding operation, [; ] represents a splicing operation, E represents an input visual prompt vector set, E' represents a teacher model input visual prompt vector set, E represents an input text prompt vector set, and E' represents a teacher model input text prompt vector set. V T T

[0039] Further, the specific loss of step 4 is represented as:

[0040]

[0041] In the above formula, N represents the dimension of the global feature, and ||·||2 represents the L2 norm.

[0042] Further, the corresponding values of λ1 and λ2 in step 5 are both set to 0.02.

[0043] Beneficial effects: The first perspective scene analysis method based on text supervision proposed in the present application can generate accurate segmentation pseudo masks. In the quantitative experiment, the method of the present application is evaluated using the mean intersection over union (mIoU), and the mIoU of the segmentation pseudo mask generated by the method proposed in the present application on the mainstream first perspective segmentation dataset EgoHand reaches 53.99%. In the first perspective scene, compared with the most advanced weakly supervised segmentation method CLIMS based on the pre-trained CLIP visual language model, the mIoU is improved by 16.87%. In the field of weakly supervised segmentation, generating more accurate pseudo masks is beneficial to training a higher performance segmentation network, so the quality of the segmentation pseudo mask is an important reference for evaluating the effectiveness of the method. Figure 2 The pseudo masks generated by the method proposed in the present application for different first perspective image inputs are shown in FIG. 1, and the task requires the model to generate the corresponding pseudo masks of the left hand, right hand and objects interacting with the hands of the wearer of the wearable device. From Figure 2 As can be seen from FIG. 1, for different first perspective scenes and different targets of interest, the method proposed in the present application can generate accurate segmentation pseudo masks, which embodies the effectiveness of the method.

[0044] ​​​The class activation map can well reflect the position focused by the model on the input image, Figure 3 The class activation map visualization result of the model proposed in the application is shown in FIG. 6 (the class activation map is generated by using the Grad-CAM method), from which it can be seen that the model proposed in the application can accurately focus on the important region in the first perspective scene, i.e., the region of the hand and the interactive object, which lays an important foundation for generating an accurate first perspective scene segmentation pseudo mask. Figure 3 The class activation map visualization result of the model proposed in the application is shown in FIG. 6 (the class activation map is generated by using the Grad-CAM method), from which it can be seen that the model proposed in the application can accurately focus on the important region in the first perspective scene, i.e., the region of the hand and the interactive object, which lays an important foundation for generating an accurate first perspective scene segmentation pseudo mask. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The first perspective scene analysis network framework based on text supervision is shown in FIG. 2.

[0046] Figure 2 The first perspective segmentation pseudo mask generated by the method is shown in FIG. 4.

[0047] Figure 3 The class activation map visualization result of the model is shown in FIG. 6. DETAILED DESCRIPTION

[0048] The application is based on a large-scale visual language pre-training model, and uses the corresponding text of the class label which is more easily obtained as supervision information for first perspective image segmentation. Specifically, the application proposes a cross-modal prompt learning module, introduces visual and text learnable prompt vectors, fine-tunes the visual language model pre-trained on third-person data, so that it can be applied to first perspective images with complex target relationships; in addition, the application also proposes a representation knowledge transfer module, which distills the feature-level knowledge of the visual language pre-training model into the fine-tuned first perspective encoder, improves the cross-modal correlation ability of the first perspective model, and thus accurately segments the target in the first perspective scene based on the class text. The application innovatively proposes a first perspective scene analysis method based on text supervision, which generates a corresponding segmentation pseudo mask for the class text of the first perspective image to alleviate the problem of pixel-level label scarcity. The application innovatively proposes a cross-modal prompt learning module and a representation knowledge transfer module. In order to solve the problem that the current large-scale pre-training visual language model cannot handle the relationship between complex targets in the first perspective scene, the cross-modal prompt learning module of the application introduces a series of learnable visual and text prompt vectors to make the model applicable to the first perspective scene, and introduces a cross-modal mapping layer between different modal learnable prompt vectors to enhance the cross-modal understanding ability of the model; in order to solve the problem that after fine-tuning, the original visual language cross-modal correlation ability of the model degrades, and thus cannot recognize various semantic targets based on the given text, the representation knowledge transfer module proposed in the application distills the feature-level knowledge of the teacher model to the first perspective model to maintain its cross-modal correlation ability, thereby generating an accurate first perspective segmentation pseudo mask.

[0049] Figure 1 In the present application, the method is divided into three stages:

[0050] The first stage is the prompt learning and knowledge transfer stage,

[0051] The second stage is the pseudo-label generation stage,

[0052] The third stage is the segmentation model training stage;

[0053] The first stage is the core part of the present application. Figure 1 In the present application, in addition to the symbols already provided in the text description, φ V→T represents the cross-modal learnable prompt vector mapping layer from vision to text; φ T→V represents the cross-modal learnable prompt vector mapping layer from text to vision; E' V represents the output vector corresponding to the position of the input visual prompt vector of the visual encoder; E' T represents the output vector corresponding to the position of the input text prompt vector of the text encoder; g' V and g V respectively represent the global visual features output by the teacher visual model and the first perspective visual model; g' T and g T respectively represent the global text features output by the teacher text model and the first perspective text model; L VKD and L TKD respectively represent the knowledge distillation loss of the visual and text modalities; L CLS represents the first perspective multi-label classification loss based on the cross-modal correlation of visual language; L SEG represents the general semantic segmentation loss (such as cross-entropy loss). Next, the present application will be described in detail in the order of steps:

[0054] Step 1: The present application selects the visual and text models in the widely used pre-trained visual language model CLIP (ViT-B / 16 version) as the first perspective visual encoder and the text encoder, which are represented as ψ V (·) and ψ T (·); for the teacher visual encoder ψ' V (·) and the teacher text encoder ψ' T (·), the present application selects the visual and language models in the larger scale CLIP (ViT-L / 14 version) to further improve the cross-modal correlation ability; the present application first inputs the first perspective image I into the teacher visual encoder ψ' V (·) and the first perspective visual encoder ψ V (·), and inputs the corresponding first perspective class text T CLS into the teacher text encoder ψ' T(·) with the first-view text encoder ψ T (·) ;

[0055] Step 2: To solve the problem that the CLIP model cannot directly process the relationship between complex targets in the first view, the present application proposes a cross-modal prompt learning module. Specifically, the present application first randomly initializes a set of learnable visual prompt vectors P V and a set of learnable text prompt vectors P T , which are respectively used as additional inputs of the first-view visual encoder ψ V (·) and the text encoder ψ T (·) ; secondly, in order to enhance the cross-modal understanding ability of the learnable prompt vectors, the present application further introduces a learnable visual-to-text prompt mapping layer φ V→T and a text-to-visual prompt mapping layer φ T→V , and performs cross-modal mapping on the visual prompt vectors P V and the text prompt vectors P T , respectively, to obtain the visual-to-text prompt vectors P V→T and the text-to-visual prompt vectors P T→V , whose specific formulas are as follows:

[0056] P V→T = φ V→T (P V ) (0.1)

[0057] P T→V = φ T→V (P T ) (0.2)

[0058] After cross-modal mapping, P V→T is regarded as a prompt vector in the text modality, and P T→V is regarded as a prompt vector in the visual modality.

[0059] Step 3: The first-view image I and the first-view category text T CLS are respectively input into the teacher visual encoder ψ' V (·) and the teacher text encoder ψ' T (·), so as to extract the teacher visual feature F' V and the teacher text feature F' T , whose specific formulas are as follows:

[0060] F' V = ψ' V (I) (0.3)

[0061] F' T = ψ' T (T CLS) (0.4)

[0062] In order to make the visual and text encoder learn the complex relationship between targets in the first perspective scene, the present application inputs the first perspective image I, the visual learnable prompt vector P V and the mapped P T→V into the visual encoder ψ V (·) for fine-tuning; similarly, the present application inputs the first perspective category text T CLS , the text learnable prompt vector P T and the mapped P V→T into the text encoder ψ T (·), and the specific formula is as follows:

[0063] [E′ V ; F V ] = ψ V ([E V ; Flatten(Conv(I))) = ψ V ([P V ; P T→V ; Flatten(Conv(I))) (0.5)

[0064] [E′ T ; F T ] = ψ T ([E T ; Embed(T CLS )) = ψ T ([P T ; P V→T ; Embed(T CLS )) (0.6)

[0065] In the above formula, Conv(·) represents the input image convolution layer, Flatten(·) represents the flattening operation of the feature, Embed(·) represents the text word embedding operation, [;] represents the splicing operation, E V represents the input visual prompt vector set, and E T represents the input text prompt vector set.

[0066] Step 4: Fine-tuning the visual and language encoders ψ V (·) and ψ T (·) will cause the original visual language cross-modal correlation ability to degrade, in order to solve this problem, the present application designs a representation knowledge transfer module. Specifically, the present application respectively fine-tunes the teacher visual feature F′ V , the teacher text feature F′ T , the first perspective visual feature F V , and the first perspective text feature F TA global average pooling operation is performed to obtain a corresponding global feature g' V , g' T , g V , g T Then, for the features of the two modalities of vision and text, a visual knowledge distillation loss L VKD and a text knowledge distillation loss L TKD are respectively designed, and the specific formulas are as follows:

[0067]

[0068] In the above formula, N represents the dimension of the global feature, and ||·||2 represents the L2 norm.

[0069] Step 5: At the end of the prompt learning and knowledge transfer stage, a first perspective multi-label classification based on visual language cross-modal association also needs to be performed. Specifically, the first perspective visual global feature g V is obtained through the cross-modal mapping layer of the CLIP model, and the mapped visual feature g is obtained. T The cosine similarity is calculated between each text feature vector in the first perspective text feature F and the calculated result is regarded as a multi-label classification prediction value y CLS .

[0070] The binary cross-entropy loss L CLS is calculated between the multi-label classification true label y and the multi-label classification prediction value y VKD . TKD (0.9)

[0071] In the above formula, λ1 and λ2 are loss balancing coefficients, and their corresponding values are both set to 0.02.

[0072] Step 6: After the first stage, there are a subsequent pseudo-label generation stage and a segmentation model training stage. In the pseudo-label generation stage, inspired by previous work, in order to further suppress the error segmentation of the background region, the present application additionally constructs a background text set T bg , and inputs T bg and all first perspective category texts T' CLS into the trained text encoder ψ T (·) together, and inputs the first perspective image I into the trained visual encoder ψ V(·), based on the visual and text features, the cross-modal similarity is calculated to obtain a final classification prediction value, and based on the calculated classification prediction value, a gradient weighted class activation mapping (Grad-CAM) method is used to obtain a class activation spectrum corresponding to each class and to refine, and finally a first visual angle segmentation pseudo mask is obtained. In the segmentation model training stage, the generated first visual angle segmentation pseudo mask is used as a label for training the segmentation model, and a general semantic segmentation model DeepLab V3 is trained to obtain a final deployable first visual angle visual scene segmentation model.

Claims

1. A first-view scene parsing method based on text supervision, comprising three stages: Stage 1: prompt learning and knowledge transfer stage, Stage 2: pseudo-label generation stage, Stage 3: segmentation model training stage; Stage 1 includes: Step 1: The visual and text model of the pre-trained visual language model CLIP ViT-B / 16 version is used as the first perspective visual encoder ψ V (·) and the first perspective text encoder ψ T (·), using the visual and language model in the CLIP ViT-L / 14 version as the teacher visual encoder ψ' V (·) and the teacher text encoder ψ' T (·); inputting the first-view image I simultaneously into the teacher visual encoder ψ' V (·) with the first-view visual encoder ψ V (·), and inputting the corresponding first-view class text T CLS into the teacher text encoder ψ' T (·) with the first-view text encoder ψ T (·); Step 2: Randomly initialize a set of learnable visual prompt vectors P V and a set of learnable textual prompt vectors P T as extra inputs to the first-view visual encoder ψ V (·) and the textual encoder ψ T (·), respectively; introduce a learnable visual-to-textual prompt mapping layer φ V→T and a textual-to-visual prompt mapping layer φ T→V and perform cross-modal mapping on the visual prompt vectors P V and the textual prompt vectors P T to obtain the visual-to-textual prompt vectors P V→T and the textual-to-visual prompt vectors P T→V , respectively. P obtained through the cross-modal mapping layer V→T Consider it as the prompt vector of text mode. Similarly, P T→V cue vector to be considered as the visual modality; Step 3: input the first-view image I and the first-view category text T CLS respectively into the teacher visual encoder ψ' V (·) and the teacher text encoder ψ' T (·), to extract the teacher visual feature F V ' and the teacher text feature F T ', the specific formula is as follows: F V ′= ψ′ V (I) F T ′= ψ′ T (T CLS ) The first-view image I, the visual learnable prompt vector P V and the mapped P T→V are jointly input into the visual encoder ψ V ; similarly, the first-view category text T CLS , the text learnable prompt vector P T and the mapped P V→T are jointly input into the text encoder ψ T ; Step 4: The teacher visual feature F V ′, the teacher text feature F T ′, the first perspective visual feature F V , the first perspective text feature F T , and the second perspective visual feature F V , the second perspective text feature F T , and the global feature g V , g T , and g VKD , and the text knowledge distillation loss L TKD ; Step 5: obtain the visual global feature g V of the first view by the cross-modal mapping layer of the CLIP model And each of the text feature vectors in the first view text feature F T Calculate the cosine similarity, and take the calculation result as the multi-label classification prediction value And calculate the binary cross-entropy loss with the multi-label classification true label y, denoted as L CLS The optimization loss L of the first stage is as follows: L = L CLS + λ1L VKD + λ2L TKD In the formula, λ1 and λ2 are loss balancing coefficients; Stage 2 is: Step 6: An additional background text set T is constructed bg , and T bg is input into the trained text encoder ψ C (·) together with all the first-view category text T LS ′ T , and the first-view image I is input into the trained visual encoder ψ V (·), the cross-modal similarity is calculated based on the visual and text features to obtain the final classification prediction value, and based on the calculated classification prediction value, the gradient weighted class activation mapping method is used to obtain the class activation spectrum corresponding to each class and to refine, and finally the first-view segmentation pseudo mask is obtained; Stage 3 is: Step 7: The generated first-view segmentation pseudo-mask is used as the label for training the segmentation model, and the general semantic segmentation model DeepLab V3 is trained to obtain the final deployable first-view visual scene segmentation model.

2. The text supervision based first-view scene parsing method of claim 1, wherein, The specific formula of step 2 is as follows: P V→T = φ V→T (P V ) P T→V = φ T→V (P T ). 3.The first-view scene parsing method based on text supervision of claim 1, wherein, The specific formula of the input prompt vector set in step 3 is as follows: [E′ V ; F V ] = ψ V ([E V ; Flatten(Conv(I))]) = ψ V ([P V ; P T→V ; Flatten(Conv(I))]) [E′ T ; F T ] = ψ T ([E T ; Embed(T CLS )]) = ψ T ([P T ; P V→T ; Embed(T CLS ))) In the above formula, Conv(·) represents the input image convolution layer, Flatten(·) represents the flattening operation of the feature, Embed(·) represents the text word embedding operation, [;] represents the splicing operation, and E V represents the set of input visual prompt vectors, E′ represents the set of visual prompt vectors input by the teacher model, and E T Represents the set of input text prompt vectors, E T ′ represents the set of text prompt vectors input to the teacher model. 4.The first-view scene parsing method based on text supervision according to claim 1, wherein, The specific loss of step 4 is represented as: In the formula, N represents the dimension of the global feature, and ||·||2 represents the L2 norm.

5. The text supervision based first-person scene parsing method of claim 1, wherein, In step 5, λ1 and λ2 are both set to 0.02.

Images