First visual angle scene analysis method based on text supervision

Abstract

The invention discloses a first visual angle scene analysis method based on text supervision, and belongs to the field of image processing, in particular to the field of first visual angle visual scene analysis. According to the invention, a cross-modal prompt learning module is provided, visual and text learnable prompt vectors are introduced, and a visual language model pre-trained on third-person data is finely adjusted, so that the visual language model can be applied to a first visual angle image with a complex target relationship; in addition, the invention also provides a representation knowledge migration module, the feature level knowledge of the visual language pre-training model is distilled to the first visual angle encoder after fine tuning, the cross-modal association capability of the first visual angle model is improved, and thus the target in the first visual angle scene is accurately segmented based on the category text. The invention innovatively provides a first view scene analysis method based on text supervision. The corresponding segmentation pseudo mask is generated by using the category text of the first view image to relieve the problem of pixel-level annotation scarcity.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, in particular to the field of first-person perspective visual scene analysis, that is, analysis is performed based on first-person perspective image input, and pixel-level segmentation and extraction of focused instances in the scene are performed. Background Art

[0002] With the continuous development of the field of computer vision and the introduction of the concept of "embodied intelligence", more and more researchers have begun to pay attention to the research of first-person visual scene segmentation related algorithms. Unlike traditional third-person images, first-person images are usually collected through wearable devices, focusing on the interaction between the hands of the wearer of the wearable device and the objects in the scene. At present, in the field of computer vision, most models are trained based on third-person images, but with the development of artificial intelligence, there is an increasing need to give machines the ability to be "self-centered" and to understand the world immersively and fine-grained from a first-person perspective. Therefore, the first-person visual scene segmentation algorithm is very worthy of study, and it can be naturally adapted to application scenarios such as humanoid robots and smart wearable devices.

[0003] There are two main challenges in first-person visual scene segmentation: on the one hand, fine-grained pixel-level annotation of first-person images or videos is very time-consuming and labor-intensive. As the scale of first-person image or video data continues to increase, the problem of scarce pixel-level annotation is bound to arise, which is an issue that cannot be ignored for the training of first-person visual scene segmentation models; on the other hand, unlike third-person scenes, first-person scenes have complex relationships between objects, and current large-scale visual language pre-training models are usually trained based on third-person data, which cannot be directly applied to first-person related downstream tasks, which also limits the segmentation performance of the model.

[0004] In the third-person weakly supervised visual scene segmentation, many excellent methods have been proposed. The current mainstream method is to use category labels or their text as supervision information, generate pixel-level segmentation pseudo labels and train the network to perform weakly supervised visual scene segmentation, and the corresponding methods can be divided into two parts: two-stage method and end-to-end method. The two-stage method is to first train the model with category labels, extract its class activation spectrum and refine it to obtain a pseudo-segmentation mask, and then use the generated pseudo-segmentation mask to train a segmentation network. For example, methods such as MCTFormer V1, MCTFormer V2, and CLIMS are all two-stage methods; the end-to-end method is that the model generates pseudo labels and performs image segmentation at the same time. Representative methods include: AFA, ToCo and other methods. In the field of first-person perspective, there are also many related works on first-person perspective hand-object interaction understanding. In order to achieve first-person perspective hand-object interaction understanding, researchers have tried various network frameworks based on CNN, RNN and Transformer. In addition, in the field of first-person perspective, there are many works based on cross-modal pre-training, open vocabulary behavior recognition, prompt learning, continuous learning, etc. The above work provides a solid foundation for the present invention.

[0005] This method is further improved on the text-based third-person weakly supervised semantic segmentation method CLIP-ES proposed in CVPR2023. The original CLIP-ES method performs third-person weakly supervised semantic segmentation based on the frozen CLIP visual language model, but the CLIP model pre-trained on third-person data cannot be directly applied to first-person scenes with complex relationships between objects. In order to solve the above problems, the present invention first introduces learnable visual and textual cues to fine-tune the model to learn the hand-object relationship unique to the first perspective; in order to maintain the cross-modal association ability of the visual language model during the fine-tuning process, the present invention further introduces the idea of ​​knowledge transfer, distilling the feature-level knowledge of the large-scale visual language model to the first-person model, so as to accurately segment the first-person visual targets based on the text of the category label.

[0006] This plan has not been published in domestic or foreign publications, has not been publicly used at home or abroad, or has not been known to the public in other ways. Summary of the invention

[0007] The purpose of the present invention is to solve the following technical problems:

[0008] First, existing weakly supervised segmentation methods are usually applicable to third-person perspective scenes. Early weakly supervised segmentation methods only rely on a single visual modality, resulting in insufficient suppression of background areas and low-quality segmentation pseudo-labels. The latest weakly supervised methods utilize pre-trained visual language models such as CLIP and use category text as supervisory information, but they cannot be directly applied to first-person perspective scenes with complex object relationships. In order to solve this problem, the present invention designs a cross-modal prompt learning module, which uses learnable visual and text prompt vectors to fine-tune the pre-trained visual language model so that it can learn complex object relationships and apply it to first-person perspective scenes. Specifically, the present invention inputs learnable visual prompts and text prompts into visual and text encoders respectively, and introduces a cross-modal mapping layer between the visual and text prompt vectors, and uses the learnable prompts that have been cross-modally mapped as inputs, thereby enhancing the cross-modal capabilities of the learnable prompts.

[0009] Second: After fine-tuning the pre-trained visual language model, the original visual language cross-modal association ability of the model will degenerate, resulting in the model being unable to recognize various visual semantic targets in the first-person scene based on text, resulting in possible false positive areas. In order to solve this problem, the present invention designs a representation knowledge transfer module. Specifically, the present invention uses a larger-scale, frozen pre-trained visual language model as a teacher model, and a model that has been fine-tuned in the first-person scene as a student model. Based on the same visual and text inputs, the visual and text features extracted by the encoders of the teacher and student models are average-pooled, and L2 loss is applied to the pooled visual features and text features as constraints, respectively, so as to distill the feature-level knowledge of the original visual language pre-trained model into the first-person model, so that the fine-tuned model can maintain the original cross-modal association ability, which is conducive to generating accurate first-person pseudo-labels.

[0010] To address the problem of scarcity of high-quality pixel-level annotations in the first-person perspective, the present invention uses category texts corresponding to image-level classification labels to supervise the first-person perspective model for multi-label classification, and generates first-person perspective image segmentation pseudo masks based on class activation spectra; to address the problem that current large-scale pre-trained visual language models are unable to handle complex target relationships in first-person perspective scenes, the present invention designs a cross-modal prompt learning module, which fine-tunes the existing visual language model by introducing visual and textual learnable prompts to learn the complex hand-object relationships in first-person perspective visual scenes; to address the problem of degradation of the cross-modal association ability of the visual language model during the fine-tuning process, the present invention designs a module for representing knowledge transfer, which distills the feature-level knowledge of the frozen teacher visual language model to the student first-person perspective model, so that the fine-tuned model can maintain the original visual language association ability, which is conducive to generating accurate first-person perspective pseudo labels. Figure 1 The specific network architecture of this method is demonstrated.

[0011] The present invention proposes a first-person scene parsing method based on text supervision, which includes three stages:

[0012] The first stage: prompt learning and knowledge transfer stage,

[0013] The second stage: pseudo label generation stage,

[0014] The third stage: segmentation model training stage;

[0015] Phase 1 includes:

[0016] Step 1: The pre-trained visual language model CLIP ViT-B / 16 version of the visual and text model is used as the first-view visual encoder ψ V (·) and the first-person text encoder ψ T (·), using the vision and language model in CLIP ViT-L / 14 version as the teacher visual encoder ψ′ V (·) and the teacher text encoder ψ′ T (·);

[0017] The first-view image I is simultaneously input into the teacher visual encoder ψ′ V (·) and the first-view visual encoder ψ V (·), and the corresponding first-person category text T CLS Input teacher text encoder ψ′ T (·) and the first-person text encoder ψ T (·);

[0018] Step 2: Randomly initialize a set of learnable visual cue vectors P V and a set of learnable textual cue vectors P T , respectively as the first-view visual encoder ψ V (·) and the text encoder ψ T (·) as an additional input; introducing a learnable visual-to-textual cue mapping layer φ V→T and a text-to-visual cue mapping layer φ T→V , and respectively for the visual cue vector P V and text hint vector P T Perform cross-modal mapping to obtain the visual to text prompt vector P V→T The text-to-visual hint vector P T→V ;

[0019] P obtained after the cross-modal mapping layer V→T Consider it as the prompt vector of text mode. Similarly, P T→V The cue vector to be considered as the visual modality;

[0020] Step 3: Separately map the first-view image I and the first-view category text T CLS Input the teacher visual encoder ψ′ V (·) and the teacher text encoder ψ′ T (·), extract the teacher visual feature F′ V and teacher text feature F′ T , the specific formula is as follows:

[0021] F′ V =ψ′ V (I)

[0022] F′ T =ψ′ T (T CLS )

[0023] The first-person image I and the visually learnable cue vector P V And the mapped P T→V Common input visual encoder ψ V (·) is fine-tuned; similarly, the first-person category text T CLS , text learnable hint vector P T And the mapped P V→T Common input text encoder ψ T (·);

[0024] Step 4: For the teacher visual features F′ V , Teacher text features F′ T , first-person visual features F V , first-person perspective text features F T Perform a global average pooling operation to obtain the corresponding global feature g′ V , g′ T , g V , g T Then, for the features of the two modalities of vision and text, we set the visual knowledge distillation loss L VKD and text knowledge distillation loss L TKD ;

[0025] Step 5: Transform the first-person perspective global feature g V Through the cross-modal mapping layer of the CLIP model, the mapped visual features are obtained and the first-person perspective text feature F T Calculate the cosine similarity for each text feature vector in and regard the calculation result as the multi-label classification prediction value And calculate the binary cross entropy loss with the true label y of the multi-label classification, denoted as L CLS ; The optimization loss L in the first stage is as follows:

[0026] L=L CLS +λ1L VKD +λ2L TKD

[0027] In the above formula, λ1 and λ2 are loss balance coefficients;

[0028] The second stage is:

[0029] Step 6: Additional background text set T is constructed bg , and T bg With all first-person perspective category text T C ' LS Jointly input the trained text encoder ψ T (·), and input the first-view image I into the trained visual encoder ψ V (·) Based on the visual and text features, the cross-modal similarity is calculated to obtain the final classification prediction value. Based on the calculated classification prediction value, the gradient-weighted class activation spectrum mapping method is used to obtain the class activation spectrum corresponding to each class and refine it, finally obtaining the first-person segmentation pseudo mask;

[0030] The third stage is:

[0031] Step 7: Use the generated first-person segmentation pseudo-mask as the label for training the segmentation model, and train the general semantic segmentation model DeepLab V3 to obtain the final deployable first-person visual scene segmentation model.

[0032] Furthermore, the specific formula of step 2 is as follows:

[0033] P V→T =φ V→T (P V )

[0034] P T→V =φ T→V (P T ).

[0035] Furthermore, the specific formula of the prompt vector set input in step 3 is as follows:

[0036] [E′ V ; F V ]=ψ V ([E V ;Flatten(Conv(I))])=ψ V ([P V ;P T→V ; Flatten(Conv(I))])

[0037] [E′T ; F T ]=ψ T ([E T ;Embed(T CLS )])=ψ T ([P T ;P V→T ;Embed(T CLS )])

[0038] In the above formula, Conv(·) represents the input image convolution layer, Flatten(·) represents the flattening operation of the feature, Embed(·) represents the text word embedding operation, [;] represents the concatenation operation, and E V represents the set of input visual prompt vectors, E′ represents the set of visual prompt vectors input by the teacher model, and E T Represents the set of input text prompt vectors, E′ T Represents the set of text prompt vectors input to the teacher model.

[0039] Furthermore, the specific loss of step 4 is expressed as:

[0040]

[0041] In the above formula, N represents the dimension of the global feature, and ||·||2 represents the L2 norm.

[0042] Furthermore, in step 5, the corresponding values ​​of λ1 and λ2 are both set to 0.02.

[0043] Beneficial effect: The first-person scene parsing method based on text supervision proposed in the present invention can generate accurate segmentation pseudo-masks. In the quantitative experiment, the mean intersection over union (mIoU) was used to evaluate the method of the present invention. The mIoU of the segmentation pseudo-mask generated by the method proposed in the present invention on the mainstream first-person segmentation dataset EgoHand reached 53.99%. In the first-person scene, it is 16.87% higher than the most advanced weakly supervised segmentation method CLIMS based on the pre-trained CLIP visual language model. In the field of weakly supervised segmentation, generating more accurate pseudo-masks is conducive to training higher-performance segmentation networks. Therefore, the quality of the segmentation pseudo-mask is an important reference for evaluating the effectiveness of the method. Figure 2 The pseudo masks generated by the proposed method for different first-person perspective image inputs are shown in the figure. The task requires the model to generate corresponding pseudo masks for the left hand, right hand, and objects interacting with the hands of the wearer of the wearable device. Figure 2 It can be seen from the figure that for different first-person perspective scenes and different targets of interest, the method proposed in the present invention can generate accurate segmentation pseudo masks, which reflects the effectiveness of the method of the present invention.

[0044] The class activation spectrum can well reflect the location where the model pays attention to the input image. Figure 3 The class activation spectrum visualization results of the model proposed in this invention are shown in Figure 2 (the class activation spectrum is generated using the Grad-CAM method). Figure 3 It can be seen that the model proposed in the present invention can accurately focus on the important areas in the first-person perspective scene, namely the areas of the hands and interactive objects, which lays an important foundation for generating accurate first-person perspective scene segmentation pseudo-masks. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of the first-person scene parsing network framework based on text supervision.

[0046] Figure 2 This is the first-person perspective segmentation pseudo mask effect image generated by this method.

[0047] Figure 3 Schematic diagram of the model's class activation spectrum visualization results. DETAILED DESCRIPTION

[0048] The present invention is based on a large-scale visual language pre-training model, and uses the corresponding text of the more easily available category labels as supervision information to perform first-person image segmentation. Specifically, the present invention proposes a cross-modal cue learning module, introduces visual and textual learnable cue vectors, and fine-tunes the visual language model pre-trained on third-person data so that it can be applied to first-person images with complex target relationships; in addition, the present invention also proposes a representation knowledge transfer module, which distills the feature-level knowledge of the visual language pre-training model into the fine-tuned first-person encoder to improve the cross-modal association ability of the first-person model, thereby accurately segmenting the targets in the first-person scene based on the category text. The present invention innovatively proposes a first-person scene parsing method based on text supervision, which uses the category text of the first-person image to generate the corresponding segmentation pseudo-mask to alleviate the problem of scarce pixel-level annotations. The present invention innovatively proposes a cross-modal cue learning module and a representation knowledge transfer module. In order to solve the problem that the current large-scale pre-trained visual language model cannot handle the complex relationship between objects in the first-person perspective scene, the cross-modal cue learning module of the present invention introduces a series of learnable visual and text cue vectors to enable the model to be applied to the first-person perspective scene, and introduces a cross-modal mapping layer between the learnable cue vectors of different modalities to enhance the model's cross-modal understanding ability; in order to solve the problem that after fine-tuning, the original visual language cross-modal association ability of the model degenerates, resulting in the inability to recognize various semantic objects based on a given text, the representation knowledge transfer module proposed in the present invention distills the feature-level knowledge of the teacher model into the first-person perspective model to maintain its cross-modal association ability, thereby generating accurate first-person perspective segmentation pseudo-masks.

[0049] Figure 1 In the present invention, the method is divided into three stages:

[0050] The first stage: prompt learning and knowledge transfer stage,

[0051] The second stage: pseudo label generation stage,

[0052] The third stage: segmentation model training stage;

[0053] The first stage is the core part of the present invention. Figure 1 In addition to the symbols that already have textual descriptions, V→T represents the cross-modal learnable cue vector mapping layer from vision to text; φ T→V Represents the cross-modal learnable cue vector mapping layer from text to vision; E′ V The output vector representing the position of the visual cue vector corresponding to the input visual encoder; E′ T The output vector representing the position of the text prompt vector corresponding to the input text encoder; g′ V With g V Represent the global visual features output by the teacher visual model and the first-person visual model respectively; g′ T With g T Represent the global text features output by the teacher text model and the first-person perspective text model respectively; L VKD With L TKD Represent the knowledge distillation loss of visual modality and textual modality respectively; L CLS represents the first-view multi-label classification loss based on the cross-modal association of vision and language; L SEG Represents a general semantic segmentation loss (such as cross entropy loss). Next, the present invention will be described in detail in a step-by-step manner:

[0054] Step 1: The present invention selects the visual and text models in the widely used pre-trained visual language model CLIP (ViT-B / 16 version) as the first-view visual encoder and text encoder, which are represented as ψ V (·) and ψ T (·); for the teacher visual encoder ψ′ V (·) and the teacher text encoder ψ′ T (·), the present invention uses the larger-scale visual and language model in CLIP (ViT-L / 14 version) to further improve the cross-modal association ability; the present invention first inputs the first-view image I into the teacher visual encoder ψ′ V (·) and the first-view visual encoder ψ V (·), and the corresponding first-person category text T CLS Input teacher text encoder ψ′ T(·) and the first-person text encoder ψ T (·);

[0055] Step 2: Aiming at the problem that the CLIP model cannot directly handle the relationship between complex objects in the first-person perspective, the present invention proposes a cross-modal cue learning module. Specifically, the present invention first randomly initializes a set of learnable visual cue vectors P V and a set of learnable textual cue vectors P T , respectively as the first-view visual encoder ψ V (·) and the text encoder ψ T (·) as an additional input; secondly, in order to enhance the cross-modal understanding ability of the learnable prompt vector, the present invention introduces a learnable visual-to-text prompt mapping layer φ V→T and a text-to-visual cue mapping layer φ T→V , and respectively for the visual cue vector P V and text hint vector P T Perform cross-modal mapping to obtain the visual to text prompt vector P V→T The text-to-visual hint vector P T→V , the specific formula is as follows:

[0056] P V→T =φ V→T (P V ) (0.1)

[0057] P T→V =φ T→V (P T ) (0.2)

[0058] P obtained after the cross-modal mapping layer V→T Consider it as the prompt vector of text mode. Similarly, P T→V will be considered as the cue vector for the visual modality.

[0059] Step 3: Separately map the first-view image I and the first-view category text T CLS Input the teacher visual encoder ψ′ V (·) and the teacher text encoder ψ′ T (·) to extract the teacher visual feature F′ V and teacher text feature F′ T , the specific formula is as follows:

[0060] F′ V =ψ′ V (I) (0.3)

[0061] F′ T =ψ′ T (T CLS) (0.4)

[0062] In order to enable the visual and text encoder to learn the complex relationship between objects in the first-person scene, the present invention transforms the first-person image I, the visual learnable cue vector P V And the mapped P T→V Common input visual encoder ψ V (·) is fine-tuned; similarly, the present invention converts the first-view category text T CLS , text learnable hint vector P T And the mapped P V→T Common input text encoder ψ T (·), the specific formula is as follows:

[0063] [E′ V ; F V ]=ψ V ([E V ;Flatten(Conv(I))])=ψ V ([P V ;P T→V ; Flatten(Conv(I))]) (0.5)

[0064] [E′ T ; F T ]=ψ T ([E T ;Embed(T CLS )])=ψ T ([P T ;P V→T ;Embed(T CLS )]) (0.6)

[0065] In the above formula, Conv(·) represents the input image convolution layer, Flatten(·) represents the flattening operation of the feature, Embed(·) represents the text word embedding operation, [;] represents the concatenation operation, and E V represents the set of input visual cue vectors, E T Represents a set of input text hint vectors.

[0066] Step 4: Align the visual and language encoder ψ V (·),ψ T (·) will lead to the degradation of its original visual language cross-modal association ability. In order to solve this problem, the present invention designs a representation knowledge transfer module. Specifically, the present invention respectively adjusts the teacher visual feature F′ V , Teacher text features F′ T , first-person visual features F V , first-person perspective text features F TPerform a global average pooling operation to obtain the corresponding global feature g′ V , g′ T , g V , g T Then, we design the visual knowledge distillation loss L for the features of the two modalities, visual and text. VKD and text knowledge distillation loss L TKD , the specific formula is as follows:

[0067]

[0068] In the above formula, N represents the dimension of the global feature, and ||·||2 represents the L2 norm.

[0069] Step 5: At the end of the prompt learning and knowledge transfer stage, first-person perspective multi-label classification based on cross-modal association of visual language is also required. Specifically, the present invention converts the first-person perspective visual global feature g V Through the cross-modal mapping layer of the CLIP model, the mapped visual features are obtained and the first-person perspective text feature F T Calculate the cosine similarity for each text feature vector in and regard the calculation result as the multi-label classification prediction value And calculate the binary cross entropy loss with the true label y of the multi-label classification, denoted as L CLS The first stage - the model optimization loss L of the prompt learning and knowledge transfer stage is as follows:

[0070] L=L CLS +λ1L VKD +λ2L TKD (0.9)

[0071] In the above formula, λ1 and λ2 are loss balance coefficients, and their corresponding values ​​are both set to 0.02.

[0072] Step 6: After the first stage, there are subsequent pseudo-label generation stages and segmentation model training stages. In the pseudo-label generation stage, inspired by previous work, in order to further suppress the erroneous segmentation of the background area, the present invention additionally constructs a background text set T bg , and T bg With all first-person category texts T′ CLS Jointly input the trained text encoder ψ T (·), and input the first-view image I into the trained visual encoder ψ V(·), based on the visual and text features, the cross-modal similarity is calculated to obtain the final classification prediction value, and based on the calculated classification prediction value, the gradient-weighted class activation spectrum mapping (Grad-CAM) method is used to obtain the class activation spectrum corresponding to each class and refine it, and finally obtain the first-person segmentation pseudo-mask. In the segmentation model training stage, the present invention uses the generated first-person segmentation pseudo-mask as the label of the training segmentation model, and trains the general semantic segmentation model DeepLab V3 to obtain the final deployable first-person visual scene segmentation model.

Claims

1. A first-person scene parsing method based on text supervision, which includes three stages: The first stage: prompt learning and knowledge transfer stage, The second stage: pseudo label generation stage, The third stage: segmentation model training stage; Phase 1 includes: Step 1: The pre-trained visual language model CLIP ViT-B / 16 version of the visual and text model is used as the first-view visual encoder ψ V (·) and the first-person text encoder ψ T (·), using the vision and language model in CLIP ViT-L / 14 version as the teacher visual encoder ψ′ V (·) and the teacher text encoder ψ′ T (·); The first-view image I is simultaneously input into the teacher visual encoder ψ′ V (·) and the first-view visual encoder ψ V (·), and the corresponding first-person category text T CLS Input teacher text encoder ψ′ T (·) and the first-person text encoder ψ T (·); Step 2: Randomly initialize a set of learnable visual cue vectors P V and a set of learnable textual cue vectors P T , respectively as the first-view visual encoder ψ V (·) and the text encoder ψ T (·) as an additional input; introducing a learnable visual-to-textual cue mapping layer φ V→T and a text-to-visual cue mapping layer φ T→V , and respectively for the visual cue vector P V and text hint vector P T Perform cross-modal mapping to obtain the visual to text prompt vector P V→T The text-to-visual hint vector P T→V ; P obtained after the cross-modal mapping layer V→T Consider it as the prompt vector of text mode. Similarly, P T→V The cue vector to be considered as the visual modality; Step 3: Separately map the first-view image I and the first-view category text T CLS Input the teacher visual encoder ψ′ V (·) and the teacher text encoder ψ′ T (·), extract the teacher visual feature F V ′ and teacher text feature F T ′, the specific formula is as follows: F V ′=ψ′ V (I) F T ′=ψ′ T (T CLS ) The first-person image I and the visually learnable cue vector P V And the mapped P T→V Common input visual encoder ψ V (·) is fine-tuned; similarly, the first-person category text T CLS , text learnable hint vector P T And the mapped P V→T Common input text encoder ψ T (·); Step 4: For each teacher visual feature F V ′, Teacher text features F T ′, first-person visual feature F V , first-person perspective text features F T Perform a global average pooling operation to obtain the corresponding global feature g′ V , g′ T , g V , g T Then, for the features of the two modalities of vision and text, we set the visual knowledge distillation loss L VKD and text knowledge distillation loss L TKD ; Step 5: Transform the first-person perspective global feature g V Through the cross-modal mapping layer of the CLIP model, the mapped visual features are obtained and the first-person perspective text feature F T Calculate the cosine similarity for each text feature vector in and regard the calculation result as the multi-label classification prediction value And calculate the binary cross entropy loss with the true label y of the multi-label classification, denoted as L CLS ; The optimization loss L in the first stage is as follows: L=L CLS +λ1L VKD +λ2L TKD In the above formula, λ1 and λ2 are loss balance coefficients; The second stage is: Step 6: Additional background text set T is constructed bg , and T bg With all first-person perspective category text T C ' LS Jointly input the trained text encoder ψ T (·), and input the first-view image I into the trained visual encoder ψ V (·) Based on the visual and text features, the cross-modal similarity is calculated to obtain the final classification prediction value. Based on the calculated classification prediction value, the gradient-weighted class activation spectrum mapping method is used to obtain the class activation spectrum corresponding to each class and refine it, finally obtaining the first-person segmentation pseudo mask; The third stage is: Step 7: Use the generated first-person segmentation pseudo-mask as the label for training the segmentation model, and train the general semantic segmentation model DeepLab V3 to obtain the final deployable first-person visual scene segmentation model.

2. A first-person perspective scene parsing method based on text supervision as claimed in claim 1, characterized in that: The specific formula of step 2 is as follows: P V→T =φ V→T (P V ) P T→V =φ T→V (P T )。 3. A first-person perspective scene parsing method based on text supervision as claimed in claim 1, characterized in that: The specific formula of the prompt vector set input in step 3 is as follows: [E′ V ;F V ]=ψ V ([E V ;Flatten(Conv(I))])=ψ V ([P V ;P T→V ;Flatten(Conv(I))]) [E′ T ;F T ]=ψ T ([E T ;Embed(T CLS )])=ψ T ([P T ;P V→T ;Embed(T CLS )]) In the above formula, Conv(·) represents the input image convolution layer, Flatten(·) represents the flattening operation of the feature, Embed(·) represents the text word embedding operation, [;] represents the concatenation operation, and E V represents the set of input visual prompt vectors, E′ represents the set of visual prompt vectors input by the teacher model, and E T Represents the set of input text prompt vectors, E T ′ represents the set of text prompt vectors input to the teacher model.

4. The first-person perspective scene parsing method based on text supervision as claimed in claim 1, characterized in that: The specific loss of step 4 is expressed as: In the above formula, N represents the dimension of the global feature, and ||·||2 represents the L2 norm.

5. The first-person perspective scene parsing method based on text supervision as claimed in claim 1, characterized in that: In step 5, the corresponding values ​​of λ1 and λ2 are both set to 0.02.

Images