Weak supervision small sample part segmentation method and device based on text driving

Abstract

The invention relates to the technical field of computer vision, in particular to a weak supervision small sample part segmentation method and device based on text driving, and the method comprises the steps: extracting image features, and generating a text prototype of a part; calculating pixel-level similarity by using a pseudo mask generation module, distributing a corresponding component category label for each pixel, generating a pseudo label, and extracting a visual component prototype from the image features; based on the text prototype and the pseudo tag, semantic information in the text prototype is embedded into spatial features of the image, so that a bimodal prototype consistency learning module is utilized to maintain the consistency of the part prototype between the visual mode and the text mode; and generating a part segmentation result through a preset bimodal collaborative segmentation strategy. Therefore, the problems that the text information is difficult to be effectively utilized by the model and the semantic consistency between the image and the text cannot be ensured due to the low alignment degree of the text and the image information and the limitation of component prototype learning in the segmentation method of the related technology are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a weakly supervised few-shot part segmentation method and device based on text driving. BACKGROUND

[0002] Semantic segmentation, as a basic task in computer vision, aims to classify each pixel in an image into a specific category. Few-shot learning has become an important research direction for accurate segmentation using less labeled data. Traditional few-shot part segmentation methods rely on pixel-level labeling of support images, which makes the method face the problem of insufficient labeled data in actual application. In some actual scenarios, only text descriptions of parts are available, and pixel-level labeled data are lacking, leading to the problem of weakly supervised few-shot segmentation. The goal of the weakly supervised few-shot segmentation task is to segment the parts in the query image through only the available part-level text labels without providing pixel-level labeled data.

[0003] In related technologies, many researchers propose a combination of weakly supervised learning and cross-modal learning. By introducing text description information, the model can combine the information of two modalities of vision and text, so as to realize accurate segmentation of image parts without pixel-level labels. This method not only reduces the dependence on expensive labeled data, but also enhances the ability of image segmentation by using semantic information.

[0004] However, due to the low alignment degree of text and image information and the limitations of part prototype learning in the segmentation method in related technologies, it is difficult for the model to effectively utilize the text information and it is impossible to guarantee the semantic consistency between the image and the text. It is urgent to improve. SUMMARY

[0005] The present application provides a weakly supervised few-shot part segmentation method and device based on text driving, to solve the problem that the text and image information in the segmentation method in related technologies have a low alignment degree and the limitations of part prototype learning, making it difficult for the model to effectively utilize the text information and impossible to guarantee the semantic consistency between the image and the text.

[0006] In an embodiment of the first aspect of the present application, a text-driven weakly supervised small sample component segmentation method is provided, comprising the following steps: processing a support image and a query image by using an image encoder to extract image features of the images; generating text features based on text descriptions of components by using a text encoder to generate text prototypes of the components; calculating pixel-level similarity based on feature matching between the image features and the text prototypes by using a pseudo-mask generation module, and assigning a corresponding component class label to each pixel based on the pixel-level similarity by using a maximum similarity strategy to generate pseudo-labels, and extracting visual component prototypes from the image features by using a mask average pooling operation; embedding semantic information in the text prototypes into spatial features of the images based on the text prototypes and the pseudo-labels to maintain consistency of the component prototypes between the visual and text modalities by using a dual-modal prototype consistency learning module; and comparing the features of the query image with the visual component prototypes and the text component prototypes and integrating the results by using a preset dual-modal collaborative segmentation strategy to generate component segmentation results.

[0007] Through the above technical solution, the embodiments of the present application can first extract image features and generate text prototypes by using an image encoder and a text encoder, thereby generating a pseudo-mask, and further using the pseudo-mask to guide initial visual prototype extraction; then semantic information in the text prototypes is embedded into spatial features of the images by using a spatial interaction mechanism, effectively enhancing semantic expression of the visual features; further, alignment of the component prototypes between the visual and text modalities is achieved by using intra-modal and cross-modal prototype consistency constraints, thereby improving consistency and generalization ability of the component representation; finally, fine segmentation of the label-free query image is performed by using a dual-modal collaborative segmentation strategy.

[0008] Optionally, in an embodiment of the present application, the dual-modal prototype consistency learning module maintains consistency of the component prototypes between the visual and text modalities, comprising: aligning prototypes of the same component in different images based on a preset intra-modal consistency mechanism; and comparing similarity between the visual component prototypes and the text component prototypes based on a preset cross-modal consistency mechanism to maintain consistency of the component prototypes in the visual and text modalities.

[0009] By the technical solution, the embodiment of the present application can introduce intra-modal consistency constraints and cross-modal consistency constraints based on the dual-modal prototype consistency learning module, ensure alignment of component prototypes in the same modal through the intra-modal consistency constraint mechanism, thereby improving consistency of component semantic representation, perform alignment operation on component prototypes in the visual modal (including support images and query images) and corresponding component prototypes in the text modal through the cross-modal consistency constraint mechanism, thereby realizing consistent representation of component semantics in different modal, and aligning prototype representation of the same component category in the visual and text two modal, thereby improving discriminability and generalization ability of component representation.

[0010] Optionally, in an embodiment of the present application, the pixel-level similarity is calculated based on feature matching between the image features and the text prototypes by the pseudo-mask generation module, including: extracting a visual feature vector corresponding to each pixel; based on the visual feature vector, calculating similarity values between each pixel and each component category.

[0011] Through the technical solution, the embodiment of the present application can calculate pixel-level similarity based on feature matching between image features and text component prototypes, and then classify each pixel into the most matching component category.

[0012] Optionally, in an embodiment of the present application, the maximum similarity strategy is adopted to assign a corresponding component category label to each pixel based on the pixel-level similarity to generate a pseudo label, including: generating a predicted probability distribution of each pixel position on all component categories according to the similarity value; selecting a category index corresponding to a maximum value according to the predicted probability distribution to obtain the pseudo mask, and determining the pseudo label.

[0013] Through the technical solution, the embodiment of the present application can reduce annotation dependence by generating a pseudo mask and determining a pseudo label to effectively make up for the problem of lack of pixel-level annotation in the training stage. The pseudo mask can be further used to provide weak supervision signals to guide subsequent extraction and alignment of component prototypes.

[0014] Optionally, in an embodiment of the present application, the calculation formula of the predicted probability distribution is:

[0015]

[0016] wherein, P ij (k) represents a predicted probability of each pixel position on the component category k, represents a pixel feature of the image (i, j) position, t k represents a text feature of the kth component category, t k′ represents a text feature of the k'th component category.

[0017] By the technical solution, the embodiment of the present application can select a category corresponding to a maximum value by confirming a probability distribution of each pixel position, and then assign a pseudo label to each pixel to obtain a pseudo mask.

[0018] The second aspect embodiment of the present application provides a text-driven weakly supervised small sample component segmentation device, which comprises: an extraction module configured to process a support image and a query image by using an image encoder to extract image features of the images; a first generation module configured to generate a text prototype of a component based on a text description of the component by using a text encoder to generate text features; a second generation module configured to calculate a pixel-level similarity based on feature matching between the image features and the text prototype by using a pseudo mask generation module, assign a corresponding component class label to each pixel based on the pixel-level similarity by using a maximum similarity strategy, generate a pseudo label, and extract a visual component prototype from the image features by using a mask average pooling operation; an embedding module configured to embed semantic information in the text prototype into spatial features of the images based on the text prototype and the pseudo label, and maintain consistency of the component prototype between visual and text modalities by using a dual-modal prototype consistency learning module; and a segmentation module configured to compare features of the query image with the visual component prototype and the text component prototype and combine the results by using a preset dual-modal collaborative segmentation strategy to generate a component segmentation result.

[0019] By the technical solution, the embodiment of the present application can first extract image features and generate a text prototype by using an image encoder and a text encoder, thereby generating a pseudo mask and further guiding initial visual prototype extraction by using the pseudo mask; then embed semantic information in the text prototype into spatial features of the images by using a spatial interaction mechanism, thereby effectively enhancing semantic expression of the visual features; further align the component prototype between visual and text modalities by using intra-modal and cross-modal prototype consistency constraints, thereby improving consistency and generalization ability of the component representation; and finally perform fine segmentation of the label-free query image by using a dual-modal collaborative segmentation strategy.

[0020] Optionally, in an embodiment of the present application, the embedding module comprises: an alignment unit configured to align prototypes of the same component in different images based on a preset intra-modal consistency mechanism; and a comparison unit configured to compare similarity between the visual component prototype and the text component prototype based on a preset cross-modal consistency mechanism, thereby maintaining consistency of the component prototypes in the visual and text modalities.

[0021] By the technical solution, the embodiment of the present application can introduce intra-modal consistency constraints and cross-modal consistency constraints based on the dual-modal prototype consistency learning module, ensure alignment of component prototypes in the same modal through the intra-modal consistency constraint mechanism, thereby improving the consistency of component semantic representation, perform alignment operation on component prototypes in the visual modal (including support images and query images) and corresponding component prototypes in the text modal through the cross-modal consistency constraint mechanism, thereby realizing consistent representation of component semantics in different modal, and aligning prototype representation of the same component category in the visual and text two modal, thereby improving the discriminability and generalization ability of component representation.

[0022] Optionally, in an embodiment of the present application, the second generation module comprises: an extraction unit configured to extract a visual feature vector corresponding to each pixel; and a calculation unit configured to calculate a similarity value between each pixel and each component category based on the visual feature vector.

[0023] Through the technical solution, the embodiment of the present application can calculate pixel-level similarity based on feature matching between image features and text component prototypes, and then classify each pixel into the most matching component category.

[0024] Optionally, in an embodiment of the present application, the second generation module further comprises: a prediction unit configured to generate a predicted probability distribution of each pixel position on all component categories according to the similarity value; and a determination unit configured to select a class index corresponding to a maximum value according to the predicted probability distribution to obtain the pseudo mask and determine the pseudo label.

[0025] Through the technical solution, the embodiment of the present application can reduce the dependence on labeling by generating a pseudo mask and determining a pseudo label, effectively making up for the lack of pixel-level labeling in the training phase. The pseudo mask can be further used to provide a weak supervision signal to guide the extraction and alignment of subsequent component prototypes.

[0026] Optionally, in an embodiment of the present application, the calculation formula of the predicted probability distribution is:

[0027]

[0028] wherein P ij (k) represents a predicted probability of each pixel position on the component category k, represents a pixel feature of the image (i, j) position, t k represents a text feature of the kth component category, t k′ represents a text feature of the k'th component category.

[0029] By the technical solution, the embodiment of the application can select a category corresponding to a maximum value by confirming a probability distribution of each pixel position, and then assign a pseudo label to each pixel to obtain a pseudo mask.

[0030] The third aspect embodiment of the application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the weakly supervised small sample component segmentation method based on text driving as described in the above embodiments.

[0031] The fourth aspect embodiment of the application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the weakly supervised small sample component segmentation method based on text driving as described above.

[0032] The fifth aspect embodiment of the application provides a computer program product, which stores a computer program, and the program is executed by a processor to implement the weakly supervised small sample component segmentation method based on text driving as described above.

[0033] Additional aspects and advantages of the application will be in part apparent and in part pointed out below in the description of the application. BRIEF DESCRIPTION OF DRAWINGS

[0034] The above and / or additional aspects and advantages of the application will become apparent and be readily appreciated from the following description of embodiments, taken in conjunction with the accompanying drawings, in which:

[0035] Figure 1 A flowchart of a weakly supervised small sample component segmentation method based on text driving according to an embodiment of the application is provided.

[0036] Figure 2 A structural schematic diagram of a space interaction module according to an embodiment of the application is provided.

[0037] Figure 3 A flowchart of a pseudo mask generation and visual prototype extraction process according to an embodiment of the application is provided.

[0038] Figure 4 A flowchart of a weakly supervised small sample component segmentation method based on text driving according to an embodiment of the application is provided.

[0039] Figure 5 A schematic diagram of a weakly supervised small sample component segmentation device based on text driving according to an embodiment of the application is provided.

[0040] Figure 6 A structural schematic diagram of an electronic device according to an embodiment of the application is provided. DETAILED DESCRIPTION

[0041] Embodiments of the present application are described below in detail with reference to the accompanying drawings, in which the same or similar numerals or symbols represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0042] A text-driven weakly supervised small sample part segmentation method and device of an embodiment of the present application are described below with reference to the accompanying drawings. In view of the problem that the segmentation method in the related art mentioned in the background art section above has low alignment of text and image information and limitations of part prototype learning, making it difficult for the model to effectively utilize text information and unable to guarantee semantic consistency between images and text, the present application provides a text-driven weakly supervised small sample part segmentation method. In the method, image features can be extracted by a frozen visual encoder, and combined with class-level text prompts, a preliminary pixel-level pseudo mask can be generated through a text-image alignment mechanism. The pseudo mask is further used to provide a weak supervision signal to guide the extraction and alignment of subsequent part prototypes. A text-driven spatial interaction mechanism can be designed to inject text part prototype information into the visual feature space. The visual part perception ability can be guided to enhance through spatial attention or conditional fusion, thereby improving the semantic discriminability and robustness of the visual prototype, solving the problem of insufficient semantic representation in the initial visual part prototype. A dual-modal prototype consistency learning framework can be introduced to fully utilize the visual prototype from the support image and the semantic information from the text label. Through similarity calculation and consistency constraint, the alignment of part prototypes between visual and text modalities is achieved. The method effectively improves the ability to accurately segment parts under small sample conditions, ensures the consistency of part representation in different modalities, and further improves the robustness and accuracy of the segmentation result. Thus, the problem that the segmentation method in the related art has low alignment of text and image information and limitations of part prototype learning, making it difficult for the model to effectively utilize text information and unable to guarantee semantic consistency between images and text, etc. is solved.

[0043] Specifically, Figure 1 A flowchart of a text-driven weakly supervised small sample part segmentation method provided by an embodiment of the present application.

[0044] As Figure 1 shown, the text-driven weakly supervised small sample part segmentation method includes the following steps:

[0045] In step S101, an image encoder is used to process the support image and the query image to extract image features of the images.

[0046] It should be understood by those skilled in the art that in computer vision and cross-modal learning tasks, support images and query images are a pair of key concepts, especially widely used in small sample learning and weakly supervised segmentation tasks.

[0047] Among them, the support image can be understood as an image providing reference information, usually with annotations (such as segmentation masks, class labels or text descriptions) to guide the model to process the query image; the query image can be understood as an image to be processed, and the model needs to analyze it (such as classification, detection, segmentation, etc.) based on the information of the support image.

[0048] In actual execution process, the embodiment of the application can process the input support image and query image through the image encoder to extract the dense feature representation of the image. The image encoder can be a Vision Transformer (ViT) and a multi-scale feature integration module.

[0049] Specifically, the dense visual feature extraction of the image through the image encoder can use a Dense Prediction Transformer (DPT) framework as the basic structure of the image encoder, which includes a frozen ViT backbone network and a trainable multi-scale feature aggregation module. The ViT backbone network uses pre-trained weights on ImageNet and remains in a frozen state during training, and only the parameters of the multi-scale aggregation module are updated. The implementation process of extracting image features is as follows:

[0050] The input support image and query image are represented as I s and I q respectively, and the pixel-level visual features are extracted through the image encoder, and the output is F s and F q , whose dimensions are where C is the number of channels, and represent the height and width of the feature map respectively. The feature vector at each pixel position (i,j) is represented as $\mathbf{F}^{s}{ij}$ and $\mathbf{F}^{q}{ij}$ respectively, whose dimensions are The ViT backbone is responsible for capturing the global context information of the learned image, and the ViT backbone network is used to extract hierarchical features from the image. The multi-scale feature aggregation module up-samples and fuses the features of each layer to obtain dense pixel-level representation.

[0051] The embodiment of the application can process the input support image and query image through the image encoder to convert the input image into a three-dimensional feature vector, so that the obtained image features can be used for subsequent visual component recognition and segmentation tasks.

[0052] In step S102, a text encoder is used to generate a text feature based on the text description of the part to generate a text prototype of the part.

[0053] The text encoder can select a CLIP pre-trained text encoder; and the text prototype of the part refers to an abstract semantic feature of the object part represented by a natural language description. This representation method plays a key role in weakly supervised part segmentation, cross-modal retrieval and other tasks.

[0054] As an implementable manner, the embodiment of the application can map the part-level text label to the same feature space as the image feature based on the category label text description through the text encoder to form a plurality of part prototype vectors. Assuming that the number of categories is K, the obtained text prototype set is represented as T=[t1, t2,..., t K ], wherein each The specific implementation process of the above scheme is as follows:

[0055] The part-level text description in the category label (such as “car wheel”, “car body” and the like) is input into a frozen text encoder, and the text encoder is preferably a Text Encoder branch of a CLIP model. The text encoder outputs a text feature representation t k of each part category k. The text features of all part categories are combined into a part prototype set T=[t1, t2,..., t K ], wherein K is the total number of parts under the category.

[0056] The embodiment of the application can convert the text description of the part into a text feature through the text encoder, and then generate a text prototype of the part. The text prototype contains semantic information of the part and can be used as a reference for comparison with the image feature.

[0057] In step S103, a pseudo mask generation module is used to calculate a pixel-level similarity based on feature matching between the image feature and the text prototype, and based on the pixel-level similarity, a maximum similarity strategy is used to assign a corresponding part category label to each pixel to generate a pseudo label, and a visual part prototype is extracted from the image feature through a mask average pooling operation.

[0058] The visual part prototype can be understood as an abstract feature template representing the composition of an object in computer vision, which automatically captures the common visual patterns at the part level through deep learning. This technology is reshaping the research paradigm of fine-grained recognition, explainable AI and cross-modal understanding.

[0059] After the pseudo label (pseudo mask) is generated, the embodiment of the application can use it as a weakly supervised signal to initialize the visual part prototype of the support graph and the query graph.

[0060] Specifically, for each semantic part category k, its corresponding visual prototype representation is extracted from the pixel features by a masked average pooling way:

[0061]

[0062] wherein, represents the pixel feature of the image (i, j) spatial position, represents the real label of the image (i, j) position, represents the support visual prototype of the kth category, is an indicator function, which takes the value of 1 if the condition is true, otherwise 0. Finally, the initial support map visual prototypes of all categories constitute the set Similarly, the initial visual prototype of the query image can be obtained

[0063] The embodiment of the application can generate a weakly supervised pseudo label mask to replace manual annotation by using the semantic similarity relationship between the pixel-level visual features extracted in step S101 and the part-level text prototype constructed in step S102; and further use the mask to calculate the initial visual part prototype, so as to be used for subsequent prototype alignment and feature guidance.

[0064] In step S104, based on the text prototype and the pseudo label, the semantic information in the text prototype is embedded into the spatial feature of the image, so as to maintain the consistency of the part prototype between the visual and text modalities by using a dual-modal prototype consistency learning module.

[0065] The consistency of the part prototype can be understood as: the feature representation of the same semantic part remains stable and can be mapped to each other under different conditions, perspectives or modalities. The core is to ensure that the part-level visual feature can still maintain semantic uniformity and geometric rationality in the changing environment, which can be understood from the following dimensions: semantic consistency, geometric consistency, modal consistency, instance consistency, dynamic consistency, etc.

[0066] As a specific implementation, in order to overcome the problem of insufficient category information in the initial visual part prototype and effectively introduce the semantic rich text prototype information, the embodiment of the application proposes a text-driven spatial interaction module, which aims to inject the semantic information in the text part prototype into the spatial dimension of the visual feature, thereby enhancing the part-level semantic perception ability and improving the discriminability and robustness of the visual prototype. The embedding process can be realized by the following process:

[0067] As shown in Figure 2 , first, the embodiment of the application can obtain the pseudo mask The text component prototype t corresponding to the label class k of each pixel position k is mapped into a spatial tensor, and a visual feature F s is constructed. s wherein:

[0068]

[0069] represents the feature vector at the (i, j) spatial position in the text feature tensor.

[0070] Subsequently, the constructed text feature tensor T s is element-wise added to the visual feature F s to obtain a fused feature tensor, and a multi-head self-attention mechanism (MSA) is further used to model the pixel relationship in space, outputting an enhanced support feature representation

[0071]

[0072] wherein, represents an element-wise addition operation, and MSA(.) represents a multi-head self-attention module applied to the input feature tensor.

[0073] Next, a pseudo mask is used to mask average pool the enhanced feature to obtain an enhanced support component prototype set In order to improve the semantic fusion effect and context understanding ability, a modulator operation is used to dynamically fuse the initial component prototype with its enhanced version . The modulator realizes the attention mechanism in the channel dimension through a multi-layer perceptron (MLP) to generate a fusion weight W s :

[0074]

[0075] wherein, [;]] represents channel dimension concatenation, and σ is a Sigmoid activation function.

[0076] Finally, the fused support component prototype is represented as:

[0077]

[0078] wherein, represents a channel-wise weighting operation. For the query image, the enhanced feature is also masked with a pseudo mask Acquiring an enhanced component prototype and the initial prototype Through the same modulation strategy fusion, the final query image component prototype V is obtained q .

[0079] The embodiment of the application can inject semantic information in the text component prototype into the spatial dimension of the visual feature through embedding operation, effectively overcome the problem of insufficient class information in the initial visual component prototype, and effectively introduce the semantic rich text prototype information, thereby enhancing the component level semantic perception ability and improving the discriminability and robustness of the visual prototype.

[0080] In step S105, the features of the query image are compared with the visual component prototype and the text component prototype by a preset bimodal collaborative segmentation strategy, and the results are integrated to generate a component segmentation result.

[0081] Specifically, for each unannotated query image to be processed, first, the pixel-level feature representation thereof is extracted. Then, the features of the query image are respectively compared with the pre-constructed visual component prototype and the text component prototype for similarity calculation. The similarity results based on the visual prototype and the similarity results based on the text prototype are obtained respectively. The two kinds of similarity results are weighted and fused to obtain a fused similarity representation. Finally, for each pixel position, the class with the maximum similarity is selected as the predicted component class thereof, thereby realizing pixel-level component segmentation of the target image. The collaborative segmentation strategy does not need to rely on pixel-level supervision annotation, and has good generalization ability and is suitable for various small sample component segmentation task scenarios.

[0082] The embodiment of the application can perform high-precision segmentation on the component regions in the target query image under the condition of lacking pixel-level supervision annotation through the preset bimodal collaborative segmentation strategy; by fusing the complementary advantages of the visual modality and the text modality, the accuracy and segmentation robustness of component recognition under the condition of small samples are significantly improved.

[0083] In summary, as shown in Figure 3 The embodiment of the application can divide the text-driven weakly supervised small sample component segmentation method into four main parts, namely, pseudo mask generation, text-driven spatial interaction, bimodal prototype consistency learning, and bimodal collaborative segmentation. After the image features and component text prototypes are obtained based on the image encoder and the text encoder, the four parts are collaboratively used to realize high-precision component-level segmentation.

[0084] Optionally, in an embodiment of the present application, the dual-modal prototype consistency learning module is used to maintain the consistency of component prototypes between the visual and text modalities, including: aligning the prototypes of the same type of components in different images based on a preset intra-modal consistency mechanism; and comparing the similarity between the visual component prototypes and the text component prototypes based on a preset cross-modal consistency mechanism to maintain the consistency of the component prototypes in the visual and text modalities.

[0085] In this embodiment of the present application, the dual-modal prototype consistency learning module includes two sub-modules, namely the intra-modal consistency constraint sub-module and the cross-modal consistency constraint sub-module, and corresponds to the preset intra-modal consistency (constraint) mechanism and the preset cross-modal consistency (constraint) mechanism, respectively.

[0086] As a specific implementation, the intra-modal consistency constraint sub-module aims to align the component prototype representations between the support image and the query image in the same modality. Since the same component in different images should have similar semantic representations, the intra-modal consistency constraint, through the way of contrast learning, constrains the consistency of the visual component prototype from the support image with the component prototype in the query image. Specifically, the intra-modal consistency loss function calculates the similarity between the support image prototype and the query image prototype, and by minimizing the loss function, ensures the alignment of the component prototypes in the same modality. The definition of the loss function is as follows:

[0087]

[0088] wherein, represents the visual component prototype representation of the nth category in the support image, represents the visual component prototype representation of the mth category in the query image, sim(.,.) is used to measure the similarity between the prototype features, and τ is a temperature coefficient used to control the smoothness of the distribution. In this embodiment, the intra-modal consistency constraint aims to make the component prototypes of the same category from different images in the same modality have high similarity, so as to enhance the stability and robustness of the prototype features, thereby improving the consistency of the component semantic representation. By maximizing the similarity between the positive sample prototypes and minimizing the similarity with other category component prototypes, the prototype learning process is effectively constrained.

[0089] Further, to improve the alignment effect between multi-modal features, the embodiment of the present application introduces a cross-modal consistency constraint mechanism. This mechanism aligns the component prototypes in the visual modal (including the support image and the query image) with the component prototypes in the corresponding text modal, thereby achieving consistent representation of component semantics in different modalities. The cross-modal constraint helps to alleviate the semantic shift problem between modalities and improves the accuracy and generalization ability of component segmentation. The above cross-modal consistency constraint can be modeled by the following loss function:

[0090]

[0091] wherein, and respectively represent the visual component prototypes in the support image and the query image, t k represents the text prototype of the corresponding component. This loss function encourages the visual component prototype to be close to its corresponding text semantics, thereby achieving consistent alignment between modalities.

[0092] Finally, the intra-modal consistency loss and the cross-modal consistency loss jointly constitute the total loss target of the dual-modal prototype consistency learning. In the optimization process, the component prototype representations in the same modality and different modalities are aligned simultaneously, thereby improving the perception ability of the model for small sample components and the cross-modal generalization performance.

[0093] The embodiment of the present application can introduce intra-modal consistency constraint and cross-modal consistency constraint based on the dual-modal prototype consistency learning module. The intra-modal consistency constraint mechanism ensures the alignment of component prototypes in the same modality, thereby improving the consistency of component semantic representation. The cross-modal consistency constraint mechanism aligns the component prototypes in the visual modal (including the support image and the query image) with the component prototypes in the corresponding text modal, thereby achieving consistent representation of component semantics in different modalities. The prototype representations of the same component category in the visual and text modalities are aligned, thereby improving the discriminability and generalization ability of component representation.

[0094] Optionally, in an embodiment of the present application, the pseudo mask generation module calculates the pixel-level similarity based on the feature matching between image features and text prototypes, including: extracting the visual feature vector corresponding to each pixel; based on the visual feature vector, calculating the similarity value between each pixel and each component category.

[0095] In actual execution process, the embodiment of the present application can extract the visual feature vector corresponding to each pixel, and perform dot product operation with all text prototypes to calculate the similarity value between each pixel and each component category.

[0096] The embodiments of this application can calculate pixel-level similarity based on feature matching between image features and text component prototypes, and then classify each pixel into the most matching component category.

[0097] Optionally, in one embodiment of this application, based on pixel-level similarity, a maximum similarity strategy is used to assign a corresponding component category label to each pixel to generate a pseudo label, including: generating a predicted probability distribution of each pixel position across all component categories based on the similarity value; selecting the category index corresponding to the maximum value of the predicted probability distribution to obtain a pseudo mask and determine the pseudo label.

[0098] Specifically, such as Figure 4 As shown, taking the supporting image as an example, for each pixel $(i,j)$, its corresponding visual feature vector is extracted and then a dot product operation is performed with all text prototypes using the following formula to measure its semantic similarity with each component category. The softmax function is used to normalize the similarity values ​​of all component categories to generate the predicted probability distribution of each pixel position in all component categories.

[0099] For each pixel location, this embodiment can select the category index corresponding to the maximum value of its predicted probability distribution, assign a pseudo-label to the pixel, and obtain a pseudo-mask. Size is The category value ranges from 1 to K.

[0100]

[0101] Perform a similar operation on the query image to obtain the pseudomask of the query image. The above steps complete the generation of pseudomasks for the supporting and query images.

[0102] The embodiments of this application can reduce the dependence on annotation by generating a pseudo-mask and determining pseudo-labels, effectively making up for the lack of pixel-level annotations in the training stage. The pseudo-mask can be further used to provide weak supervision signals to guide the extraction and alignment of subsequent component prototypes.

[0103] Optionally, in one embodiment of this application, the formula for calculating the predicted probability distribution is:

[0104]

[0105] Among them, P ij (k) represents the predicted probability of each pixel location in part category k. t represents the pixel feature at position (i, j) in the image. k The text feature representing the k-th component category, t k′ The text feature represents the k′-th component category.

[0106] In actual implementation, after measuring the semantic similarity between each pixel and each component category, the embodiment of the application can normalize the similarity values of all component categories by using a softmax function to generate a prediction probability distribution of each pixel position on all component categories.

[0107] The embodiment of the application can select the category corresponding to the maximum value of the probability distribution of each pixel position, and then assign a pseudo label to each pixel to obtain a pseudo mask.

[0108] The weakly supervised small sample component segmentation method based on text driving proposed in the embodiment of the application can extract image features through a frozen visual encoder, and combine a class-level text prompt to generate an initial pixel-level pseudo mask through a picture-text alignment mechanism. The pseudo mask is further used to provide a weak supervision signal to guide the extraction and alignment of subsequent component prototypes. A text-driven spatial interaction mechanism can be designed to inject the information of a text component prototype into a visual feature space. The visual component perception ability can be guided to be enhanced through spatial attention or conditional fusion, so as to improve the semantic discriminability and robustness of the visual prototype, and solve the problem of insufficient semantic representation in the initial visual component prototype. A dual-modal prototype consistency learning framework can be introduced to fully utilize the visual prototype from the support image and the semantic information from the text label. Through similarity calculation and consistency constraint, the alignment of the component prototype between the visual and text modalities is realized. This method effectively improves the ability to accurately segment components under small sample conditions, ensures the consistency of component representation in different modalities, and further improves the robustness and accuracy of the segmentation result. Thus, the problems in the related art that the segmentation method cannot effectively utilize the text information and cannot guarantee the semantic consistency between the image and the text due to the low alignment degree of the text and the image information and the limitation of the component prototype learning are solved.

[0109] Secondly, refer to the attached Figure 5 The weakly supervised small sample component segmentation device based on text driving proposed in the embodiment of the application is described.

[0110] Figure 5 is a block schematic diagram of the weakly supervised small sample component segmentation device based on text driving in the embodiment of the application.

[0111] As Figure 5 shown, the weakly supervised small sample component segmentation device based on text driving 10 includes an extraction module 100, a first generation module 200, a second generation module 300, an embedding module 400, and a segmentation module 500.

[0112] The extraction module 100 is configured to process support images and query images by using an image encoder to extract image features of the images.

[0113] The first generation module 200 is configured to generate a text feature based on a text description of a component by using a text encoder, so as to generate a text prototype of the component.

[0114] The second generation module 300 is configured to calculate a pixel-level similarity based on a feature matching between the image feature and the text prototype by using a pseudo mask generation module, and assign a corresponding component category label to each pixel based on the pixel-level similarity by using a maximum similarity strategy, so as to generate a pseudo label, and extract a visual component prototype from the image feature by using a mask average pooling operation.

[0115] The embedding module 400 is configured to embed semantic information in the text prototype into spatial features of the image based on the text prototype and the pseudo label, so as to maintain consistency of the component prototype between the visual and text modalities by using a dual-modal prototype consistency learning module.

[0116] The segmentation module 500 is configured to compare the feature of the query image with the visual component prototype and the text component prototype by using a preset dual-modal collaborative segmentation strategy, and integrate the results, so as to generate a component segmentation result.

[0117] Optionally, in an embodiment of the present application, the embedding module 400 comprises an alignment unit and a comparison unit; the alignment unit is configured to align prototypes of the same component in different images based on a preset intra-modal consistency mechanism; and the comparison unit is configured to compare similarity between the visual component prototype and the text component prototype based on a preset cross-modal consistency mechanism, so as to maintain consistency of the component prototype in the visual and text modalities.

[0118] Optionally, in an embodiment of the present application, the second generation module 300 comprises an extraction unit and a calculation unit; the extraction unit is configured to extract a visual feature vector corresponding to each pixel; and the calculation unit is configured to calculate a similarity value between each pixel and each component category based on the visual feature vector.

[0119] Optionally, in an embodiment of the present application, the second generation module 300 further comprises a prediction unit and a determination unit; the prediction unit is configured to generate a prediction probability distribution of each pixel position on all component categories according to the similarity value; and the determination unit is configured to select a category index corresponding to a maximum value according to the prediction probability distribution, so as to obtain a pseudo mask and determine the pseudo label.

[0120] Optionally, in an embodiment of the present application, a calculation formula of the prediction probability distribution is as follows:

[0121]

[0122] wherein, P ij (k) represents a prediction probability of each pixel position on a component category k, pixel feature representing the image (i, j) position, t k text feature representing the kth component class, t k′ text feature representing the k'th component class.

[0123] It should be noted that the foregoing explanation of the text-driven weakly supervised small sample component segmentation method embodiment also applies to the text-driven weakly supervised small sample component segmentation device of this embodiment, which will not be described here.

[0124] The text-driven weakly supervised small sample component segmentation device provided by the embodiments of the present application can extract image features through a frozen visual encoder, and combine class-level text prompts to generate a preliminary pixel-level pseudo mask through a text-image alignment mechanism. The pseudo mask is further used to provide a weak supervision signal to guide the extraction and alignment of subsequent component prototypes. A text-driven spatial interaction mechanism can be designed to inject text component prototype information into the visual feature space. Spatial attention or conditional fusion can be used to guide the enhancement of visual component perception ability, thereby improving the semantic discriminability and robustness of visual prototypes and solving the problem of insufficient semantic representation in initial visual component prototypes. A dual-modal prototype consistency learning framework can be introduced to fully utilize visual prototypes from support images and semantic information from text labels. Through similarity calculation and consistency constraints, component prototypes can be aligned between visual and text modalities. This method effectively improves the ability to accurately segment components under small sample conditions and ensures the consistency of component representation in different modalities, thereby further improving the robustness and accuracy of the segmentation results. Thus, the problems in the related art that the alignment degree of text and image information is low and the component prototype learning is limited, making it difficult for the model to effectively utilize text information and unable to guarantee the semantic consistency between images and text are solved.

[0125] Figure 6 The structure schematic diagram of an electronic device provided by the embodiments of the present application is shown. The electronic device can include:

[0126] The memory 601, the processor 602, and the computer program stored in the memory 601 and executable on the processor 602.

[0127] The processor 602 executes the program to implement the text-driven weakly supervised small sample component segmentation method provided in the above embodiments.

[0128] Further, the electronic device further includes:

[0129] The communication interface 603 is used for communication between the memory 601 and the processor 602.

[0130] The memory 601 is configured to store a computer program capable of being executed on the processor 602.

[0131] The memory 601 can include a high-speed RAM memory, and can further include a non-volatile memory, for example, at least one disk memory.

[0132] If the memory 601, the processor 602 and the communication interface 603 are independently implemented, the communication interface 603, the memory 601 and the processor 602 can be connected with each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 6 Only one thick line is used in the figure to represent that there is only one bus or only one type of bus.

[0133] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can complete communication between each other through an internal interface.

[0134] The processor 602 can be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0135] The embodiment of the present application further provides a computer readable storage medium, which has a computer program stored thereon, and the program is executed by a processor to implement the text-driven weakly supervised small sample component segmentation method as above.

[0136] The embodiment of the present application further provides a computer program product, which has a computer program stored thereon, and the program is executed by a processor to implement the text-driven weakly supervised small sample component segmentation method as above.

[0137] In the description of the application, reference to "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that a particular feature, structure, material, or characteristic being described is included in at least one embodiment or example of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment or example. Furthermore, the described specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples. In addition, the usage of "N" means at least two, for example, two, three or the like, unless explicitly stated otherwise.

[0138] Furthermore, the terms "first", "second", or the like, are used merely as a designation of certain elements or features, and do not imply or connote relative importance or a specific order of categorization of the indicated features. Accordingly, features described as "first" or "second" can be explicitly or implicitly included in at least one of the features. In the description of the application, the term "N" means at least two, for example, two, three, etc., unless explicitly stated otherwise.

[0139] Any process or method descriptions or blocks in flow charts or otherwise described herein represent embodiments which can be managed as one or more modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions or steps, and alternate implementations are possible. In some embodiments, the processes and methods described can be executed by one or more apparatuses or devices, either directly or after conversion to another language. Alternatively, the processes and methods described herein can be executed by more than one apparatus or device working in concert.

[0140] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of executable instructions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a product of the manufacturing and / or processing. The computer-readable medium can include, but is not limited to, the following: an electronic connection (an electronic device with one or N wires), a portable computer diskette (a magnetic device), a RAM (random access memory), a ROM (read-only memory), an EPROM (erasable programmable ROM) or a Flash memory, an optical fiber, and a portable CD ROM. In addition, the computer-readable medium can even be paper or other suitable medium upon which the program can be printed, because the program can be electronically captured, via the optically scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in an electronic manner into a computer storage medium, and then stored in the computer storage medium.

[0141] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware and in another embodiment, the hardware can be implemented using any or a combination of the following technologies, which are all well known in the art: a discrete logic circuit having logic gates for implementing logic functions on data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0142] Those of skill in the art would understand that the steps carried out in the above-mentioned embodiments can be implemented by programs instructing relevant hardware to complete all or part of the steps, and the programs can be stored in a computer-readable storage medium. When the programs are executed, the programs include one of the steps of the method embodiments or a combination thereof.

[0143] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0144] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A text-driven weakly supervised few-shot part segmentation method, characterized in that, The method comprises the following steps: processing the support image and the query image by using an image encoder to extract image features of the image; generating text features based on the text description of the component by using a text encoder to generate a text prototype of the component; calculating pixel-level similarity based on feature matching between the image features and the text prototype by using a pseudo-mask generation module, and assigning a corresponding component category label to each pixel based on the pixel-level similarity by using a maximum similarity strategy to generate a pseudo-label, and extracting a visual component prototype from the image features by using a mask average pooling operation; embedding semantic information in the text prototype into spatial features of the image based on the text prototype and the pseudo-label to maintain consistency of the component prototype between the visual and text modalities by using a dual-modal prototype consistency learning module; comparing the features of the query image with the visual component prototype and the text component prototype and synthesizing the results by using a preset dual-modal collaborative segmentation strategy to generate a component segmentation result.

2. The method of claim 1, wherein, The method of maintaining consistency of the component prototype between the visual and text modalities by using the dual-modal prototype consistency learning module comprises: aligning the prototypes of the same type of component in different images based on a preset intra-modal consistency mechanism; comparing the similarity between the visual component prototype and the text component prototype based on a preset cross-modal consistency mechanism to maintain consistency of the component prototype in the visual and text modalities.

3. The method of claim 1, wherein, The method of calculating pixel-level similarity based on feature matching between the image features and the text prototype by using the pseudo-mask generation module comprises: extracting a visual feature vector corresponding to each pixel; calculating similarity values between each pixel and each component category based on the visual feature vector.

4. The method of claim 1, wherein, The method of assigning a corresponding component category label to each pixel based on the pixel-level similarity by using a maximum similarity strategy to generate a pseudo-label comprises: generating a predicted probability distribution of each pixel position on all component categories according to the similarity values; selecting a category index corresponding to a maximum value according to the predicted probability distribution to obtain the pseudo-mask and determine the pseudo-label.

5. The method of claim 4, wherein, The calculation formula of the predicted probability distribution is: where P ij (k) denotes the predicted probability of each pixel location being in component class k, denotes the pixel feature of image location (i, j), t k denotes the text feature of the kth component class, t k′ denotes the text feature of the k'th component class.

6. A text-driven weakly supervised few-shot component segmentation apparatus, comprising: The method comprises the following steps: extracting image features of the image by processing the support image and the query image by using an image encoder; generating a text prototype of the component based on text features generated by a text encoder based on the text description of the component by using a first generation module; calculating pixel-level similarity based on feature matching between the image features and the text prototype by using a pseudo-mask generation module, and assigning a corresponding component category label to each pixel based on the pixel-level similarity by using a maximum similarity strategy to generate a pseudo-label, and extracting a visual component prototype from the image features by using a mask average pooling operation by using a second generation module; embedding semantic information in the text prototype into spatial features of the image based on the text prototype and the pseudo-label to maintain consistency of the component prototype between the visual and text modalities by using a dual-modal prototype consistency learning module by using an embedding module; comparing the features of the query image with the visual component prototype and the text component prototype and synthesizing the results by using a preset dual-modal collaborative segmentation strategy to generate a component segmentation result. The segmentation module is configured to compare and synthesize the features of the query image with the visual component prototype and the text component prototype based on a preset bimodal collaborative segmentation strategy to generate a component segmentation result.

7. The apparatus of claim 6, wherein, The embedding module comprises: An alignment unit configured to align the prototypes of the same type of component in different images based on a preset intra-modal consistency mechanism; A comparison unit configured to compare the similarity between the visual component prototype and the text component prototype based on a preset cross-modal consistency mechanism to maintain the consistency of the component prototypes in the visual and text modalities.

8. An electronic device, comprising: The computer program is executed by the processor to implement the text-driven weakly supervised few-shot component segmentation method according to any one of claims 1-5. The program is executed by the processor to implement the text-driven weakly supervised few-shot component segmentation method according to any one of claims 1-5.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed to implement the text-driven weakly supervised few-shot component segmentation method according to any one of claims 1-5.

10. A computer program product comprising a computer program, characterized in that, ​

Images