Image segmentation method, method and device for training image segmentation model

Abstract

Embodiments of the present application disclose an image segmentation method, a method and device for training an image segmentation model. The main technical solution includes: obtaining a to-be-segmented image as a query image, and obtaining a support image of the query image and a target description text corresponding to the query image; generating a first pseudo mask for the query image and the support image using the target description text; respectively encoding the query image and the support image to obtain a first feature representation of the query image and a first feature representation of the support image; performing correlation matching on the first feature representation of the query image and the first feature representation of the support image using the first pseudo mask of the query image and the first pseudo mask of the support image; and performing decoding processing using the result of the correlation matching to obtain a segmentation result of the query image, wherein the segmentation result includes position information of a target region. The present application can improve the accuracy of image segmentation in a small sample scenario.

Description

Technical Field

[0001] This application relates to the fields of image processing and artificial intelligence technology, and in particular to an image segmentation method, a method and apparatus for training an image segmentation model. Background Technology

[0002] Image segmentation is the technique and process of dividing an image into several specific regions with unique properties and identifying the target region of interest. It is a crucial preprocessing step for image recognition and computer vision. Image segmentation has many applications in fields such as medicine, autonomous driving, and satellite imaging.

[0003] For images such as remote sensing images that contain multiple targets, have complex scenes, and vary in size, obtaining a large number of mask labels is extremely difficult. Image segmentation models based on few-shot learning often have poor segmentation accuracy for such images. Therefore, improving the accuracy of image segmentation has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, this application provides an image segmentation method, a method and apparatus for training an image segmentation model, in order to improve the accuracy of image segmentation in small sample scenarios.

[0005] This application provides the following solution: Firstly, an image segmentation method is provided, the method comprising: Obtain the image to be segmented as the query image, and obtain the supporting image of the query image and the target description text corresponding to the query image, wherein both the supporting image and the query image contain the target indicated by the target description text; The target description text is used to generate a first pseudo-mask for the query image and the support image respectively. The first pseudo-mask is obtained by masking the target region in the corresponding image. The query image and the support image are encoded respectively to obtain a first feature representation of the query image and a first feature representation of the support image; Using the first pseudomask of the query image and the first pseudomask of the support image, relevance matching is performed on the first feature representation of the query image and the first feature representation of the support image; The results of the correlation matching are used for decoding to obtain the segmentation results of the query image, and the segmentation results include the location information of the target region.

[0006] According to one achievable method in an embodiment of this application, generating the first pseudo-mask includes: performing the following steps using the query image and the supporting image as images to be masked, respectively: The target description text and the image to be masked are input into the mask generator to obtain the second pseudo-mask of the image to be masked; Obtain the first feature representation of the image to be masked; Using the second pseudo-mask and the first feature representation of the image to be masked, obtain the first target prototype representation and the first background prototype representation of the image to be masked; The first pseudo-mask of the image to be masked is obtained by using the similarity between the first feature representation of the image to be masked and the first target prototype representation and the first background prototype representation, respectively.

[0007] According to one achievable method in an embodiment of this application, obtaining a first target prototype representation of the image to be masked using the second pseudo-mask and a first feature representation of the image to be masked includes: Using the second pseudomask and the first feature representation of the query image, a second target prototype representation of the query image is obtained; and using the second pseudomask and the first feature representation of the support image, a second target prototype representation of the support image is obtained. The second target prototype representation of the query image and the second target prototype representation of the supporting image are weighted, and the weighted representation is used as the first target prototype representation of the image to be masked.

[0008] According to one achievable method in an embodiment of this application, obtaining a first background prototype representation of the image to be masked using the second pseudo-mask and a first feature representation of the image to be masked includes: Using the first feature representation and the second pseudo-mask of the image to be masked, the background feature representation of the image to be masked is obtained; By using the first mapping value and the background feature representation of the image to be masked, a first background prototype representation of the image to be masked is obtained, wherein the first mapping value is obtained by mapping the background feature representation of the image to be masked and the first feature representation.

[0009] According to one achievable method in an embodiment of this application, performing relevance matching on the first feature representation of the query image and the first feature representation of the support image using the first pseudomask of the query image and the first pseudomask of the support image includes: The first feature representation of the query image is enhanced using the first pseudo-mask of the query image, and the first feature representation of the support image is enhanced using the first pseudo-mask of the support image; Relevance matching is performed using the enhanced first feature representation of the query image and the enhanced first feature representation of the support image.

[0010] According to one achievable method in an embodiment of this application, the relevance matching using the enhanced first feature representation of the query image and the enhanced first feature representation of the support image includes: Cross-attention processing is performed using the enhanced first feature representation of the query image and the enhanced first feature representation of the support image; Using the result of the cross-attention processing and the first feature representation of the support image, a third target prototype representation of the support image is obtained; The third prototype representation of the supporting image is relevance matched with the enhanced first feature representation of the query image, and / or the third prototype representation of the supporting image is relevance matched with the first feature representation of the query image.

[0011] According to one achievable embodiment of this application, the method further includes: obtaining N fourth target prototype representations of the supporting image using a first pseudomask of the supporting image and a first feature representation of the supporting image, where N is a positive integer; and determining the target association representation of the supporting image using the correlation between the N fourth target prototype representations of the supporting image and the first feature representation of the supporting image. Decoding using the results of the correlation matching includes: decoding using the results of the correlation matching and the target association representation of the supporting image.

[0012] According to one achievable method in an embodiment of this application, using the first pseudomask of the supporting image and the first feature representation of the supporting image, N fourth target prototype representations of the supporting image are obtained, including: Using the first pseudomask of the supporting image, N seed points are determined from the target region of the supporting image; Using the N seed points and the first feature representation of the support image, the vectors corresponding to the patches in the support image in the first feature representation are clustered to obtain N fourth target prototype representations of the support image.

[0013] According to one achievable method in an embodiment of this application, determining N seed points from the target region of the supporting image using the first pseudomask of the supporting image includes: N initial seed points are determined in the target region shown by the first pseudo-mask of the supporting image; For each initial seed point, the following steps are performed: the similarity between the tiles within a preset distance range from the initial seed point and the fifth target prototype representation is calculated, and the tiles whose similarity meets the preset similarity requirements are selected as the determined seed points. The fifth target prototype is represented as either the fifth target prototype of the query image or the fifth target prototype of the support image. The fifth target prototype of the query image is obtained by using the first pseudo-mask of the query image and the first feature representation of the query image. The fifth target prototype of the support image is obtained by using the first pseudo-mask of the support image and the first feature representation of the support image.

[0014] According to one achievable method in an embodiment of this application, the target association representation of the supporting image is determined by utilizing the correlation between the N fourth target prototype representations of the supporting image and the first feature representation of the supporting image, including: For each patch in the support image, determine the average similarity between the vector corresponding to the patch in the first feature representation and the N fourth target prototype representations of the support image; The target association representation of the supporting image is obtained by using the average similarity value of each tile in the supporting image.

[0015] Secondly, an image segmentation method is provided, the method comprising: The remote sensing image to be segmented is obtained as the query image, and the supporting image and the land feature description text corresponding to the query image are obtained, wherein the supporting image is a remote sensing image that contains the land feature indicated by the land feature description text in the query image; The first pseudo-mask is generated for the query image and the support image respectively using the land feature description text. The first pseudo-mask is obtained by masking the land feature area in the corresponding image. The query image and the support image are encoded respectively to obtain a first feature representation of the query image and a first feature representation of the support image; Using the first pseudomask of the query image and the first pseudomask of the support image, relevance matching is performed on the first feature representation of the query image and the first feature representation of the support image; The results of the correlation matching are used for decoding to obtain the segmentation results of the query image, which include the location information of the ground feature areas.

[0016] Thirdly, a method for training an image segmentation model is provided, the method comprising: Acquire training data including multiple training samples, wherein the training samples include: query image samples, supporting image samples of the query image samples, target description text corresponding to the query image samples, and mask labels annotating the target regions in the query image samples, wherein both the query image samples and the supporting image samples contain the target indicated by the target description text; An image segmentation model is trained using the training data. The image segmentation model generates a first pseudo-mask for both the query image sample and the supporting image sample using the target description text. The first pseudo-mask is obtained by masking the target region in the corresponding image sample. The query image sample and the supporting image sample are encoded to obtain a first feature representation of the query image sample and a first feature representation of the supporting image sample. The first pseudo-mask of the query image sample and the first pseudo-mask of the supporting image sample are used to perform correlation matching on the first feature representation of the query image sample and the first feature representation of the supporting image sample. The result of the correlation matching is then decoded to obtain a segmentation result for the query image sample, the segmentation result including the location information of the target region. The training objective includes minimizing the difference between the segmentation result of the image segmentation model on the query image sample and the corresponding mask label.

[0017] According to one achievable method in an embodiment of this application, generating the first pseudo-mask includes: using the query image sample and the support image sample as images to be masked, respectively, and performing the following steps: The target description text and the image to be masked are input into the mask generator to obtain the second pseudo-mask of the image to be masked; Obtain the first feature representation of the image to be masked; Using the second pseudo-mask and the first feature representation of the image to be masked, obtain the first target prototype representation and the first background prototype representation of the image to be masked; The first pseudo-mask of the image to be masked is obtained by using the similarity between the first feature representation of the image to be masked and the first target prototype representation and the first background prototype representation, respectively.

[0018] According to one achievable method in an embodiment of this application, performing relevance matching on the first feature representation of the query image sample and the first feature representation of the support image sample using the first pseudomask of the query image sample and the first pseudomask of the support image sample includes: The first feature representation of the query image sample is enhanced using the first pseudo-mask of the query image sample, and the first feature representation of the support image sample is enhanced using the first pseudo-mask of the support image sample; Relevance matching is performed using the enhanced first feature representation of the query image sample and the enhanced first feature representation of the support image sample.

[0019] According to one achievable method in an embodiment of this application, the relevance matching using the enhanced first feature representation of the query image sample and the enhanced first feature representation of the support image sample includes: Cross-attention processing is performed using the enhanced first feature representation of the query image sample and the enhanced first feature representation of the support image sample; Using the result of the cross-attention processing and the first feature representation of the supporting image sample, a third target prototype representation of the supporting image sample is obtained; The third target prototype representation of the supporting image sample is relevance matched with the enhanced first feature representation of the query image sample, and / or the third target prototype representation of the supporting image sample is relevance matched with the first feature representation of the query image sample.

[0020] According to one achievable embodiment of this application, the method further includes: obtaining N fourth target prototype representations of the supporting image sample using the first pseudomask of the supporting image sample and the first feature representation of the supporting image sample, where N is a positive integer; and determining the target association representation of the supporting image sample by utilizing the correlation between the N fourth target prototype representations of the supporting image sample and the first feature representation of the supporting image sample. Decoding using the results of the correlation matching includes: decoding using the results of the correlation matching and the target association representation of the supporting image samples.

[0021] Fourthly, an image segmentation apparatus is provided, the apparatus comprising: The input unit is configured to acquire an image to be segmented as a query image, and to acquire a supporting image of the query image and a target description text corresponding to the query image, wherein both the supporting image and the query image contain the target indicated by the target description text; The pseudo-mask generation unit is configured to generate a first pseudo-mask for the query image and the support image respectively using the target description text, wherein the first pseudo-mask is obtained by masking the target region in the corresponding image; The encoding unit is configured to encode the query image and the support image respectively to obtain a first feature representation of the query image and a first feature representation of the support image; The relevance matching unit is configured to perform relevance matching on the first feature representation of the query image and the first feature representation of the support image using the first pseudomask of the query image and the first pseudomask of the support image; The decoding unit is configured to perform decoding processing using the results of the correlation matching to obtain a segmentation result of the query image, the segmentation result including the location information of the target region.

[0022] Fifthly, an apparatus for training an image segmentation model is provided, the apparatus comprising: The sample acquisition unit is configured to acquire training data including multiple training samples, wherein the training samples include: query image samples, supporting image samples of the query image samples, target description text corresponding to the query image samples, and mask labels annotating the target regions in the query image samples, wherein both the query image samples and the supporting image samples contain the target indicated by the target description text; The model training unit is configured to train an image segmentation model using the training data. The image segmentation model generates a first pseudo-mask for both the query image sample and the support image sample using the target description text. The first pseudo-mask is obtained by masking the target region in the corresponding image sample. The query image sample and the support image sample are encoded to obtain a first feature representation of the query image sample and a first feature representation of the support image sample. The first pseudo-mask of the query image sample and the first pseudo-mask of the support image sample are used to perform correlation matching on the first feature representation of the query image sample and the first feature representation of the support image sample. The correlation matching result is used for decoding to obtain a segmentation result for the query image sample, the segmentation result including the location information of the target region. The training objective includes minimizing the difference between the segmentation result of the image segmentation model for the query image sample and the corresponding mask label.

[0023] According to a sixth aspect, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any one of the first to third aspects.

[0024] According to the seventh aspect, an electronic device is provided, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any one of the first to third aspects.

[0025] According to the specific embodiments provided in this application, the following technical effects are disclosed: 1) Based on the first pseudomask, this application performs relevance matching between the first feature representation of the query image and the first feature representation of the supporting images, guided by the first pseudomask, and predicts the segmentation result based on the relevance matching result. This approach can achieve more accurate image segmentation on the basis of a coarse first pseudomask, even in small sample scenarios, requiring only the target description text as guidance.

[0026] 2) After generating the second pseudo-mask using a mask generator, this application further utilizes the second pseudo-mask and the first feature representation of the image to determine the first target prototype representation and the first background prototype representation of the image. Then, it uses the similarity between the first feature representation of the image and the first target prototype representation and the first background prototype representation, respectively, to regenerate a more accurate first pseudo-mask. In other words, it provides a method for correcting pseudo-masks, improving the coarse second pseudo-mask generated by the mask generator.

[0027] 3) This application uses the first pseudomask of the query image and the support image to perform target region enhancement processing on the first feature representation of the query image and the support image respectively, and then uses the enhanced first feature representation of the query image and the support image to perform correlation matching, thereby improving the feature correlation of the target region in the query image and the support image, so as to further improve the accuracy of image segmentation.

[0028] 4) In the process of relevance matching, this application uses the enhanced query image and the first feature representation of the support image to perform cross-attention processing. Based on the result of cross-attention processing, the third target prototype representation of the support image is obtained, thereby incorporating spatial information into the relevance matching process and obtaining more fine-grained relevance information to further improve the accuracy of image segmentation.

[0029] 5) This application can extract a fourth target prototype representation of multiple localities for supporting images, thereby obtaining a more detailed target association representation to further improve the accuracy of image segmentation.

[0030] 6) When extracting multiple local fourth target prototype representations from the supporting image, this application selects the plots near the initial seed point as seed points by comparing them with the fifth prototype representations of the query image or supporting image. This seed point selection mechanism can reduce the misleading effect of the inaccuracy of the first pseudo-label on the seed point selection, improve the accuracy of seed point selection, thereby improving the accuracy of the fourth target prototype representation, and ultimately improving the accuracy of image segmentation.

[0031] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a system architecture diagram applicable to the embodiments of this application; Figure 2 A flowchart of the image segmentation method provided in the embodiments of this application; Figure 3 A schematic structural diagram of an image segmentation model provided in an embodiment of this application; Figure 4 A schematic structural diagram of another image segmentation model provided in the embodiments of this application; Figure 5 A flowchart illustrating the method for training an image segmentation model provided in this application embodiment; Figure 6 A schematic block diagram of an image segmentation apparatus provided in an embodiment of this application; Figure 7 A structural diagram of the apparatus for training an image segmentation model provided in the embodiments of this application; Figure 8 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0035] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0036] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0037] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0038] Several methods exist that can perform image segmentation using a mask generator, guided by target description text. The mask generator acquires feature representations of both the target description text and the image to be segmented. These two representations are then used together to predict the segmentation result, i.e., the location information of the target region within the image. However, this approach yields segmentation results with relatively poor accuracy.

[0039] In view of this, this application provides a novel approach that, based on coarse segmentation, further utilizes the correlation matching results of feature representations between the image to be segmented and supporting images to predict the segmentation result of the image to be segmented. To facilitate understanding of the overall scheme of this application, the system architecture to which this application is applied is first briefly described. Figure 1 An exemplary system architecture that can be applied to embodiments of this application is shown, such as Figure 1 As shown, the system architecture includes a model training device and an image segmentation device.

[0040] The model training device is used for offline model training. That is, after acquiring training data, the method provided in the embodiments of this application can be used to train the model and obtain an image segmentation model.

[0041] Image segmentation devices are used online to perform image segmentation processing on images to be segmented using pre-trained image segmentation models, and obtain image segmentation results.

[0042] The model training device and image segmentation device can be configured as separate servers, or they can be configured on the same server or server cluster, or they can be configured on a separate or the same cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Servers (VPs) services, such as high management difficulty and weak service scalability. The model training device and image segmentation device can also be configured on computer terminals with strong computing capabilities.

[0043] It should be noted that, in addition to performing image segmentation online, the above-mentioned image segmentation device can also perform image segmentation offline, for example, segmenting a batch of images to be segmented separately.

[0044] It should be understood that Figure 1 The number of model training devices, image segmentation devices, and image segmentation models shown is merely illustrative. Depending on implementation needs, any number of model training devices, image segmentation devices, and image segmentation models can be included.

[0045] It should be noted that the terms "first," "second," etc., used in this disclosure do not have limitations on size, order, or quantity; they are merely used to distinguish between them by name. For example, "first feature representation" and "second feature representation" are used to distinguish between two feature representations by name. Similarly, "first pseudomask" and "second pseudomask" are used to distinguish between two pseudomasks by name. "First target prototype representation," "second target prototype representation," "third target prototype representation," "fourth target prototype representation," etc., are used to distinguish between different target prototype representations by name, and so on.

[0046] Figure 2 This is a flowchart of an image segmentation method provided in an embodiment of this application. The method can be... Figure 1 The image segmentation device in the system shown is executed using an image segmentation model. For example... Figure 2 As shown, the method may include the following steps: Step 202: Obtain the image to be segmented as the query image, and obtain the supporting image of the query image and the target description text corresponding to the query image, wherein both the supporting image and the query image contain the target indicated by the target description text.

[0047] Step 204: Generate a first pseudo-mask for the query image and the supporting images respectively using the target description text. The first pseudo-mask is obtained by masking the target region in the corresponding image.

[0048] Step 206: Encode the query image and the support image respectively to obtain the first feature representation of the query image and the first feature representation of the support image.

[0049] Step 208: Using the first pseudomask of the query image and the first pseudomask of the support image, perform correlation matching on the first feature representation of the query image and the first feature representation of the support image.

[0050] Step 210: Decode the results of correlation matching to obtain the segmentation results of the query image. The segmentation results include the location information of the target region.

[0051] As can be seen from the above process, this application can perform correlation matching between the first feature representation of the query image and the first feature representation of the supporting image based on the coarse first pseudo-mask, and predict a more accurate segmentation result based on the correlation matching result.

[0052] The steps described above are described in detail below. First, step 202, namely "obtaining the image to be segmented as the query image, and obtaining the supporting images of the query image and the target description text corresponding to the query image", will be described in detail with reference to the embodiment.

[0053] The images to be segmented in this application typically contain different targets in different application fields. For example, in the field of satellite imaging, the images to be segmented are usually remote sensing images, and the purpose of image segmentation is to determine the location of ground features contained in the image. As another example, in the field of face recognition, the images to be segmented are usually images containing faces, and the purpose of image segmentation is to determine the location of the facial regions contained in the image. Yet another example is in the medical field, the images are usually medical images, and the purpose of image segmentation is to determine the location of organ regions or lesion regions contained in the image, and so on.

[0054] In this embodiment, textual prompts are used to guide image segmentation of the query image. These prompts are called target description text. The target description text describes the target in the image to be segmented, and the purpose of image segmentation is to segment the region of the target indicated by the target description text in the image to be segmented. For example, for an image to be segmented containing a horse, in order to segment the region of the horse from the image to be segmented, target description text such as "This is a horse" or "A picture of a horse" can be used as prompts.

[0055] The image segmentation method used in this embodiment is similar to meta-learning, where the image to be segmented is used as the query image, and images containing the same target as the query image are used as support images. For example, if the query image contains a horse, then the support images will also contain a horse. The support images obtained for the query image can be one or more.

[0056] The following describes step 204, namely "generating a first pseudomask for the query image and the supporting image respectively using the target description text", in detail with reference to the embodiments.

[0057] The steps implemented in this process can be achieved using an image segmentation model. For a more intuitive understanding, the structure of an image segmentation model can be found in [link to relevant documentation]. Figure 3 .

[0058] First, let's take querying images as an example, such as... Figure 3As shown, in this step, a mask generator from the prior art, such as CLIP or MaskCLIP, can be used to perform preliminary image segmentation on the query image. That is, the target description text and the query image are input into the mask generator to obtain the initial segmentation result of the query image, and then the initial segmentation result is used for masking.

[0059] CLIP is a visual-language pre-trained model consisting of an image encoder and a text encoder. The image encoder extracts feature vectors from the query image, while the text encoder extracts feature vectors from the target description. These feature vectors are then combined to classify patches in the query image, predicting whether each patch belongs to the target region, thus obtaining the initial segmentation result. MASKCLIP is an improvement on CLIP. It refines the global attention pooling layer of the image encoder in CLIP, removes the query and key embedding layers, and recombines the value embedding layer and the last linear layer into two separate 1×1 convolutional layers. Since both CLIP and MaskCLIP are existing models, they will not be described in detail here.

[0060] One feasible approach is to directly use the initial segmentation result for masking to obtain the first pseudo-mask mentioned above. However, since the image segmentation result of the mask generator is relatively coarse, a more preferred approach provided in this embodiment is to use the initial segmentation result for masking to obtain a second pseudo-mask, and then further refine this second pseudo-mask to obtain the first pseudo-mask of the query image, denoted as... .

[0061] The above-mentioned correction process may include the following steps S1 to S3: Step S11: Obtain the first feature representation of the query image.

[0062] The first feature representation of the query image can be obtained by the encoding module or by other networks with feature extraction capabilities. The first feature representation of the query image is... express.

[0063] Step S12: Using the second pseudo-mask of the query image and the first feature representation of the image to be masked, obtain the first target prototype representation and the first background prototype representation of the query image.

[0064] Prototype is a concept introduced when Prototype Networks were proposed. It refers to representing each category with a vector, so the prototype representation is the vector representation of a category. A common approach is to cluster objects of a category and then use the vector representation of the cluster centroid (i.e., the cluster center) as the vector representation of the prototype for that category.

[0065] In the embodiments of this application, the target prototype representation and the background prototype representation are the prototype representations of the tiles in the target area and the tiles in the background area, respectively.

[0066] As one possible implementation, in step S2 above, when obtaining the first target prototype representation and the first background prototype representation of the query image using the second pseudo-mask and the first feature representation of the query image, the feature vectors of each patch belonging to the target region can be determined using the second pseudo-mask and the first feature representation of the query image. These feature vectors are then aggregated to obtain the feature vectors of the cluster centroids, thus obtaining the first target prototype representation. Alternatively, the feature vectors of each patch not belonging to the target region can be determined using the second pseudo-mask and the first feature representation of the query image. These feature vectors are then aggregated to obtain the feature vectors of the cluster centroids, which serve as the first background prototype representation.

[0067] However, the first target prototype representation and the first background prototype representation of the query image obtained by the above method are not accurate. To improve accuracy, this application provides a more preferred implementation. When determining the first target prototype representation of the query image, the second target prototype representation of the query image can be obtained first using the second pseudomask and the first feature representation of the query image; and the second target prototype representation of the supporting image can be obtained using the second pseudomask and the first feature representation of the supporting image; the second target prototype representation of the query image and the second target representation of the supporting image are weighted, and the representation obtained after weighting is used as the first target prototype representation of the query image.

[0068] For example, the second target representation of the query image mentioned above. The following formula can be used to determine it: (1) in,( , This represents the location of a tile in the query image. and This indicates the width and height of the image being queried. This represents the feature vector corresponding to the patch in the first feature representation of the query image. This indicates the value corresponding to the patch in the second pseudomask of the query image.

[0069] Using the same method, a second target representation supporting the image can be obtained. Since the target representations of the query image and the supporting image should theoretically have a high degree of similarity, the first target prototype representation of the query image can be obtained using formula (2). : (2) in, This is a preset balance factor, i.e., a weighting coefficient. It can also serve as the first target prototype representation supporting the image.

[0070] Since the backgrounds of the query image and the supporting images are very different, the first background prototype representation of the query image is... It is determined independently. The background feature representation of the query image can be obtained using the first feature representation and the second pseudomask of the query image. The first background prototype representation of the query image is obtained by using the first mapping value and the background feature representation of the query image. The first mapping value is obtained by mapping the background feature representation of the query image and the first feature representation.

[0071] The first background prototype representation of the query image The following formula (3) can be used: (3) in, (1- (4) To query the first feature representation of the image, This is the second pseudomask for querying the image.

[0072] Step S13: Utilize the similarity between the first feature representation of the query image and the first target prototype representation and the first background prototype representation to obtain the first pseudo-mask of the query image.

[0073] In this step, we can first calculate the similarity between the feature vector corresponding to each patch in the query image in the first feature representation and the first target prototype representation, and the similarity between the feature vector corresponding to each patch in the query image in the first feature representation and the first background prototype representation. Based on this, we can re-divide the target region and the non-target region to obtain the first pseudo-mask. The similarity can be calculated using methods such as polycosine similarity.

[0074] For example, if the similarity between the feature vector corresponding to a patch in the query image and the first background prototype representation of the query image is greater than the similarity between the feature vector and the first target prototype representation, then the patch is considered to belong to a non-target region. Conversely, if the similarity is less than the similarity, the patch is considered to belong to the target region.

[0075] As can be seen from the above pseudo-mask correction process, based on the coarse second pseudo-mask generated by the mask generator, the target region of the query image can be re-divided by utilizing the similarity between the first feature representation of the query image and the first target prototype representation and the second background prototype representation, respectively, thus eliminating obvious errors in the second pseudo-mask and obtaining a more accurate first pseudo-mask.

[0076] The method for generating the first pseudo-mask for the supporting image is similar to that for the query image, and will not be elaborated here.

[0077] The pseudo-masks involved in the embodiments of this application are all obtained by masking the patches of the target area in the corresponding image. For example, the patches of the target area are marked as 1, and the patches of the non-target area, i.e. the background area, are marked as 0.

[0078] The following describes in detail step 206, namely, "encoding the query image and the support image respectively to obtain the first feature representation of the query image and the first feature representation of the support image", with reference to the embodiments.

[0079] like Figure 3 As shown, in this step, the query image and support images can be encoded using an encoding unit. The encoding unit can be an image encoder such as ViT (Vision Transformer). One encoding unit or two encoding units can be used for the query image and support images, but the parameters of the two encoding units are shared.

[0080] When encoding an image (i.e., a query image or a supporting image), the image encoder performs token-based embedding processing to obtain the embedding representation of each token. Then, it encodes the embedding representation of each token to obtain the first feature representation of the image.

[0081] Each token in an image can include a start symbol or a tile. The token-based embedding process described above includes at least two parts: image embedding and location embedding. Image embedding involves encoding each token into a graph vector representation. Location embedding involves encoding the position of each token within the image sequence to obtain a location representation.

[0082] The following describes in detail step 208, namely, "using the first pseudomask of the query image and the first pseudomask of the support image to perform correlation matching on the first feature representation of the query image and the first feature representation of the support image", with reference to the embodiments.

[0083] This step can be done by Figure 3The relevance matching unit in the model structure shown performs the operation. Since both the query image and the support images contain the target indicated by the target description text, but the backgrounds typically differ significantly, theoretically, the query image and support images have a higher feature correlation in the target region and a lower feature correlation in the background region. Based on this, the target region in the query image can be effectively predicted.

[0084] One possible approach is to first enhance the first feature representation of the query image using a first pseudomask of the query image, and then enhance the first feature representation of the support image using a first pseudomask of the support image; finally, relevance matching is performed using the enhanced first feature representation of the query image and the enhanced first feature representation of the support image.

[0085] More specifically, the relevance matching then using the enhanced first feature representation of the query image and the enhanced first feature representation of the support image can specifically include the following steps S21~S23: Step S21: Perform cross-attention processing using the enhanced first feature representation of the query image and the enhanced first feature representation of the support image.

[0086] This step aims to determine the degree of influence, or interaction, of the feature vectors of each patch in the support image on the feature vectors of each patch in the query image. An attention map can be obtained using the following formula. : (5) in,( () utilizes the first pseudomask of the query image. Enhancement processing is performed on the first feature representation of the query image, ( () utilizes the first pseudomask of the supporting image. Enhancement processing is applied to the first feature representation of the supporting image. ) is a mapping function. The superscript T indicates the transpose of the matrix.

[0087] The attention map is a matrix of the same size as the query image, with each value corresponding to a tile in the query image, representing the degree of influence of tiles at the same location in the support image on that tile.

[0088] Step S22: Using the results of cross-attention processing and the first feature representation of the supporting image, obtain the third target prototype representation of the supporting image.

[0089] In this embodiment of the application, the first feature representation and attention map of the supporting image can be used. The product is used to highlight the target region, and a more concentrated target prototype representation, namely the third target prototype representation, is extracted by clustering icon region patches. For example, the following formula can be used: (6) in, For tiles ( , The corresponding value in the attention map. For tiles ( , The vector corresponding to the first feature representation of the supporting image. For tiles ( , The value corresponding to the first pseudomask of the supporting image.

[0090] Step S23: Perform relevance matching between the third target prototype representation of the supporting image and the first feature representation of the enhanced query image, and / or perform relevance matching between the third target prototype representation of the supporting image and the first feature representation of the query image.

[0091] For example, relevance matching is performed between the third prototype representation of the supporting image and the first feature representation of the enhanced query image to obtain the first relevance map. It can be determined using the following formula: ) (7) Second related graph The following formula can be used to determine it: = (8) The following describes step 210, namely, "using the result of correlation matching for decoding to obtain the segmentation result of the query image, the segmentation result including the location information of the target region," in conjunction with an embodiment.

[0092] In this step, the results of the relevance matching can be input. Figure 3 The decoding unit shown is used to decode the results of correlation matching and predict whether each patch in the query image belongs to the target region.

[0093] The results of the above correlation matching can be the first correlation plot. It can also be a second related graph. It can also be the first related graph. Second correlation graph If it's the first related graph Second correlation graph The first related graph can be Second correlation graph After concatenation, the resulting concatenation matrix is ​​provided to the decoding unit.

[0094] The decoding unit can employ a classification network to map the relevance matching results to specific classification results. The classification in this embodiment mainly includes target regions and non-target regions (i.e., background regions). Specifically, it predicts whether each patch in the query image belongs to the target region and obtains the segmentation result based on the location information of the patches belonging to the target region. For example, if the target is a horse, the region corresponding to the horse in the query image can be obtained from the segmentation result.

[0095] In the above Figure 2 The process shown and Figure 3 Based on the model architecture shown, it can be further included as follows: using the first pseudomask of the supporting image and the first feature representation of the supporting image, N fourth target prototype representations of the supporting image are obtained, where N is a positive integer; using the correlation between the N fourth target prototype representations of the supporting image and the first feature representation of the supporting image, the target association representation of the supporting image is determined. The corresponding model architecture at this time can be as follows: Figure 4 As shown, this further processing can be performed by the prototype supervision unit.

[0096] The prototype supervision unit primarily highlights the association mapping between the first feature representation of the support image and the target by determining the target prototype representation of the support image and calculating its correlation with the first feature representation of the support image. If the value of N is 1, a coarser-grained target association method is used. A larger value for N results in finer granularity, but this also increases the impact on computational performance. Therefore, the value can be chosen based on experience or experimental conditions; for example, N can be set to 5.

[0097] The process of obtaining N fourth target prototype representations of the supporting image using the first pseudo-mask and the first feature representation of the supporting image may include the following steps S31 to S32: Step S31: Using the first pseudo-mask of the supporting image, determine N seed points from the target region of the supporting image.

[0098] In this step, N seed points can be randomly selected from the target region based on the target region indicated by the first pseudo-mask of the supporting image. However, since the first pseudo-mask is a relatively coarse masking result, the selected N seed points may actually be located in the background region. Therefore, this application embodiment provides a more preferred method: First, N initial seed points are determined within the target region shown by the first pseudo-mask of the supporting image. These initial seed points can be randomly selected from the target region shown by the first pseudo-mask, or they can be selected as far away as possible from the mask boundaries and other seed points. In other words, the seed points should be as far apart as possible from each other, and from the mask boundaries (i.e., the boundary between the target region and the background region).

[0099] Then, the positions of the initial seed points can be further corrected to determine the final seed points. That is, for each initial seed point, the following steps are performed: the similarity between the tiles within a preset distance range from the initial seed point and the fifth target prototype representation is calculated, and the tiles whose similarity meets the preset similarity requirements are selected as the determined seed points.

[0100] For example, around the initial seed point * Sampling is performed on the grid. The value is a preset positive integer. The similarity between the sampled points and the fifth target prototype in the supporting image is calculated as follows to obtain the similarity score. : (9) in For sampling points ( , The feature vector corresponding to the first feature representation of the supporting image, and the fifth target prototype representation. It can be the fifth target prototype representation of the query image, or the fifth target prototype representation of the supporting image.

[0101] The fifth target prototype representation of the query image is obtained using the first pseudomask and the first feature representation of the query image. The fifth target prototype representation of the support image is obtained using the first pseudomask and the first feature representation of the support image. It can be determined in a manner similar to that shown in formula (1).

[0102] Step S32: Using N seed points and the first feature representation of the support image, cluster the vectors corresponding to the patches in the support image in the first feature representation to obtain N fourth target prototype representations of the support image.

[0103] After determining N seed points, the vectors corresponding to each patch in the support image in the first feature representation are clustered using the N seed points. In fact, this means clustering to obtain N clusters. Each cluster is actually a target sub-region. The feature vector of the centroid of each cluster can be used to obtain N fourth target prototype representations of the support image.

[0104] After determining the fourth target prototype representation, when determining the target association representation of the supporting image by utilizing the correlation between the N fourth target prototype representations and the first feature representation of the supporting image, the average similarity between the vector corresponding to each patch in the supporting image in the first feature representation and the N fourth target prototype representations of the supporting image can be determined. The target association representation of the supporting image is then obtained using the average similarity value corresponding to each patch in the supporting image. For example, the target association representation can be determined using the following formula. : (10) in, To support the image's first The fourth objective prototype is represented.

[0105] See also Figure 4 Determine the target association representation supporting the image. Then, the target association representation of the supporting image will be... Input decoding unit. The decoding unit will support target-related representation of the image. First related graph Second correlation graph After concatenation, the resulting concatenation matrix is ​​decoded to obtain the segmentation result of the query image.

[0106] If multiple support images are used, then each support image can be obtained separately. , and For all , and Decode after splicing, or process each supported image separately. , and We calculate the average to obtain the averaged result. , and Then, the data is concatenated and decoded.

[0107] Figure 5 This is a flowchart of a method for training an image segmentation model provided in an embodiment of this application. This method can be... Figure 1 The model training device in the system shown is executed. For example... Figure 5 As shown, the method may include the following steps: Step 502: Obtain training data including multiple training samples. The training samples include: query image samples, supporting image samples of query image samples, target description text corresponding to query image samples, and mask labels for the target regions in query image samples. Both query image samples and supporting image samples contain the target indicated by the target description text.

[0108] In this embodiment, a small number of images containing the same target can be acquired, target description text can be provided for these images, and mask labels can be used to annotate the target regions in these images. The annotated mask labels can be considered as accurate labels, which are the targets learned by the image segmentation model, and can be annotated manually.

[0109] Step 504: Train an image segmentation model using training data. The image segmentation model generates a first pseudo-mask for both the query image sample and the support image sample using the target description text. The first pseudo-mask is obtained by masking the target region in the corresponding image sample. Encode the query image sample and the support image sample to obtain the first feature representation of the query image sample and the first feature representation of the support image sample. Use the first pseudo-mask of the query image sample and the first pseudo-mask of the support image sample to perform correlation matching on the first feature representation of the query image sample and the first feature representation of the support image sample. Decode the result of the correlation matching to obtain the segmentation result of the query image sample. The segmentation result includes the location information of the target region. The training objective is to minimize the difference between the segmentation result of the query image sample obtained by the image segmentation model and the corresponding mask label.

[0110] The structure of an image segmentation model can be adopted Figure 3 The structure shown is as follows. The pseudomask generation unit generates a first pseudomask for both the query image sample and the support image sample.

[0111] Specifically, the pseudo-mask generation unit can use the query image and the supporting image as images to be masked respectively: input the target description text and the image to be masked into the mask generator to obtain the second pseudo-mask of the image to be masked; obtain the first feature representation of the image to be masked; use the second pseudo-mask and the first feature representation of the image to be masked to obtain the first target prototype representation and the first background prototype representation of the image to be masked; use the similarity between the first feature representation of the image to be masked and the first target prototype representation and the first background prototype representation respectively to obtain the first pseudo-mask of the image to be masked.

[0112] The encoding unit encodes the query image sample and the support image sample respectively to obtain the first feature representation of the query image sample and the first feature representation of the support image sample.

[0113] The correlation matching unit uses the first pseudomask of the query image sample and the first pseudomask of the supporting image sample to perform correlation matching on the first feature representation of the query image sample and the first feature representation of the supporting image sample.

[0114] As one possible implementation, the relevance matching unit can enhance the first feature representation of the query image using the first pseudo-mask of the query image, and enhance the first feature representation of the support image using the first pseudo-mask of the support image; and perform relevance matching using the enhanced first feature representation of the query image and the enhanced first feature representation of the support image.

[0115] The relevance matching unit can perform cross-attention processing using the enhanced first feature representation of the query image and the enhanced first feature representation of the support image; use the result of the cross-attention processing and the first feature representation of the support image to obtain the third target prototype representation of the support image; perform relevance matching between the third target prototype representation of the support image and the enhanced first feature representation of the query image, and / or perform relevance matching between the third target prototype representation of the support image and the first feature representation of the query image.

[0116] The decoding unit uses the results of correlation matching to perform decoding processing to obtain the segmentation results of the query image sample. The segmentation results include the location information of the target region.

[0117] Furthermore, such as Figure 4 As shown, the prototype supervision unit can use the first pseudomask of the supporting image and the first feature representation of the supporting image to obtain N fourth target prototype representations of the supporting image, where N is a positive integer; and use the correlation between the N fourth target prototype representations of the supporting image and the first feature representation of the supporting image to determine the target association representation of the supporting image.

[0118] Accordingly, the decoding unit can perform decoding processing using the results of correlation matching and the target association representation supporting the image.

[0119] As one possible approach, the prototype supervision unit can use the first pseudomask of the support image to determine N seed points from the target region of the support image; and use the N seed points and the first feature representation of the support image to cluster the vectors corresponding to the patches in the first feature representation of the support image to obtain N fourth target prototype representations of the support image.

[0120] As one possible approach, the prototype supervision unit can determine the average similarity between the vector corresponding to each patch in the first feature representation and the N fourth target prototype representations of the support image for each patch in the support image; and use the average similarity between each patch in the support image to obtain the target association representation of the support image.

[0121] For details on the specific structure and principles of the image segmentation model, please refer to the relevant descriptions in the embodiments of the image segmentation method, which will not be elaborated here.

[0122] The training objective for an image segmentation model can include minimizing the difference between the segmentation result of the image segmentation model on a query image sample and its corresponding mask label. A loss function can be pre-designed for this training objective. In each iteration, the model parameters are updated using methods such as gradient descent based on the value of the loss function, until a preset training termination condition is met. This termination condition can include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0123] Additionally, during model training, an image can be selected as the query image from images containing the same target, along with supporting images. The image segmentation model predicts the segmentation result for the query image. Then, the query image is used as the supporting image, and the supporting image is used as the query image. For each new query image, the segmentation result is predicted. The differences between the two segmentation results and the mask labels of the two images are then combined. For example, two loss functions obtained from the differences between the segmentation results and corresponding mask labels are weighted and summed to obtain the total loss function. The value of the total loss function is then used to update the model parameters in reverse.

[0124] The method provided in this application can be applied to various application scenarios. Here, we will take remote sensing image segmentation in the field of satellite imaging as an example for a brief description. Remote sensing images are characterized by large size variations, complex targets, and difficulties in annotation. The method provided in the embodiments of this application can achieve accurate segmentation of remote sensing images based on small samples.

[0125] Specifically, it can be adopted Figure 3 The image segmentation model is shown. First, the remote sensing image to be segmented is obtained as the query image, along with supporting images and corresponding feature description text. The supporting images are remote sensing images containing the features indicated by the feature description text in the query image.

[0126] Remote sensing images are typically obtained by capturing images of the ground using satellites, spacecraft, or other means. The segmentation requirement for remote sensing images often involves segmenting specific types of land features. These features can include roads, buildings, vegetation, water bodies, and so on. For example, to segment roads in a remote sensing image, descriptive text such as "This is a road image" or "The image contains roads" can be set for the land feature.

[0127] Then, the feature description text is used to generate the first pseudo-mask for the query image and the supporting image respectively. The first pseudo-mask is obtained by masking the feature area in the corresponding image.

[0128] The query image and the support image are then encoded to obtain the first feature representation of the query image and the first feature representation of the support image.

[0129] Next, using the first pseudomask of the query image and the first pseudomask of the supporting image, correlation matching is performed on the first feature representation of the query image and the first feature representation of the supporting image.

[0130] Finally, the results of correlation matching are used for decoding to obtain the segmentation results of the query image. The segmentation results include the location information of the land feature regions, such as the location information of the road region, i.e., which tiles belong to the road region.

[0131] Alternatively, it can also be used Figure 4 The image segmentation model shown uses the first pseudomask and the first feature representation of the supporting image to obtain N fourth target prototype representations of the supporting image, where N is a positive integer. The target association representation of the supporting image is determined by the correlation between the N fourth target prototype representations and the first feature representation of the supporting image. The decoding unit concatenates the target association representation of the supporting image with the aforementioned correlation matching results, and then decodes the concatenated matrix to obtain the segmentation result of the query image.

[0132] For specific details, please refer to the relevant records in the previous method embodiments, which will not be repeated here.

[0133] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0134] Figure 6 This is a schematic block diagram of the image segmentation apparatus provided in the embodiments of this application, such as... Figure 6 As shown, the device may include: an input unit 601, a pseudo-mask generation unit 602, an encoding unit 603, a correlation matching unit 604, and a decoding unit 605, and may further include a prototype supervision unit 606. The main functions of each component are as follows: The input unit 601 is configured to acquire the image to be segmented as the query image, and to acquire the supporting image of the query image and the target description text corresponding to the query image, wherein both the supporting image and the query image contain the target indicated by the target description text.

[0135] The pseudo-mask generation unit 602 is configured to generate a first pseudo-mask for the query image and the supporting image respectively using the target description text. The first pseudo-mask is obtained by masking the target region in the corresponding image.

[0136] The encoding unit 603 is configured to encode the query image and the support image respectively to obtain the first feature representation of the query image and the first feature representation of the support image.

[0137] The relevance matching unit 604 is configured to perform relevance matching on the first feature representation of the query image and the first feature representation of the support image using the first pseudomask of the query image and the first pseudomask of the support image.

[0138] Decoding unit 605 is configured to perform decoding processing using the results of correlation matching to obtain segmentation results of the query image, the segmentation results including the location information of the target region.

[0139] As one possible implementation, the pseudo-mask generation unit 602 can be specifically configured to perform the following processing on the query image and the support image as images to be masked: Input the target description text and the image to be masked into the mask generator to obtain the second pseudo-mask of the image to be masked; obtain the first feature representation of the image to be masked; use the second pseudo-mask and the first feature representation of the image to be masked to obtain the first target prototype representation and the first background prototype representation of the image to be masked; use the similarity between the first feature representation of the image to be masked and the first target prototype representation and the first background prototype representation to obtain the first pseudo-mask of the image to be masked.

[0140] As one possible implementation, the pseudo-mask generation unit 602 can be specifically configured to: obtain a second target prototype representation of the query image using a second pseudo-mask and a first feature representation of the query image; obtain a second target prototype representation of the support image using a second pseudo-mask and a first feature representation of the support image; perform weighted processing on the second target prototype representation of the query image and the second target prototype representation of the support image, and use the weighted representation as the first target prototype representation of the image to be masked.

[0141] As another possible implementation, the pseudo-mask generation unit 602 can be specifically configured to: obtain the background feature representation of the image to be masked using the first feature representation and the second pseudo-mask; and obtain the first background prototype representation of the image to be masked using the first mapping value and the background feature representation of the image to be masked, wherein the first mapping value is obtained by mapping the background feature representation and the first feature representation of the image to be masked.

[0142] As one possible implementation, the relevance matching unit 604 can be specifically configured to: enhance the first feature representation of the query image using the first pseudo-mask of the query image, and enhance the first feature representation of the support image using the first pseudo-mask of the support image; and perform relevance matching using the enhanced first feature representation of the query image and the enhanced first feature representation of the support image.

[0143] The relevance matching unit 604 can perform cross-attention processing using the enhanced first feature representation of the query image and the enhanced first feature representation of the support image; use the result of the cross-attention processing and the first feature representation of the support image to obtain the third target prototype representation of the support image; perform relevance matching between the third prototype representation of the support image and the enhanced first feature representation of the query image, and / or perform relevance matching between the third prototype representation of the support image and the first feature representation of the query image.

[0144] Furthermore, the prototype supervision unit 606 can use the first pseudomask of the supporting image and the first feature representation of the supporting image to obtain N fourth target prototype representations of the supporting image, where N is a positive integer; and use the correlation between the N fourth target prototype representations of the supporting image and the first feature representation of the supporting image to determine the target association representation of the supporting image.

[0145] Accordingly, the decoding unit 605 can perform decoding processing using the results of correlation matching and the target association representation supporting the image.

[0146] As one possible implementation, the prototype supervision unit 606 can be specifically configured to: determine N seed points from the target region of the support image using the first pseudo-mask of the support image; and cluster the vectors corresponding to the patches in the support image in the first feature representation using the N seed points and the first feature representation of the support image to obtain N fourth target prototype representations of the support image.

[0147] Furthermore, when the prototype supervision unit 606 determines N seed points from the target region of the support image using the first pseudo-mask of the support image, it can determine N initial seed points in the target region shown by the first pseudo-mask of the support image. For each initial seed point, it performs the following: calculates the similarity between the patch within a preset distance range from the initial seed point and the fifth target prototype representation, and selects the patch whose corresponding similarity meets the preset similarity requirement as the determined seed point. The fifth target prototype representation is either the fifth target prototype representation of the query image or the fifth target prototype representation of the support image. The fifth target prototype representation of the query image is obtained by using the first pseudo-mask of the query image and the first feature representation of the query image, and the fifth target prototype representation of the support image is obtained by using the first pseudo-mask of the support image and the first feature representation of the support image.

[0148] Figure 7 This is a structural diagram of a device for training an image segmentation model provided in an embodiment of this application. The device corresponds to... Figure 1 The model training device in the system shown. Figure 7 As shown, the device may include a sample acquisition unit 701 and a model training unit 702. The main functions of each component are as follows: The sample acquisition unit 701 is configured to acquire training data including multiple training samples. The training samples include: query image samples, supporting image samples of the query image samples, target description text corresponding to the query image samples, and mask labels for the target regions in the query image samples. Both the query image samples and the supporting image samples contain the target indicated by the target description text.

[0149] The model training unit 702 is configured to train an image segmentation model using training data. The image segmentation model generates a first pseudo-mask for both the query image sample and the support image sample using the target description text. The first pseudo-mask is obtained by masking the target region in the corresponding image sample. The query image sample and the support image sample are encoded to obtain a first feature representation of the query image sample and a first feature representation of the support image sample. The first pseudo-mask of the query image sample and the first pseudo-mask of the support image sample are used to perform correlation matching on the first feature representation of the query image sample and the first feature representation of the support image sample. The result of the correlation matching is decoded to obtain the segmentation result of the query image sample, which includes the location information of the target region. The training objective is to minimize the difference between the segmentation result of the image segmentation model for the query image sample and the corresponding mask label.

[0150] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0151] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0152] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0153] And an electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0154] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0155] in, Figure 8An exemplary architecture of an electronic device is shown, which may include a processor 810, a video display adapter 811, a disk drive 812, an input / output interface 813, a network interface 814, and a memory 820. The processor 810, video display adapter 811, disk drive 812, input / output interface 813, network interface 814, and memory 820 can communicate with each other via a communication bus 830.

[0156] The processor 810 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0157] The memory 820 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 820 can store the operating system 821 for controlling the operation of the electronic device 800, and the basic input / output system (BIOS) 822 for controlling the low-level operations of the electronic device 800. Additionally, it can store a web browser 823, a data storage management system 824, and an image segmentation device / model training device 825, etc. The aforementioned image segmentation device / model training device 825 can be the application program that specifically implements the aforementioned steps in the embodiments of this application. In summary, when the technical solution provided in this application is implemented through software or firmware, the relevant program code is stored in the memory 820 and is called and executed by the processor 810.

[0158] The input / output interface 813 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0159] Network interface 814 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0160] Bus 830 includes a pathway for transmitting information between various components of the device, such as processor 810, video display adapter 811, disk drive 812, input / output interface 813, network interface 814, and memory 820.

[0161] It should be noted that although the above-described device only shows the processor 810, video display adapter 811, disk drive 812, input / output interface 813, network interface 814, memory 820, bus 830, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0162] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0163] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image segmentation method, characterized in that, The method includes: Obtain the image to be segmented as the query image, and obtain the supporting image of the query image and the target description text corresponding to the query image, wherein both the supporting image and the query image contain the target indicated by the target description text; The query image and the support image are respectively used as images to be masked. The target description text and the images to be masked are input into the mask generator to obtain the initial segmentation result of the images to be masked. The initial segmentation result is then used to perform masking processing to obtain the second pseudo-mask of the images to be masked. The second pseudo-mask of the images to be masked includes the second pseudo-mask of the query image and the second pseudo-mask of the support image. The query image and the support image are encoded respectively to obtain the first feature representation of the query image and the first feature representation of the support image; Using the second pseudomask of the query image and the first feature representation of the query image, a first target prototype representation and a first background prototype representation of the query image are obtained. The similarity between the corresponding feature vectors of each patch in the query image in the first feature representation and the first target prototype representation and the first background prototype representation of the query image is calculated to re-divide the target region and non-target region of the query image. The first pseudomask of the query image is obtained based on the re-dividement result. Also, using the second pseudomask of the supporting image and the first feature representation of the supporting image, a first target prototype representation and a first background prototype representation of the supporting image are obtained. The similarity between the corresponding feature vectors of each patch in the supporting image in the first feature representation and the first target prototype representation and the first background prototype representation of the supporting image is calculated to re-divide the target region and non-target region of the supporting image. The first pseudomask of the supporting image is obtained based on the re-dividement result. Using the first pseudomask of the query image and the first pseudomask of the support image, relevance matching is performed on the first feature representation of the query image and the first feature representation of the support image; The results of the correlation matching are used for decoding to obtain the segmentation results of the query image, and the segmentation results include the location information of the target region.

2. The method according to claim 1, characterized in that, Obtaining the first target prototype representation of the query image by using the second pseudomask of the query image and the first feature representation of the query image includes: Using the second pseudomask of the query image and the first feature representation of the query image, a second target prototype representation of the query image is obtained; and using the second pseudomask of the support image and the first feature representation of the support image, a second target prototype representation of the support image is obtained; the second target prototype representation of the query image and the second target prototype representation of the support image are weighted, and the representation obtained after weighting is used as the first target prototype representation of the query image; Obtaining the first background prototype representation of the query image using the second pseudomask and the first feature representation of the query image includes: Using the first feature representation of the query image and the second pseudomask of the query image, a background feature representation of the query image is obtained; using the first mapping value and the background feature representation of the query image, a first background prototype representation of the query image is obtained, wherein the first mapping value is obtained by mapping the background feature representation of the query image and the first feature representation of the query image.

3. The method according to claim 1, characterized in that, Using the first pseudomask of the query image and the first pseudomask of the support image, performing relevance matching on the first feature representation of the query image and the first feature representation of the support image includes: The first feature representation of the query image is enhanced using the first pseudo-mask of the query image, and the first feature representation of the support image is enhanced using the first pseudo-mask of the support image; Relevance matching is performed using the enhanced first feature representation of the query image and the enhanced first feature representation of the support image.

4. The method according to claim 3, characterized in that, The relevance matching using the enhanced first feature representation of the query image and the enhanced first feature representation of the support image includes: Cross-attention processing is performed using the enhanced first feature representation of the query image and the enhanced first feature representation of the support image; Using the result of the cross-attention processing and the first feature representation of the support image, a third target prototype representation of the support image is obtained; The third prototype representation of the supporting image is relevance matched with the enhanced first feature representation of the query image, and / or the third prototype representation of the supporting image is relevance matched with the first feature representation of the query image.

5. The method according to claim 1, characterized in that, The method further includes: using the first pseudomask of the supporting image and the first feature representation of the supporting image to obtain N fourth target prototype representations of the supporting image, where N is a positive integer; and using the correlation between the N fourth target prototype representations of the supporting image and the first feature representation of the supporting image to determine the target association representation of the supporting image. Decoding using the results of the correlation matching includes: decoding using the results of the correlation matching and the target association representation of the supporting image.

6. The method according to claim 5, characterized in that, Using the first pseudomask of the supporting image and the first feature representation of the supporting image, N fourth target prototype representations of the supporting image are obtained, including: Using the first pseudomask of the supporting image, N seed points are determined from the target region of the supporting image; Using the N seed points and the first feature representation of the support image, the vectors corresponding to the patches in the support image in the first feature representation are clustered to obtain N fourth target prototype representations of the support image.

7. The method according to claim 5, characterized in that, The target association representation of the supporting image is determined by utilizing the correlation between the N fourth target prototype representations of the supporting image and the first feature representation of the supporting image, including: For each patch in the support image, determine the average similarity between the vector corresponding to the patch in the first feature representation and the N fourth target prototype representations of the support image; The target association representation of the supporting image is obtained by using the average similarity value of each tile in the supporting image.

8. An image segmentation method, characterized in that, The method includes: The remote sensing image to be segmented is obtained as the query image, and the supporting image and the land feature description text corresponding to the query image are obtained, wherein the supporting image is a remote sensing image that contains the land feature indicated by the land feature description text in the query image; The query image and the supporting image are used as images to be masked. The land feature description text and the images to be masked are input into a mask generator to obtain an initial segmentation result of the images to be masked. The initial segmentation result is then used for masking to obtain a second pseudo-mask of the images to be masked. The second pseudo-mask of the images to be masked includes a second pseudo-mask of the query image and a second pseudo-mask of the supporting image. The query image and the supporting image are encoded to obtain a first feature representation of the query image and a first feature representation of the supporting image. Using the second pseudomask of the query image and the first feature representation of the query image, a first target prototype representation and a first background prototype representation of the query image are obtained. The similarity between the corresponding feature vectors of each patch in the query image in the first feature representation and the first target prototype representation and the first background prototype representation of the query image is calculated to re-divide the target region and non-target region of the query image. The first pseudomask of the query image is obtained based on the re-dividement result. Also, using the second pseudomask of the supporting image and the first feature representation of the supporting image, a first target prototype representation and a first background prototype representation of the supporting image are obtained. The similarity between the corresponding feature vectors of each patch in the supporting image in the first feature representation and the first target prototype representation and the first background prototype representation of the supporting image is calculated to re-divide the target region and non-target region of the supporting image. The first pseudomask of the supporting image is obtained based on the re-dividement result. Using the first pseudomask of the query image and the first pseudomask of the support image, relevance matching is performed on the first feature representation of the query image and the first feature representation of the support image; The results of the correlation matching are used for decoding to obtain the segmentation results of the query image, which include the location information of the ground feature areas.

9. A method for training an image segmentation model, characterized in that, The method includes: Acquire training data including multiple training samples, wherein the training samples include: query image samples, supporting image samples of the query image samples, target description text corresponding to the query image samples, and mask labels annotating the target regions in the query image samples, wherein both the query image samples and the supporting image samples contain the target indicated by the target description text; An image segmentation model is trained using the training data. This model uses the query image and the support images as images to be masked, respectively. The target description text and the images to be masked are input into a mask generator to obtain an initial segmentation result for the images to be masked. The initial segmentation result is then used for masking to obtain a second pseudo-mask for the images to be masked. The second pseudo-mask includes a second pseudo-mask for the query image and a second pseudo-mask for the support images. The query image samples and the support image samples are encoded to obtain a first feature representation for the query image samples and a first feature representation for the support image samples. Using the second pseudo-mask and the first feature representation of the query image, a first target prototype representation and a first background prototype representation of the query image are obtained. The similarity between the corresponding feature vectors of each patch in the query image in the first feature representation and the first target prototype representation and the first background prototype representation of the query image are calculated to re-divide the target and non-target regions of the query image. Based on the re-division results... A first pseudomask of the query image is obtained. Then, using the second pseudomask of the supporting image and the first feature representation of the supporting image, a first feature representation of the supporting image is obtained. A first target prototype representation and a first background prototype representation of the supporting image are also obtained. The similarity between the corresponding feature vectors of each patch in the supporting image in the first feature representation and the first target prototype representation and the first background prototype representation of the supporting image are calculated to re-divide the target and non-target regions of the supporting image. Based on the re-division result, a first pseudomask of the supporting image is obtained. Using the first pseudomask of the query image sample and the first pseudomask of the supporting image sample, correlation matching is performed on the first feature representation of the query image sample and the first feature representation of the supporting image sample. The result of the correlation matching is used for decoding to obtain a segmentation result for the query image sample, where the segmentation result includes the location information of the target region. The training objective includes minimizing the difference between the segmentation result of the image segmentation model for the query image sample and the corresponding mask label.

10. An image segmentation apparatus, characterized in that, The device includes: The input unit is configured to acquire an image to be segmented as a query image, and to acquire a supporting image of the query image and a target description text corresponding to the query image, wherein both the supporting image and the query image contain the target indicated by the target description text; The second pseudo-mask generation unit is configured to take the query image and the support image as images to be masked, respectively, input the target description text and the images to be masked into a mask generator to obtain an initial segmentation result of the images to be masked, and use the initial segmentation result to perform masking processing to obtain a second pseudo-mask of the images to be masked; the second pseudo-mask of the images to be masked includes a second pseudo-mask of the query image and a second pseudo-mask of the support image; The encoding unit is configured to encode the query image and the support image respectively to obtain a first feature representation of the query image and a first feature representation of the support image; The first pseudomask generation unit is configured to use the second pseudomask of the query image and the first feature representation of the query image to obtain a first target prototype representation and a first background prototype representation of the query image; calculate the similarity between the corresponding feature vector of each patch in the query image in the first feature representation and the first target prototype representation and the first background prototype representation of the query image, respectively, to re-divide the target region and non-target region of the query image; and obtain the first pseudomask of the query image based on the re-divide result; and to use the second pseudomask of the support image and the first feature representation of the support image to obtain a first target prototype representation and a first background prototype representation of the support image; calculate the similarity between the corresponding feature vector of each patch in the support image in the first feature representation and the first target prototype representation and the first background prototype representation of the support image, respectively, to re-divide the target region and non-target region of the support image; and obtain the first pseudomask of the support image based on the re-divide result. The relevance matching unit is configured to perform relevance matching on the first feature representation of the query image and the first feature representation of the support image using the first pseudomask of the query image and the first pseudomask of the support image; The decoding unit is configured to perform decoding processing using the results of the correlation matching to obtain a segmentation result of the query image, the segmentation result including the location information of the target region.

11. An apparatus for training an image segmentation model, characterized in that, The device includes: The sample acquisition unit is configured to acquire training data including multiple training samples, wherein the training samples include: query image samples, supporting image samples of the query image samples, target description text corresponding to the query image samples, and mask labels annotating the target regions in the query image samples, wherein both the query image samples and the supporting image samples contain the target indicated by the target description text; The model training unit is configured to train an image segmentation model using the training data. The image segmentation model utilizes the query image and the support image as images to be masked, inputting the target description text and the images to be masked into a mask generator to obtain an initial segmentation result for the images to be masked. The initial segmentation result is then used for masking to obtain a second pseudo-mask for the images to be masked. The second pseudo-mask for the images to be masked includes a second pseudo-mask for the query image and a second pseudo-mask for the support image. The query image samples and the support image samples are encoded to obtain a first feature representation for the query image samples and a first feature representation for the support image samples. Using the second pseudo-mask of the query image and the first feature representation of the query image, a first target prototype representation and a first background prototype representation of the query image are obtained. The similarity between the corresponding feature vectors of each patch in the query image in the first feature representation and the first target prototype representation and the first background prototype representation of the query image are calculated to re-divide the target and non-target regions of the query image. The segmentation result yields a first pseudomask for the query image. Then, using the second pseudomask of the supporting image and the first feature representation of the supporting image, a first feature representation of the supporting image is obtained. The first target prototype representation and the first background prototype representation of each patch in the supporting image are obtained. The similarity between the corresponding feature vector of each patch in the supporting image in the first feature representation and the first target prototype representation and the first background prototype representation of the supporting image are calculated to re-segment the target and non-target regions of the supporting image. Based on the re-segmentation result, a first pseudomask for the supporting image is obtained. Using the first pseudomask of the query image sample and the first pseudomask of the supporting image sample, correlation matching is performed on the first feature representation of the query image sample and the first feature representation of the supporting image sample. The result of the correlation matching is used for decoding to obtain a segmentation result for the query image sample, which includes the location information of the target region. The training objective includes minimizing the difference between the segmentation result of the image segmentation model for the query image sample and the corresponding mask label.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 9.

13. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 9.

Images