Method, apparatus, device and storage medium for image segmentation

Abstract

Embodiments of the present disclosure provide a method, device, equipment and storage medium for image segmentation. The method comprises: extracting an image feature representation of a target image by using a trained image encoder; for each of a plurality of categories, generating a text feature representation corresponding to a name of the category by using a trained text encoder, determining a candidate segmentation map for the target image and a category confidence of the category based on the image feature representation and the text feature representation; selecting at least one category related to the target image from the plurality of categories based on a plurality of category confidences respectively determined for the plurality of categories; and determining a target segmentation map for the target image based on the candidate segmentation map and the category confidence determined for the selected at least one category. Thus, the image segmentation function can be realized by using the image encoder and the text encoder with feature extraction function without the need for labeled data for image segmentation.

Description

TECHNICAL FIELD

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and in particular, to a method, an apparatus, a device and a computer readable storage medium for image segmentation. BACKGROUND

[0002] Image segmentation, also known as image semantic segmentation, is a process of dividing an image into several regions with specific characteristics to help extract the target of interest in the image. Image segmentation is an important step from image processing to image analysis tasks. The result of image segmentation is to divide a digital image into mutually disjoint regions. The process of image segmentation is also a labeling process, i.e. pixels belonging to the same category are assigned the same number. Image segmentation methods are mainly divided into the following categories: threshold-based segmentation method, region-based segmentation method, edge-based segmentation method and segmentation method based on specific theory, etc. SUMMARY

[0003] In a first aspect of the present disclosure, a method for image segmentation is provided. The method comprises: extracting an image feature representation of a target image by using a trained image encoder; for each of a plurality of categories, generating a text feature representation corresponding to a name of the category by using a trained text encoder, determining a candidate segmentation map for the target image and a category confidence of the category based on the image feature representation and the text feature representation, the candidate segmentation map indicating whether a corresponding pixel in the target image belongs to the category; selecting at least one category related to the target image from the plurality of categories based on a plurality of category confidences respectively determined for the plurality of categories; and determining a target segmentation map for the target image based on the candidate segmentation map and the category confidence determined for the selected at least one category, the target segmentation map indicating whether a corresponding pixel in the target image belongs to a category in the at least one category.

[0004] In a second aspect of the disclosure, an apparatus for image segmentation is provided. The apparatus comprises: an image feature extraction module configured to extract an image feature representation of a target image by utilizing a trained image encoder; a feature processing module configured to, for each of a plurality of categories, generate a text feature representation corresponding to a name of the category by utilizing a trained text encoder, determine a candidate segmentation map for the target image and a category confidence of the category based on the image feature representation and the text feature representation, the candidate segmentation map indicating whether a respective pixel in the target image belongs to the category; a category selection module configured to select at least one category related to the target image from the plurality of categories based on a plurality of category confidences respectively determined for the plurality of categories; and a segmentation map determination module configured to determine a target segmentation map for the target image based on the candidate segmentation map and the category confidence determined for the selected at least one category, the target segmentation map indicating whether a respective pixel in the target image belongs to a category in the at least one category.

[0005] In a third aspect of the disclosure, an electronic device is provided. The device comprises at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by the at least one processing unit, cause the electronic device to perform the method of the first aspect.

[0006] In a fourth aspect of the disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has stored thereon a computer program, which is executable by a processor to implement the method of the first aspect.

[0007] It should be understood that the contents described in this section are not intended to limit the key features or important features of the embodiments of the disclosure, nor are they used to limit the scope of the disclosure. Other features of the disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0008] The above and other features, advantages, and aspects of embodiments of the disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

[0009] Figure 1 A schematic diagram showing an example environment in which embodiments of the disclosure can be implemented is shown;

[0010] Figure 2A , Figure 2B , Figure 2C and Figure 2D schematic diagrams of a light-weight supervised semantic segmentation scheme are shown respectively;

[0011] Figure 2E andFigure 2F The diagrams show the training architecture of the image segmentation model based on the open-class method.

[0012] Figure 2G A schematic diagram of the architecture of the GroupViT model based on the text supervision method is shown;

[0013] Figure 3 A schematic diagram of an architecture for image segmentation according to some embodiments of the present disclosure is shown;

[0014] Figure 4 A schematic diagram of an architecture for image segmentation according to some embodiments of the present disclosure is shown;

[0015] Figure 5 A schematic diagram of model pre-training according to some embodiments of the present disclosure is shown;

[0016] Figure 6 A flowchart of a process for image segmentation according to some embodiments of the present disclosure is shown;

[0017] Figure 7 A schematic structural block diagram of an apparatus for image segmentation according to some embodiments of the present disclosure is shown; and

[0018] Figure 8 A block diagram of an electronic device that may implement one or more embodiments of the present disclosure is shown. Detailed Implementation

[0019] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0020] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the relationship between various data. For example, the above-mentioned relationship can be obtained based on various technical solutions that are currently known and / or will be developed in the future.

[0021] It can be understood that the data involved in the technical solutions of the present disclosure (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the corresponding laws and regulations and relevant provisions.

[0022] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type of personal information involved in the present disclosure, the use range, the use scenario, etc. should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0023] For example, in response to receiving the active request of the user, the prompt information is sent to the user to explicitly prompt the user that the operation requested to be executed will require the acquisition and use of the personal information of the user. Thus, the user can voluntarily choose whether to provide the personal information to the software or hardware such as the electronic device, the application program, the server or the storage medium, etc. that executes the operation of the technical solutions of the present disclosure according to the prompt information.

[0024] As an optional but non-limiting embodiment, in response to receiving the active request of the user, the prompt information is sent to the user in the form of a pop-up window, and the prompt information can be presented in the form of text in the pop-up window. In addition, the pop-up window can also carry selection controls for the user to select "agree" or "disagree" to provide the personal information to the electronic device.

[0025] It can be understood that the above notification and acquisition of user authorization process is only illustrative and does not limit the embodiments of the present disclosure, and other ways that meet the relevant laws and regulations can also be applied to the embodiments of the present disclosure.

[0026] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. The neural network model is an example of a model based on deep learning. In this paper, "model" can also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", which are used interchangeably in this paper.

[0027] A “neural network” is a machine learning network based on deep learning. A neural network is capable of processing an input and providing a corresponding output, which generally includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications generally include many hidden layers, increasing the depth of the network. The layers of a neural network are connected in sequence, such that the output of a previous layer is provided as input to a subsequent layer, with the input layer receiving the input to the neural network and the output of the output layer as the final output of the neural network. Each layer of the neural network includes one or more nodes (also referred to as processing nodes or neurons), each of which processes input from the previous layer.

[0028] Generally, machine learning can include three stages, namely a training stage, a testing stage, and an application stage (also referred to as an inference stage). In the training stage, a given model can be trained using a large amount of training data, iteratively updating parameter values until the model is able to obtain consistent inferences from the training data that satisfy an expected objective. Through training, the model can be considered to have learned an association (also referred to as a mapping) from input to output from the training data. The parameter values of the trained model are determined. In the testing stage, test inputs are applied to the trained model to test whether the model is able to provide correct outputs, thereby determining the performance of the model. In the application stage, the model can be used to process actual inputs based on the parameter values obtained through training to determine corresponding outputs.

[0029] Figure 1 A schematic diagram illustrating an example environment 100 in which embodiments of the present disclosure can be implemented is shown. As shown, the example environment 100 can include an electronic device 110. Figure 1

[0030] In this example environment 100, the electronic device 110 can implement image segmentation through a trained image segmentation model 112. In some embodiments, after the electronic device 110 obtains a target image 101, the electronic device 110 inputs the target image 101 into the image segmentation model 112, which in turn can output an image segmentation result 102 corresponding to the target image 101.

[0031] The target image 101 can be, for example, any image in any format (e.g., JPG format, PNG format, WEBP format, etc.), any size, any color (e.g., color image, black and white image, grayscale image), etc. The image segmentation result 102 can be, for example, but is not limited to, a segmentation map, a text sequence, an attention map, etc. associated with the target image 101.

[0032] ​The image segmentation model 112 may, for example, be any neural network that can perform image segmentation, including but not limited to a fully convolutional network (FCN), a convolutional neural network (CNN), a recurrent neural network (RNN), and the like, embodiments of the present disclosure are not limited in this respect. In some embodiments, the image segmentation model 112 can be stored locally at the electronic device 110, which can directly utilize the local image segmentation model 112 to implement image segmentation when a task associated with image segmentation needs to be performed. In some embodiments, the image segmentation model 112 can also be a model stored in the cloud, which the electronic device 110 can send the target image 101 to the image segmentation model 112 in the cloud and obtain the image segmentation result 102 output by the image segmentation model 112 in the cloud when a task associated with image segmentation needs to be performed.

[0033] The electronic device 110 can be any type of device with computing capability, including a terminal device or a server device. The terminal device can be any type of mobile terminal, fixed terminal, or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a game device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. The server device may, for example, include a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0034] It should be understood that the structure and function of the environment 100 are described for illustrative purposes only, without implying any limitation on the scope of the present disclosure.

[0035] Traditional image segmentation schemes may, for example, include a method based on light supervision semantic segmentation. Light supervision semantic segmentation is to implement image segmentation function using sample data containing a small amount of labeled data (e.g., sample image-text pairs) or using sample data with unlabeled data. Semantic segmentation is a basic task in computer vision, which aims to segment different semantic objects on an image, or in other words, to assign semantic categories to the pixels of the corresponding objects.

[0036] Figures 2A to 2D Schematic diagrams of light supervision semantic segmentation schemes are shown respectively. In Figures 2A to 2D In the above, DB denotes a domain with segmentation labels, DO denotes a domain with open corpus information, and DT denotes a target domain.

[0037] Figure 2AA schematic diagram of a lightweight supervised semantic segmentation scheme of a zero-shot method is shown. The zero-shot method trains an image segmentation model using labeled samples on a source domain DB, and then tests on a target domain DT. Generally, the image segmentation performance of the zero-shot method is poor because the model is difficult to generalize on completely unseen classes.

[0038] Figure 2B A schematic diagram of a lightweight supervised semantic segmentation scheme of an open-vocabulary method is shown. The open-vocabulary method adds a domain DO with open corpus information to the zero-shot method. The corpus information in the DO can assist the image segmentation model in the migration process from the DB to the DT, and make up for more parameter differences.

[0039] Figure 2E and Figure 2F A schematic diagram of a training architecture of an image segmentation model based on the open-vocabulary method is shown. In the architecture 200E shown in Figure 2E In the architecture 200E shown in, after pre-training, a sample image-text pair 201 and a segmentation annotation 202 are provided to a text encoder 250 and an image encoder 260 of the pre-trained image segmentation model. The text encoder 250 and the image encoder 260 together can generate a segmentation map 204 corresponding to the sample image after model fine-tuning 203. In the architecture 200F shown in Figure 2F In the architecture 200F shown in, a sample image-text pair 201 and a segmentation annotation 202 are provided to a text encoder 250 and an image encoder 260 of the image segmentation model. The text encoder 250 and the image encoder 260 can generate a segmentation map 204 corresponding to the sample image after model fine-tuning 203. Therefore, the open-vocabulary method still needs segmentation annotations, and also needs additional model fine-tuning, which can result in poor flexibility of the image segmentation model in different scenarios.

[0040] Figure 2C A schematic diagram of a lightweight supervised semantic segmentation scheme of a weakly supervised method is shown. The weakly supervised method does not need segmentation labels, but needs class labels of images (the class labels of images can be labels in the DO). Generally, it can be considered that the class labels of images completely cover the target domain DT. The image segmentation model can learn to segment the corresponding objects from the images through the class labels of the images. However, the weakly supervised method still needs to obtain class labels, and the labels often need to manually mark the object classes present in the images, which can result in difficulty in expanding the data set for model training, and poor image segmentation performance.

[0041] Figure 2D A schematic diagram of a lightweight supervised semantic segmentation scheme of a Text-Supervised method is shown. The Text-Supervised method is based on pre-training of image-text pairs, and does not require segmentation annotations, and can enable image segmentation models to train image segmentation capabilities in a zero-shot manner.

[0042] Figure 2G A schematic diagram of an architecture of a GroupViT model based on the Text-Supervised method is shown. As shown in the architecture 200G, only sample image-text pairs 201 are provided to a text encoder 250 and a GroupViT 270, which in turn can output segmentation maps 204 corresponding to the sample images. Since the GroupViT model needs to implement image segmentation through its unique non-universal backbone network structure, the GroupViT model has poor flexibility, and since the GroupViT model does not support replacing other backbone structures, the GroupViT model also has poor scalability. Figure 2G

[0043] As mentioned earlier, most image segmentation models need to use sample data containing annotation data for training. In the semantic segmentation task, the annotation data needs to indicate the class of each pixel in the image. Since the annotation data requires a lot of manpower, and it is difficult to obtain annotation data containing all classes, the image segmentation performance of the trained image segmentation model is poor. Text-supervised semantic segmentation provides a new way of thinking for this problem, which supports pre-training of models using image / text pairs, and zero-shot transfer to semantic segmentation, that is, it can enable cross-modal pre-training models to perform unannotated image segmentation. However, the traditional GroupViT model based on the Text-Supervised method has its unique non-universal backbone structure, which cannot implement unannotated image segmentation through a universal image encoder, which makes the GroupViT model less flexible.

[0044] According to an example embodiment of the present disclosure, an improved scheme for image segmentation is provided. According to the scheme, an image feature representation of a target image can be extracted using a trained image encoder, and a plurality of class-specific text feature representations can be generated using a trained text encoder. The image feature representation and the text feature representation are used to determine a plurality of class-specific candidate segmentation maps of the target image and a plurality of class-specific class confidences. The plurality of class-specific class confidences are used to determine at least one class related to the target image. The candidate segmentation map and the class confidence corresponding to the at least one class are used to determine a target segmentation map of the target image. In this way, the image segmentation function can be realized using the image encoder and the text encoder with feature extraction function without the need for annotation data for image segmentation, realizing the flexible application of the model, and ensuring the efficiency and accuracy of image segmentation. ​

[0045] Some example embodiments of the present disclosure will be hereinafter described with continuous reference to the drawings.

[0046] Figure 3 A schematic diagram of an architecture 300 for image segmentation is shown, according to some embodiments of the present disclosure. The architecture 300 can be implemented at the electronic device 110, the image encoder 310 and / or the text encoder 320 in the architecture 300 can be implemented by or with the image segmentation model 112 of the electronic device 110. Figure 1 The architecture 300 will be described with reference to the environment 100. Figure 1 Figure 1 As shown in FIG. 3, the architecture 300 includes an image encoder 310, a text encoder 320, a determination unit 330, and a target segmentation map generation unit 340. In some embodiments, the image encoder 310 can be constructed as a machine learning model or a neural network suitable for processing visual data. The text encoder 320 can be constructed as a machine learning model or a neural network suitable for processing textual data. In some embodiments, the image encoder 310 and / or the text encoder 320 can be respectively implemented based on one or more Transformer blocks or various variants of the Transformer blocks. In addition to the Transformer blocks, one or more of the image encoder 310 and / or the text encoder 320 can be based on other types of models or neural networks, such as a Unet architecture, a SegNet architecture, a fully convolutional network (FCN), a convolutional neural network (CNN), a recurrent neural network (RNN), etc. The specific type of model structure can be selected according to the actual application needs, without any limitation here.

[0047] As shown in FIG. 3, the architecture 300 includes an image encoder 310, a text encoder 320, a determination unit 330, and a target segmentation map generation unit 340. In some embodiments, the image encoder 310 can be constructed as a machine learning model or a neural network suitable for processing visual data. The text encoder 320 can be constructed as a machine learning model or a neural network suitable for processing textual data. In some embodiments, the image encoder 310 and / or the text encoder 320 can be respectively implemented based on one or more Transformer blocks or various variants of the Transformer blocks. In addition to the Transformer blocks, one or more of the image encoder 310 and / or the text encoder 320 can be based on other types of models or neural networks, such as a Unet architecture, a SegNet architecture, a fully convolutional network (FCN), a convolutional neural network (CNN), a recurrent neural network (RNN), etc. The specific type of model structure can be selected according to the actual application needs, without any limitation here. Figure 3

[0048] The image encoder 310 is configured to extract an image feature representation 302 of the target image 301. The feature representation can generally be in the form of a multi-dimensional vector. In this document, the “feature representation” (shortened as “feature”) is also sometimes referred to as an encoded representation, a vector representation, etc. In some embodiments, the image feature representation 302 can characterize the color, texture, shape of objects in the image, and / or other attributes of the image.

[0049] In some embodiments, the image encoder 310 can be a CNN-structure-based image encoder, which can directly output the corresponding image feature representation 302 based on the input target image 301.

[0050] ​​As an alternative, the image encoder 310 can extract a plurality of image features for the target image 301, and obtain the image feature representation 302 by aggregating the extracted image features. In some embodiments, the image encoder 310 can be a transformer-based image encoder, which can extract token-wise image features of the target image 301, where each token corresponds to an image patch, and obtain the image feature representation 302 by token-wise image feature aggregation. The above example embodiments are described in detail below with reference to Figure 4 The above example embodiments are described in detail below with reference to

[0051] As shown in Figure 4 , the image encoder 310 can extract a plurality of image features 401 for a plurality of image patches of the input target image 301. For example, if the size of the target image 301 is LxL (i.e., the target image 301 contains LxL image patches), the transformer-based image encoder 310 can output LxL image features corresponding to the target image 301 based on the target image 301.

[0052] The plurality of image features 401 are provided to the aggregation unit 410, which can aggregate the plurality of image features 401 to obtain the image feature representation 302 of the target image 301. The aggregation unit 410 can employ an aggregation function to aggregate the plurality of image features 401. The aggregation function can be, for example, an average aggregation function (AVG), a maximum aggregation function (MAX), a minimum aggregation function (MIN), a summation aggregation function (SUM), etc. In some embodiments, the commonly used average aggregation function can be used.

[0053] Since an image often contains a large number of background elements, the aggregation result obtained by directly aggregating the features of the image will be dominated by the background elements. To improve the sensitivity of the model to the foreground objects, i.e., to improve the accuracy of the image segmentation result, in some embodiments, the aggregation unit 410 can also arrange the obtained plurality of image features 401 in descending order of value in the spatial dimension. The aggregation unit 410 can only aggregate the plurality of image features 401 arranged in the front to obtain the image feature representation 302.

[0054] Referring back to Figure 3 , the text encoder 320 is configured to generate a text feature representation 304 corresponding to the name of a category. Here, the category is related to the target image 301. In some embodiments, a class set associated with the target image 301 can be obtained in advance, which can contain all categories associated with the target image 301.

[0055] In some embodiments, for each category in the set of categories, the text encoder 320 may acquire a text sequence 303 containing the "category name" and generate a corresponding text feature representation 304 based on the text sequence 303 containing the "category name". The category may be, for example, the category of objects contained in the target image 301, and the text sequence 303 containing the "category name" may be, for example, a text sequence containing the "category name" such as "bicycle", "animal", "shoes", etc. The embodiments of this disclosure are not limited in this respect.

[0056] Regarding the generation of the text sequence 303 containing "the name of the category", in some embodiments, the name of the category can be expanded into a text sequence 303 containing "the name of the category".

[0057] like Figure 4 As shown, at least one category name 402 is provided to the filling unit 420, which can generate at least one text sequence 303 containing the "category name" by filling the category name 402 into at least one prompt word template. Specifically, for each category, the filling unit 420 can fill the category name 402 into at least one prompt word template through prompt engineering to obtain at least one text sequence 303 containing the "category name". Prompt engineering may include, for example, zero-shot prompting, few-shot prompting, chain-of-thought prompting (CoT), zero-shot CoT, self-consistency, generate knowledge prompting, and automatic prompt engineer (APE), etc., and the embodiments of this disclosure are not limited in this respect.

[0058] For example, with the category name 402 being "bicycle", the filling unit 420 can fill "bicycle" into at least one prompt word template to obtain a plurality of text sequences. The plurality of text sequences can be, for example, "picture of a bicycle", "painting of a bicycle", "a bicycle", and so on. In some embodiments, the at least one prompt word template comprises a plurality of different prompt word templates, and thus the at least one text sequence 303 comprising the category name can comprise a plurality of different text sequences, each comprising the category name. In this way, different text sequences can be generated with different prompt word templates, which can achieve diversified language expressions of the category name and help improve the accuracy of feature extraction by the subsequent text encoder 320.

[0059] Referring back to Figure 3 The text feature representation 304 can represent the number, order, and / or other attributes of the words in the text. In some embodiments, to ensure that the image feature representation 302 and the text feature representation 304 can be subsequently processed by the determination unit 330 directly, the image feature representation 302 and the text feature representation 304 are of the same dimension. For example, when the image feature representation 302 is of 1024 dimensions, the text feature representation 304 is also of 1024 dimensions.

[0060] Regarding the specific extraction manner of the text feature representation 304, similar to the image feature representation 302, the text encoder 320 can directly extract the text feature representation 304 of the text sequence, and the text encoder 320 can also be configured to extract a plurality of sequence features of the text sequence. The text feature representation 304 can be obtained by aggregating the plurality of sequence features output by the text encoder 320. For example, the text encoder 320 based on the transformer structure can extract token-by-token text features of the text sequence, each token corresponding to a word. The text feature representation 304 can be obtained by aggregating the token-by-token text features.

[0061] As Figure 4As shown, the text encoder 320 can extract at least one sequence feature 403 for the text sequence 303 containing the name of the category, respectively. Specifically, the multiple text units (e.g., single words or phrases) in the text sequence 303 containing the name of the category can be tokenized to convert into embedded vector representation. For example, a vocabulary can be defined, which includes T text units. In this way, each word in the vocabulary can be converted into a T-dimensional one-hot vector (i.e., only one position is 1 and the rest are 0 in T dimensions). However, a language model (e.g., which can be a fully connected (FC) network) can be learned to map the T-dimensional one-hot vector to a smaller D-dimensional vector (V >> D). In this way, each text unit can be uniquely mapped to a D-dimensional vector. By performing the mapping for each text unit in the text sequence, the text sequence 303 containing the name of the category can be mapped into a feature sequence.

[0062] The feature sequence corresponding to the text sequence 303 containing the name of the category is input to the text encoder 320 for feature extraction to obtain at least one sequence feature 403. The number of the at least one sequence feature 403 is the number of text units in the text sequence 303 containing the name of the category. For example, if there are T prompt words for each category, the text sequence 303 containing the name of the category can be mapped into a feature sequence, and the text encoder 320 based on the transformer structure can output T text features (i.e., sequence features 403) corresponding to the feature sequence corresponding to the text sequence 303 containing the name of the category.

[0063] The at least one sequence feature 403 is provided to the aggregation unit 430, which can aggregate the at least one sequence feature 403 to obtain the text feature representation 304 of the category. Similar to the aggregation unit 410, the aggregation unit 430 can employ an aggregation function to aggregate the at least one sequence feature 403. In some embodiments, the aggregation unit 430 can also arrange the obtained at least one sequence feature 403 in descending order of value in the spatial dimension. The aggregation unit 430 can only aggregate the at least one sequence feature 403 ranked in the front to obtain the text feature representation 304 of the category.

[0064] It can be understood that, although Figure 4 Although two aggregation units (the aggregation unit 410 and the aggregation unit 430) are shown in FIG. 4, and the two aggregation units are respectively used to aggregate image features or sequence features, in some embodiments, only one aggregation unit can be included, which can aggregate the input features, and embodiments of the present disclosure are not limited in this respect.

[0065] Referring back to FIG. 4, Figure 3The image feature representation 302 and the text feature representation 304 are provided to a determination unit 330. For each category in the set of categories, the determination unit 330 is configured to determine, based on the image feature representation 302 and the text feature representation 304 corresponding to the category, a candidate segmentation map 305 for the target image 301 and a category confidence 306 for the category, the candidate segmentation map indicating whether a respective pixel in the target image belongs to the category. In some embodiments, the determination unit 330 can be implemented based on one or more Transformer blocks or various variants of Transformer blocks. In addition to Transformer blocks, the determination unit 330 can be based on other types of models or neural networks, such as SENet architecture, Residual Attention Network (RAN), BAM architecture, CBAM architecture, etc. The specific type of model structure can be selected according to actual application needs.

[0066] As to the specific manner of determining the candidate segmentation map 305, in some embodiments, the determination unit 330 can directly obtain the candidate segmentation map 305 for the target image 301 based on the image feature representation 302 and the text feature representation 304 corresponding to the category. In some embodiments, after obtaining the image feature representation 302 and the text feature representation 304 corresponding to the category, the determination unit 330 can determine an attention map based on the two, the attention map indicating a plurality of correlation degrees between the category and a plurality of image blocks in the target image 301. The determination unit 330 can generate the candidate segmentation map 305 by processing the attention map.

[0067] Taking an example in which the determination unit 330 includes one or more Transformer blocks, the determination unit 330 can define the text feature representation 304 as a query feature input to each Transformer block, and define the image feature representation 302 as a key feature and a value feature input to each Transformer block. The processing of the Transformer block can be represented as follows:

[0068]

[0069] where Q represents the query feature, K represents the key feature, V represents the value feature, d k represents the number of columns of Q and K, i.e., the feature dimension. The above processing can be understood as that a self-attention weight matrix is calculated using the query feature Q and the key feature K, the Softmax is used to normalize the matrix to obtain a normalized self-attention weight matrix, and the value feature V is weighted and summed using the normalized self-attention weight matrix to obtain the attention weight Attention.

[0070] The self-attention weights are calculated by the determination unit 330. An attention map can be determined based on these self-attention weights, and a candidate segmentation map 305 can be generated by processing the attention map.

[0071] In some embodiments, where the image encoder 310 is configured to extract multiple image features for multiple image patches of the target image 301, the determining unit 330 may also be configured to determine an attention map based on the text feature representation 304 and the multiple image features. Reference continues below. Figure 4 Describe such an example implementation.

[0072] like Figure 4 As shown, the determining unit 330 may include an attention map determining unit 440 and a candidate segmentation map generating unit 450, which are respectively used to perform the subtask of determining the attention map and the subtask of generating the candidate segmentation map 305. In some embodiments, image features 401 and text feature representations 304 corresponding to the category are provided together to the attention map determining unit 440, and the attention map determining unit 440 is configured to determine the attention map 404 based on the text feature representations 304 and multiple image features 401.

[0073] Specifically, in an example where there are LxL image features 401, the attention map determination unit 440 can calculate the attention map 404 by combining the text feature representation 304 with the LxL image features. Specific calculation methods may include performing inner products or other appropriate aggregation operations on the text feature representation 304 and the LxL image features respectively. As an example, the attention map determination unit 440 can perform an inner product on the text feature representation 304 and the LxL image features to obtain an attention map 404 of size LxL.

[0074] Furthermore, attention map 404 is provided to candidate segmentation map generation unit 450, which can generate candidate segmentation map 305 by processing attention map 404. Specifically, candidate segmentation map generation unit 450 can upsample attention map 404 to a size corresponding to the target image 301, obtaining upsampled attention map 404. Upsampling can be implemented by specific methods, such as, but not limited to, bilinear interpolation, transposed convolution, upsampling, up pooling, and subpixel convolution, etc.

[0075] The image resolution of the attention map 404 after up-sampling is the same size as the image resolution of the target image 301. The candidate segmentation map generation unit 450 can further generate the candidate segmentation map 305 by applying a conditional random field (CRF) process to the up-sampled attention map 404. The conditional random field (CRF) can include, for example, a dense conditional random field (DenseCRF), a linear-chain conditional random field (linear-CRF), a Gaussian conditional random field (G-CRF), a Markov random field (MRF), and the like. The candidate segmentation map generation unit 450 can optimize the attention map 404 by the CRF, determine the class corresponding to each pixel in the image, and output the candidate segmentation map 305. In some embodiments, the candidate segmentation map 305 for a single class can be a 0-1 binary distribution map, in which 0 can indicate that the pixel does not conform to this class, and 1 can indicate that the pixel conforms to this class. It should be understood that the meanings represented by 0 and 1 above are only exemplary, and in other embodiments according to the present disclosure, 1 can also be used to indicate that the pixel does not conform to this class, and 0 can be used to indicate that the pixel conforms to this class.

[0076] In some embodiments, the determination unit 330 can further include a confidence determination unit 460 configured to perform a subtask of determining the class confidence 306 of the class. Specifically, the confidence determination unit 460 can perform a calculation on the image feature representation 302 and the text feature representation 304 corresponding to the class, and the result of the calculation can indicate the class confidence 306 of the class. The specific calculation method can include, for example, inner product or other appropriate aggregation operations, etc. on the image feature representation 302 and the text feature representation 304 corresponding to the class. For example, the confidence determination unit 460 can perform an inner product on the image feature representation 302 and the text feature representation 304 corresponding to the class to obtain the class confidence 306 corresponding to the class.

[0077] With reference to the foregoing Figure 3 The candidate segmentation maps 305 and the class confidences 306 corresponding to the plurality of classes are provided to the target segmentation map generation unit 340. The target segmentation map generation unit 340 is configured to select at least one class related to the target image 301 from the plurality of classes based on the plurality of class confidences 306 determined for the plurality of classes, respectively. The target segmentation map generation unit 340 is further configured to determine a target segmentation map 307 for the target image 301 based on the candidate segmentation map 305 and the class confidence 306 determined for the selected at least one class, the target segmentation map 307 indicating whether a corresponding pixel in the target image 301 belongs to a class in the at least one class.

[0078] In some embodiments, the target segmentation map generation unit 340 can select a first number of classes from the plurality of classes based on the plurality of class confidences 306 respectively determined for the plurality of classes. Specifically, the target segmentation map generation unit 340 can sort the plurality of class confidences 306, for example, in descending order. The target segmentation map generation unit 340 can then select the classes corresponding to the first number of class confidences 306 with the highest ranks from the plurality of class confidences 306. As an example, if there are N classes in total, the target segmentation map generation unit 340 can select the top M classes from the N classes, where M can be, for example, N / 2. In this way, the target segmentation map generation unit 340 can filter out the N-M classes with the lowest ranks, which tend to contain a large amount of noise, and thus can reduce the noise interference on the image segmentation, helping to improve the accuracy of the target segmentation map generated subsequently and improve the effect of the image segmentation.

[0079] Further, the target segmentation map generation unit 340 can determine a threshold confidence based on the first number of class confidences 306 respectively corresponding to the first number of classes. Specifically, the target segmentation map generation unit 340 can calculate the first number of class confidences 306, and the calculation result is the threshold confidence. The specific calculation method can include, for example, summing, averaging, calculating the standard deviation, calculating the variance, etc. of the first number of class confidences 306, and can also include any combination of the foregoing. In some embodiments, the target segmentation map generation unit 340 can determine the threshold confidence based on the mean (μ) and the standard deviation (σ) of the first number of class confidences 306. As an example, the threshold confidence can be determined as the sum of the mean and the standard deviation (μ+σ).

[0080] The target segmentation map generation unit 340 can select at least one class from the first number of classes whose corresponding class confidence 306 exceeds the threshold confidence. For example, the target segmentation map generation unit 340 can select at least one class from the M classes whose corresponding class confidence 306 exceeds the threshold confidence (e.g., μ+σ).

[0081] In some embodiments, the at least one class selected by the target segmentation map generation unit 340 comprises at least two classes. The target segmentation map generation unit 340 can determine, for each pixel in the target image 301, a target class of the at least two classes to which the pixel belongs based on the at least two class confidences 306 corresponding to the at least two classes. Specifically, the target segmentation map generation unit 340 can stack the at least two candidate segmentation maps 305 corresponding to the at least two selected classes, and perform an Argmax operation on each pixel to select the class with the maximum confidence for each pixel to obtain the target segmentation map 307. The Argmax operation is to multiply the at least two candidate segmentation maps 305 by their respective class confidences 306, and the result with the maximum value is the class confidence of the target class corresponding to each pixel. Thus, the target segmentation map generation unit 340 can determine the target segmentation map 307 for the target image 301 based on the plurality of candidate segmentation maps 305 and the plurality of class confidences 306 corresponding to the plurality of classes.

[0082] According to the above scheme, the target segmentation map 307 generated by the target segmentation map generation unit 340 based on the 0-1 binary segmentation maps determined for the individual classes can still be a multi-value segmentation map. For example, if the 0-1 binary segmentation maps of three classes are finally selected for stacking, the value 0 can be used to indicate the first class, the value 1 can be used to indicate the second class, and the value 3 can be used to indicate the third class in the target segmentation map 307. In some embodiments, the target segmentation map 307 can also include the value 4, which can indicate the background of the image. It should be understood that the above values 0-4 and their meanings are only exemplary, and in other embodiments according to the present disclosure, different classes can also be represented by other values.

[0083] It should also be understood that Figure 3 and Figure 4 The images, features, sequences, and related diagrams given in the above description are for the purpose of explanation and illustration only, and do not have any limitation on the embodiments of the present disclosure.

[0084] The above description Figure 3 and Figure 4 describes an example of image segmentation using a trained text encoder and an image encoder. The specific training examples of the text encoder and the image encoder will be further described below.

[0085] In some embodiments, the text encoder 320 and the image encoder 310 can be pre-trained to let both learn the relationship between the text sequence and the image, which helps to ensure the accuracy of the subsequent extraction of the text feature representation and the image feature representation using the trained text encoder and image encoder. In some embodiments, to ensure the flexibility of the text encoder and the image encoder in various image processing scenarios, the image encoder and the text encoder are trained based on unannotated training data, which includes a large number of sample image-text pairs. Based on the unannotated training data, the image encoder and the text encoder can learn how to extract the features of the image modal data and the features of the text modal data based on contrastive learning.

[0086] The sample image in the training data is provided to the image encoder, and the image encoder determines the corresponding sample image feature representation based on the sample image. The sample text sequence in the training data is provided to the text encoder, and the text encoder can determine the corresponding sample text feature representation based on the sample text sequence.

[0087] In some embodiments, the sample image feature representation and the sample text feature representation can be directly output by the image encoder and the text encoder, or can be obtained through other processing. For example, if the image encoder and the text encoder are both based on the transformer structure, the image encoder can output the corresponding image feature based on the input sample image, and the sample image feature representation can be obtained by aggregating the image feature. The text encoder can output the corresponding sequence feature based on the input sample text sequence, and the sample text feature representation can be obtained by aggregating the sequence feature.

[0088] Further, the loss function can be calculated for all sample data, and the parameters of the image encoder and the text encoder can be adjusted based on the calculation result of the loss function. The loss function can be, for example, L2 loss function, L1 loss function, Smooth L1 loss function, huber loss function, softmax loss function, etc. In some embodiments, the contrastive learning loss (InfoNCE) between the sample image feature representation and the sample text feature representation can be calculated, and the learning goal of the image encoder and the text encoder is to increase the similarity of the positive sample pair and reduce the similarity of the negative sample pair.

[0089] Taking the image encoder and the text encoder in the CLIP model as an example, the training data can be directly input into the model, and the model can update the parameters of the image encoder and the text encoder based on the calculation result of the loss function. Figure 5 A schematic diagram of model pre-training according to some embodiments of the present disclosure is shown.

[0090] AsFigure 5 As shown, for a training data containing N sample image-text pairs, the image encoder 310 can generate N image feature representations 503 based on the training images 501, and the text encoder 320 can generate N text feature representations 504 based on the training text sequences 502. The model can combine the N image feature representations 503 and the N text feature representations 504 two by two and predict the similarity of N 2 possible sample image-text pairs. Here, the similarity directly calculates the cosine similarity of the text features and the image features, that is Figure 5 the matrix shown. Here, there are N positive samples, that is, the text and image that really belong to a pair (the diagonal elements in the matrix), and the remaining N 2 -N sample image-text pairs are negative samples.

[0091] In summary, the pre-trained image encoder and the text encoder are used for feature extraction in image segmentation. In this way, the image segmentation function can be realized by using the image encoder and the text encoder with feature extraction function without the need for labeled data for image segmentation, realizing flexible application of the model, and ensuring the efficiency and accuracy of image segmentation.

[0092] Figure 6 A flowchart of a process 600 for image segmentation according to some embodiments of the present disclosure is shown. The process 600 may, for example, be implemented at the electronic device 110. For ease of discussion, the process 600 will be described with reference to the environment 100. Figure 1

[0093] At block 610, the electronic device 110 extracts an image feature representation of a target image using the trained image encoder.

[0094] At block 620, the electronic device 110, for each of a plurality of categories, generates a text feature representation corresponding to a name of the category using the trained text encoder, determines a candidate segmentation map for the target image and a category confidence of the category based on the image feature representation and the text feature representation, the candidate segmentation map indicating whether a corresponding pixel in the target image belongs to the category.

[0095] In some embodiments, generating the text feature representation for each of the plurality of categories includes: generating at least one text sequence containing the name of the category; extracting at least one sequence feature of the at least one text sequence using the text encoder, respectively; and generating the text feature representation by aggregating the at least one sequence feature.

[0096] In some embodiments, the at least one text sequence includes a plurality of different text sequences, each text sequence including the name of the category. ​

[0097] In some embodiments, generating the at least one text sequence comprises: generating the at least one text sequence by respectively filling in names of the categories into the at least one prompt word template.

[0098] In some embodiments, determining the candidate segmentation map for each category of the plurality of categories comprises: determining an attention map based on the text feature representation and the image feature representation, the attention map indicating a plurality of relevancies between the category and a plurality of image patches in the target image; and generating the candidate segmentation map by processing the attention map.

[0099] In some embodiments, generating the candidate segmentation map by processing the attention map comprises: up-sampling the attention map to a size corresponding to the target image to obtain an up-sampled attention map; and generating the candidate segmentation map by applying a conditional random field (CRF) process to the up-sampled attention map.

[0100] In some embodiments, extracting the image feature representation comprises: extracting, with an image encoder, a plurality of image features for a plurality of image patches of the target image; and determining the image feature representation by aggregating the plurality of image features, and wherein determining the attention map comprises: determining the attention map based on the text feature representation and the plurality of image features.

[0101] In some embodiments, the image encoder and the text encoder are trained based on unannotated training data, the training data comprising sample image-text pairs.

[0102] At block 630, the electronic device 110 selects, from the plurality of categories, at least one category that is relevant to the target image based on the plurality of category confidences respectively determined for the plurality of categories.

[0103] In some embodiments, selecting the at least one category comprises: selecting, from the plurality of categories, a first number of categories based on the plurality of category confidences respectively determined for the plurality of categories; determining a threshold confidence based on a first number of category confidences respectively corresponding to the first number of categories; and selecting, from the first number of categories, at least one category whose corresponding category confidence exceeds the threshold confidence.

[0104] In some embodiments, determining the threshold confidence comprises: determining the threshold confidence based on a mean and a standard deviation of the first number of category confidences.

[0105] At block 640, the electronic device 110 determines a target segmentation map for the target image based on the candidate segmentation map and the category confidence respectively determined for the selected at least one category, the target segmentation map indicating whether a respective pixel in the target image belongs to a category of the at least one category.

[0106] In some embodiments, the selected at least one category includes at least two categories, and wherein determining the target segmentation map includes: for each pixel in the target image, determining, based on at least two category confidences corresponding to the at least two categories, a target category of the at least two categories to which the pixel belongs.

[0107] Figure 7 A schematic structural block diagram of an apparatus 700 for image segmentation according to some embodiments of the present disclosure is shown. The apparatus 700 can be implemented as or included in the electronic device 110. Various modules / components in the apparatus 700 can be implemented by hardware, software, firmware, or any combination thereof.

[0108] As shown, the apparatus 700 includes an image feature extraction module 710 configured to extract, by using a trained image encoder, an image feature representation of a target image. The apparatus 700 further includes a feature processing module 720 configured to, for each category of a plurality of categories, generate, by using a trained text encoder, a text feature representation corresponding to a name of the category, determine, based on the image feature representation and the text feature representation, a candidate segmentation map for the target image and a category confidence for the category, the candidate segmentation map indicating whether a respective pixel in the target image belongs to the category. The apparatus 700 further includes a category selection module 730 configured to select, from the plurality of categories, at least one category related to the target image based on a plurality of category confidences respectively determined for the plurality of categories. The apparatus 700 further includes a segmentation map determination module 740 configured to determine, based on the candidate segmentation map and the category confidence determined for the selected at least one category, a target segmentation map for the target image, the target segmentation map indicating whether a respective pixel in the target image belongs to a category of the at least one category.

[0109] In some embodiments, the feature processing module 720 includes a text sequence generation module configured to generate at least one text sequence containing the name of the category, a sequence feature extraction module configured to extract, by using the text encoder, at least one sequence feature of the at least one text sequence respectively, and a text feature generation module configured to generate the text feature representation by aggregating the at least one sequence feature.

[0110] In some embodiments, the at least one text sequence includes a plurality of different text sequences, each text sequence including the name of the category.

[0111] In some embodiments, the text sequence generation module is further configured to generate the at least one text sequence by respectively filling the name of the category into at least one prompt word template.

[0112] In some embodiments, the segmentation map determination module 740 includes an attention map determination module configured to determine, based on the text feature representation and the image feature representation, an attention map indicating a plurality of relevancies between the class and a plurality of image patches in the target image, and a segmentation map generation module configured to generate the candidate segmentation map by processing the attention map.

[0113] In some embodiments, the segmentation map generation module is further configured to upsample the attention map to a size corresponding to the target image to obtain an upsampled attention map, and generate the candidate segmentation map by applying a conditional random field (CRF) process to the upsampled attention map.

[0114] In some embodiments, the image feature extraction module 710 is further configured to extract, using the image encoder, a plurality of image features for a plurality of image patches of the target image, and determine the image feature representation by aggregating the plurality of image features. In these embodiments, the attention map determination module is further configured to determine the attention map based on the text feature representation and the plurality of image features.

[0115] In some embodiments, the class selection module 730 is further configured to select, from the plurality of classes, a first number of classes based on a plurality of class confidences respectively determined for the plurality of classes, determine a threshold confidence based on a first number of class confidences respectively corresponding to the first number of classes, and select, from the first number of classes, at least one class corresponding to a class confidence exceeding the threshold confidence.

[0116] In some embodiments, the class selection module 730 is further configured to determine the threshold confidence based on a mean and a standard deviation of the first number of class confidences.

[0117] In some embodiments, the selected at least one class includes at least two classes, and the segmentation map determination module 740 is further configured to determine, for each pixel in the target image, a target class to which the pixel belongs among the at least two classes based on at least two class confidences respectively corresponding to the at least two classes.

[0118] In some embodiments, the image encoder and the text encoder are trained based on unlabeled training data, the training data including a sample image-text pair.

[0119] Figure 8 A block diagram of an electronic device 800 is shown that can implement one or more embodiments of the disclosure. It should be understood that Figure 8 The electronic device 800 shown is merely exemplary and should not be construed as limiting the scope of the embodiments described herein. Figure 8 The electronic device 800 shown can be used to implement the electronic device 110 of Figure 1 the embodiments described herein.

[0120] like Figure 8 As shown, electronic device 800 is in the form of a general-purpose computing device. Components of electronic device 800 may include, but are not limited to, one or more processors or processing units 810, memory 820, storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. Processing unit 810 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 820. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 800.

[0121] Electronic device 800 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 820 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 830 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 800.

[0122] Electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 8 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 820 may include computer program product 825 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0123] The communication unit 840 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 800 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 800 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0124] Input device 850 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. Output device 860 can be one or more output devices, such as a display, a speaker, a printer, etc. Electronic device 800 can also communicate with one or more external devices (not shown), such as a storage device, a display device, etc., through communication unit 840, as desired, in order to communicate with a user in order to interact with electronic device 800, or to communicate with any device (e.g., a network card, a modem, etc.) that enables electronic device 800 to communicate with one or more other electronic devices. Such communication can be carried out via an input / output (I / O) interface (not shown).

[0125] According to an example embodiment of the present disclosure, there is provided a computer-readable storage medium having stored thereon computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an example embodiment of the present disclosure, there is also provided a computer program product tangibly stored on a non-transitory computer-readable medium and including computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method described above.

[0126] Various aspects of the disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0127] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0128] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0129] The computer program product of the present disclosure can be a computer program product, which is a machine-readable medium (media) having instances of the software embodied thereon, such as computer software, firmware, wireless application protocol (WAP), middleware or microcode. For example, a computer program product can be a floppy disk, a CD-ROM, a DVD, a Blu-ray Disc™, a flash drive, a memory stick, a magnetic tape, or a hard disk drive. The machine-readable medium can be a single medium, or multiple media, of the same or different type. The computer program product can be one or more computer program components embodied in medium and / or transmission signals. The computer program product can have one or more computer readable and / or computer executable components embodied in medium and / or transmission signals. The computer program product can be one or more computer readable and / or computer executable components embodied in medium and / or transmission signals.

[0130] The implementations of the disclosure have been described above with the intent to be illustrative rather than limiting. Although the implementations of the disclosure have been described with regard to one or more implementations, it will be recognized that various modifications and changes can be made by those skilled in the art that will achieve the desirable results of the implementations, such as comparable technical effects. Accordingly, the disclosure is not to be limited by what has been presented in the description above, but instead can be defined by the appended claims and their equivalents.

Claims

1. A method for image segmentation, comprising: Extract image feature representations of the target image using a trained image encoder; For each of the multiple categories The trained text encoder is used to generate text feature representations corresponding to the names of the categories. Based on the image feature representation and the text feature representation, a candidate segmentation map and a category confidence score for the target image are determined, wherein the candidate segmentation map indicates whether the corresponding pixel in the target image belongs to the category; Based on the multiple category confidence levels determined for each of the multiple categories, at least one category related to the target image is selected from the multiple categories; as well as A target segmentation map is determined for the target image based on candidate segmentation maps and category confidence scores for at least one selected category, the target segmentation map indicating whether a corresponding pixel in the target image belongs to a category among the at least one category.

2. The method of claim 1, wherein generating the text feature representation for each of the plurality of categories comprises: Generate at least one text sequence containing the name of the category; The text encoder is used to extract at least one sequence feature from each of the at least one text sequence; as well as The text feature representation is generated by aggregating the at least one sequence feature.

3. The method of claim 2, wherein the at least one text sequence comprises a plurality of different text sequences, each text sequence comprising the name of the category.

4. The method of claim 2, wherein generating the at least one text sequence comprises: The at least one text sequence is generated by filling the category names into at least one prompt word template.

5. The method of claim 1, wherein determining the candidate segmentation map for each of the plurality of categories comprises: Based on the text feature representation and the image feature representation, an attention map is determined, which indicates multiple relevances between the category and multiple image patches in the target image; as well as The candidate segmentation map is generated by processing the attention map.

6. The method of claim 5, wherein generating the candidate segmentation map by processing the attention map comprises: The attention map is upsampled to a size corresponding to the target image to obtain an upsampled attention map; as well as The candidate segmentation map is generated by applying a Conditional Random Field (CRF) process to the upsampled attention map.

7. The method of claim 5, wherein extracting the image feature representation comprises: The image encoder is used to extract multiple image features from the multiple image patches of the target image; as well as The image feature representation is determined by aggregating the multiple image features, and Determining the attention map includes: determining the attention map based on the text feature representation and the plurality of image features.

8. The method of claim 1, wherein selecting the at least one category comprises: Based on the multiple category confidence levels determined for each of the multiple categories, a first number of categories are selected from the multiple categories; Based on the category confidence scores corresponding to the first number of categories, a threshold confidence score is determined. as well as Select at least one category from the first number of categories whose category confidence exceeds the threshold confidence level.

9. The method of claim 8, wherein determining the threshold confidence level comprises: The threshold confidence level is determined based on the mean and standard deviation of the confidence levels for the first number of categories.

10. The method of claim 1, wherein the selected at least one category comprises at least two categories, and wherein determining the target segmentation map comprises: For each pixel in the target image, based on the confidence scores of at least two categories corresponding to the at least two categories, it is determined that the pixel belongs to the target category among the at least two categories.

11. The method of claim 1, wherein the image encoder and the text encoder are trained based on unlabeled training data, the training data comprising sample image-text pairs.

12. An apparatus for image segmentation, comprising: The image feature extraction module is configured to extract image feature representations of the target image using a trained image encoder; The feature processing module is configured to generate a text feature representation corresponding to the name of the category using a trained text encoder for each of a plurality of categories, and to determine a candidate segmentation map and a category confidence for the target image based on the image feature representation and the text feature representation, wherein the candidate segmentation map indicates whether the corresponding pixel in the target image belongs to the category. The category selection module is configured to select at least one category related to the target image from the plurality of categories based on a plurality of category confidences determined for each of the plurality of categories. as well as The segmentation map determination module is configured to determine a target segmentation map for the target image based on candidate segmentation maps determined for at least one selected category and category confidence, the target segmentation map indicating whether a corresponding pixel in the target image belongs to a category among the at least one category.

13. The apparatus of claim 12, wherein the feature processing module comprises: A text sequence generation module is configured to generate at least one text sequence containing the name of the category; The sequence feature extraction module is configured to extract at least one sequence feature from the at least one text sequence using the text encoder. as well as The text feature generation module is configured to generate the text feature representation by aggregating the at least one sequence feature.

14. The apparatus of claim 13, wherein the at least one text sequence comprises a plurality of different text sequences, each text sequence comprising the name of the category.

15. The apparatus of claim 13, wherein the text sequence generation module is further configured to: The at least one text sequence is generated by filling the category names into at least one prompt word template.

16. The apparatus of claim 12, wherein the segmentation map determination module comprises: An attention map determination module is configured to determine an attention map based on the text feature representation and the image feature representation, the attention map indicating multiple relevances between the category and multiple image patches in the target image; as well as The segmentation map generation module is configured to generate the candidate segmentation map by processing the attention map.

17. The apparatus of claim 16, wherein the segmentation map generation module is further configured to: The attention map is upsampled to a size corresponding to the target image to obtain an upsampled attention map; and The candidate segmentation map is generated by applying a Conditional Random Field (CRF) process to the upsampled attention map.

18. The apparatus of claim 16, wherein the image feature extraction module is further configured to: The image encoder is used to extract multiple image features from the multiple image patches of the target image; and The image feature representation is determined by aggregating the multiple image features, and The attention map determination module is further configured to determine the attention map based on the text feature representation and the plurality of image features.

19. An electronic device comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 11.

20. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images