Image segmentation method, device, storage medium and electronic device

Abstract

The present application discloses an image segmentation method, device, storage medium and electronic device, the method comprising: obtaining at least one reference image and a mask image of the reference image; inputting the reference image and the mask image of the reference image into a first neural network for processing, obtaining reference classification features for indicating the visual content classification information and reference image features for indicating the feature information of the visual content at various positions in the image; inputting a query image, the reference classification features and the reference image features into a second neural network, and under the guidance of the reference image features, performing target segmentation on the query image to obtain at least one target segmentation result, and determining the confidence of each target segmentation result based on the reference classification features; based on the confidence, determining the final segmentation result of the image segmentation and outputting it. By using the present application, image segmentation can be performed based on user needs, and system performance can be effectively improved.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to an image segmentation method, a storage medium and an electronic device. Background Art

[0002] With the advancement of neural network technology and image processing technology, intelligent systems based on image processing have been increasingly widely used.

[0003] In most intelligent systems based on image processing, images need to be segmented to separate the desired target area. In current segmentation processing applications, it is hoped that targeted image segmentation can be performed based on the visual content specified by the user. Among the existing segmentation processing methods, some solutions are based on interactive calibration tools to achieve targeted image segmentation, some solutions use text prompts to describe user needs to perform image segmentation, and some solutions are based on the open source Segment Everything (SAM) model to perform image segmentation and then screen the desired targets one by one. However, the image segmentation implemented by these solutions has the problems of requiring full manual participation, or the user's needs cannot provide suitable text, or the visual content obtained by segmentation is too fragmented and inefficient, and it is impossible to obtain satisfactory segmentation results, making the performance of the image segmentation system unable to meet user needs. Summary of the invention

[0004] The present application provides an image segmentation method, a device storage medium and an electronic device, which can perform image segmentation based on user needs and effectively improve system performance.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] An image segmentation method, characterized by comprising:

[0007] Acquire at least one reference image and a mask image of the reference image, wherein the mask image is used to mark visual content that needs to be segmented;

[0008] Inputting the reference image and the mask image of the reference image into a first neural network for processing, to obtain reference classification features for indicating classification information of the visual content and reference image features for indicating feature information of the visual content at various positions in the image;

[0009] Inputting the query image, the reference classification feature and the reference image feature into a second neural network, performing target segmentation on the query image under the guidance of the reference image feature to obtain at least one target segmentation result, and determining the confidence of each target segmentation result based on the reference classification feature;

[0010] Based on the confidence level, a final segmentation result of the image segmentation is determined and outputted.

[0011] Preferably, the step of inputting the reference image and the mask image of the reference image into a first neural network for processing comprises:

[0012] In the first neural network,

[0013] Using a first reference feature extraction module to extract image features from the reference image to obtain primary image features;

[0014] Based on the guidance of the mask image of the reference image, using a second reference feature extraction module to perform secondary feature extraction on the primary image features to obtain the reference image features;

[0015] The reference image features and the primary image features are subjected to embedding layer processing using a reference embedding layer module to obtain the reference classification features.

[0016] Preferably, the performing object segmentation on the query image comprises:

[0017] In the second neural network,

[0018] Using a first query feature extraction module to extract image features from the query image to obtain primary query image features and high-resolution query image features;

[0019] Based on the reference image feature, using a second query feature extraction module to perform secondary feature extraction on the primary query image feature to obtain at least one conditional query image feature;

[0020] The high-resolution query image features and the conditional query image features are convolved to obtain a response map of the visual specified content corresponding to each of the conditional query image features as a target segmentation result.

[0021] Preferably, the determining the confidence of each target segmentation result based on the reference classification feature includes:

[0022] In the second neural network, using a query embedding layer module, embedding layer processing is performed on the primary query image feature and each of the conditional query image features to obtain a query classification feature corresponding to each of the conditional query image features;

[0023] For each of the query classification features, the similarity between it and the reference classification feature is calculated as the confidence of the target segmentation result corresponding to the query classification feature; wherein the target segmentation result corresponding to the query classification feature is the target segmentation result corresponding to the conditional query image feature corresponding to the query classification feature.

[0024] Preferably, the determining and outputting the final segmentation result of the image segmentation comprises:

[0025] For instance segmentation requirements, the target segmentation result corresponding to the query classification feature whose confidence is greater than the set confidence threshold is selected and output as the final segmentation result of the image segmentation;

[0026] And / or, for semantic segmentation requirements, each confidence level is multiplied by the corresponding target segmentation result, and then the multiplication results are summed up for each image element to obtain a summed result; when the summed result is greater than a set summed threshold, the summed result is output as the final segmentation result of the image segmentation.

[0027] Preferably, the performing secondary feature extraction on the primary image features by using the second reference feature extraction module comprises:

[0028] The preset reference vector, the primary image features and the mask image of the reference image are input into the second reference feature extraction module, and are processed by multiple cascaded Transformer layers in the second reference feature extraction module to perform secondary feature extraction.

[0029] Preferably, the performing secondary feature extraction on the primary query image features by using the second query feature extraction module comprises:

[0030] After the preset N query vectors are spliced ​​with the reference image features, they are input into the second query feature extraction module together with the primary query image features, and are processed by multiple cascaded Transformer layers in the second query feature extraction module to perform secondary feature extraction to obtain N conditional query image features; wherein N is 1 or the maximum number of targets output under instance segmentation requirements.

[0031] Preferably, the reference vector and the query vector are determined in advance by jointly training the first neural network and the second neural network.

[0032] Preferably, when there are multiple reference images, the method further comprises:

[0033] Averaging the multiple reference image features obtained by the second reference feature extraction module as updated reference image features, and inputting the updated reference image features into the second neural network;

[0034] The multiple reference classification features obtained by the reference embedding layer module are averaged as updated reference classification features and input into the second neural network.

[0035] Preferably, when there are multiple reference images,

[0036] Performing the splicing of the N query vectors with reference image features determined corresponding to all reference images;

[0037] When calculating the similarity, the similarity between the query classification feature and the reference classification feature determined corresponding to each reference image is calculated, and the maximum value between the similarities is selected as the confidence of the target segmentation result corresponding to the query classification feature.

[0038] An image segmentation device, comprising: a reference image acquisition unit, a first neural network unit, a second neural network unit and a post-processing output unit;

[0039] The reference image acquisition unit is used to acquire at least one reference image and a mask image of the reference image, wherein the mask image is used to mark visual content that needs to be segmented;

[0040] The first neural network unit is used to input the reference image and the mask image of the reference image into a first neural network for processing, so as to obtain reference classification features for indicating the classification information of the visual content and reference image features for indicating feature information of the visual content at various positions in the image;

[0041] The second neural network unit is used to input the query image, the reference classification feature and the reference image feature into a second neural network, perform target segmentation on the query image under the guidance of the reference image feature, obtain at least one target segmentation result, and determine the confidence of each target segmentation result based on the reference classification feature;

[0042] The post-processing output unit is used to determine and output a final segmentation result of the image segmentation based on the confidence level.

[0043] Preferably, in the first neural network unit, inputting the reference image and the mask image of the reference image into the first neural network for processing comprises:

[0044] In the first neural network,

[0045] Using a first reference feature extraction module to extract image features from the reference image to obtain primary image features;

[0046] Based on the guidance of the mask image of the reference image, using a second reference feature extraction module to perform secondary feature extraction on the primary image features to obtain the reference image features;

[0047] The reference image features and the primary image features are subjected to embedding layer processing using a reference embedding layer module to obtain the reference classification features.

[0048] Preferably, in the second neural network unit, the performing object segmentation on the query image comprises:

[0049] In the second neural network,

[0050] Using a first query feature extraction module to extract image features from the query image to obtain primary query image features and high-resolution query image features;

[0051] Based on the reference image feature, using a second query feature extraction module to perform secondary feature extraction on the primary query image feature to obtain at least one conditional query image feature;

[0052] The high-resolution query image features and the conditional query image features are convolved to obtain a response map of the visual specified content corresponding to each of the conditional query image features as a target segmentation result.

[0053] Preferably, in the second neural network, determining the confidence of each target segmentation result based on the reference classification feature includes:

[0054] In the second neural network, using a query embedding layer module, embedding layer processing is performed on the primary query image feature and each of the conditional query image features to obtain a query classification feature corresponding to each of the conditional query image features;

[0055] For each of the query classification features, the similarity between it and the reference classification feature is calculated as the confidence of the target segmentation result corresponding to the query classification feature; wherein the target segmentation result corresponding to the query classification feature is the target segmentation result corresponding to the conditional query image feature corresponding to the query classification feature.

[0056] Preferably, in the post-processing output unit, the determining and outputting of the final segmentation result of the image segmentation comprises:

[0057] For instance segmentation requirements, the target segmentation result corresponding to the query classification feature whose confidence is greater than the set confidence threshold is selected and output as the final segmentation result of the image segmentation;

[0058] And / or, for semantic segmentation requirements, each confidence level is multiplied by the corresponding target segmentation result, and then the multiplication results are summed up for each image element to obtain a summed result; when the summed result is greater than a set summed threshold, the summed result is output as the final segmentation result of the image segmentation.

[0059] Preferably, in the first neural network unit, the second reference feature extraction module is used to perform secondary feature extraction on the primary image features, comprising:

[0060] The preset reference vector, the primary image features and the mask image of the reference image are input into the second reference feature extraction module, and are processed by multiple cascaded Transformer layers in the second reference feature extraction module to perform secondary feature extraction.

[0061] Preferably, in the second neural network unit, the second query feature extraction module is used to perform secondary feature extraction on the primary query image features, comprising:

[0062] After the preset N query vectors are spliced ​​with the reference image features, they are input into the second query feature extraction module together with the primary query image features, and are processed by multiple cascaded Transformer layers in the second query feature extraction module to perform secondary feature extraction to obtain N conditional query image features; wherein N is 1 or the maximum number of targets output under instance segmentation requirements.

[0063] Preferably, when there are multiple reference images, the first neural network unit is further used to average the multiple reference image features obtained by the second reference feature extraction module as updated reference image features, and input them into the second neural network; and is also used to average the multiple reference classification features obtained by the reference embedding layer module as updated reference classification features, and input them into the second neural network.

[0064] Preferably, when there are multiple reference images,

[0065] In the second neural network unit, the N query vectors are spliced ​​with the reference image features determined corresponding to all reference images; when calculating the similarity, the similarity between the query classification features and the reference classification features determined corresponding to each reference image is calculated, and the maximum value between the similarities is selected as the confidence of the target segmentation result corresponding to the query classification feature.

[0066] A computer-readable storage medium having computer instructions stored thereon, characterized in that when the instructions are executed by a processor, any of the above-mentioned image segmentation methods can be implemented.

[0067] An electronic device comprising at least a computer-readable storage medium and a processor;

[0068] The processor is used to read executable instructions from the computer-readable storage medium and execute the instructions to implement any one of the above-mentioned image segmentation methods.

[0069] As can be seen from the above technical solution, in this application, firstly, a reference image and a mask image of the reference image are obtained, and the visual content to be segmented on the reference image is marked by the mask image; the reference image and the mask image of the reference image are input into the first neural network for processing, and reference classification features for indicating the classification information of the visual content and reference image features for indicating the feature information of the visual content at each position in the image are obtained, so as to characterize the reference image and the feature information of the segmented visual content by using the reference classification features and the reference image features. Next, the query image, the reference classification features and the reference image features are input into the second neural network, and in the second neural network, based on the guidance of the reference image features, the query image is segmented to obtain at least one target segmentation result that is consistent with the specified visual content to be segmented in the reference image at the image feature level, and the confidence of each target segmentation result is determined based on the reference classification features, so as to mark the credibility of the target segmentation result from the classification level of the visual content to be segmented on the reference image. Finally, based on the confidence, the final segmentation result of the image segmentation is determined and output. Through the above processing, the target category that needs to be segmented can be marked in the form of a reference image, and the query image can be segmented based on the guidance of the reference image to obtain a segmentation result consistent with the target category marked by the reference image, which can effectively improve the performance of image segmentation and meet user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 Schematic diagram of the basic process of the image segmentation method in this application;

[0071] Figure 2 is an example of a reference image and its mask image;

[0072] Figure 3 This is a schematic diagram of a specific process of the image segmentation method in a specific embodiment of the present application;

[0073] Figure 4 This is a schematic diagram of the architecture of the first neural network in a specific embodiment of the present application;

[0074] Figure 5 This is a schematic diagram of the architecture of the second neural network in a specific embodiment of the present application;

[0075] Figure 6 is another example of a reference image and its mask image;

[0076] Figure 7 for Figure 6 Schematic diagram of semantic segmentation results obtained based on the reference image and mask image shown;

[0077] Figure 8 for Figure 6 Schematic diagram of instance segmentation results obtained based on the reference image and mask image shown;

[0078] Fig. 9 Schematic diagram of the basic structure of the image segmentation device in this application;

[0079] Fig.10 It is a schematic diagram of the basic structure of the electronic device in this application. DETAILED DESCRIPTION

[0080] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.

[0081] The basic idea of ​​this application is to mark the visual content that needs to be segmented in the form of an image, and perform segmentation processing of the query image under the guidance of a reference image, thereby effectively improving the performance of image segmentation, especially for batch image segmentation, which can effectively improve the segmentation efficiency.

[0082] Figure 1 FIG. 1 is a schematic diagram of the basic flow of the image segmentation method in this application. Figure 1 As shown, the method includes:

[0083] Step 101: Obtain at least one reference image and a mask image of the reference image.

[0084] The mask image is used to mark the visual content that needs to be segmented. It can usually be a binary mask image. For example, if a user needs to obtain a segmentation image of a bird in a batch of images, then the mask image can be obtained as follows: Figure 2 As shown, a reference image including a bird is provided, and the bird as a target is marked in the reference image to obtain a corresponding mask image.

[0085] There may be multiple reference images used as segmentation guides. For example, the postures of birds in multiple different reference images are different. In this way, when performing object segmentation on the query image, birds in various postures can be effectively segmented.

[0086] Step 102, input the reference image and the mask image of the reference image into the first neural network for processing, and obtain reference classification features for indicating visual content classification information and reference image features for indicating feature information of visual content at various positions in the image.

[0087] The first neural network is a pre-trained neural network, which is used to process the reference image and its mask image, and obtain the feature information of the visual content indicated by the mask image, specifically including classification features and image features. In order to distinguish it from the subsequent processing process of the query image, the classification features obtained by the first neural network are referred to as reference classification features, and the image features obtained by the first neural network are referred to as reference image features, which are used to indicate the feature information of the segmented visual content at each position in the image.

[0088] Through the processing of this step, the visual content that needs to be segmented in the reference image can be described from the classification and image feature information of each pixel position.

[0089] Step 103, input the query image, reference classification features and reference image features into the second neural network, perform target segmentation on the query image under the guidance of the reference image features, obtain at least one target segmentation result, and determine the confidence of each target segmentation result based on the reference classification features.

[0090] The second neural network is used to perform segmentation processing on the query image, and the target segmentation is performed based on the reference image features determined in step 102, so as to segment the target consistent with the visual content features marked by the reference image as much as possible. At the same time, for each target segmentation result, its confidence is determined based on the reference classification feature, that is, the credibility of the corresponding target segmentation result. Usually, the classification feature corresponding to each target segmentation result can be determined, and the classification feature is compared with the reference classification feature. The closer the two are, the higher the credibility of the corresponding target segmentation result, that is, the higher the confidence.

[0091] Step 104: Based on the confidence level, determine the final segmentation result of the image segmentation and output it.

[0092] The target segmentation result and the corresponding confidence are determined through step 103, and the confidence is determined based on the reference classification features. Therefore, the confidence can reflect the degree to which each target segmentation result matches the reference classification features. Then, the final image segmentation result can be determined based on these target segmentation results and their confidence.

[0093] So far, Figure 1 The method flow shown ends. Through the above process, on the one hand, the reference image and its mask image are used to mark the visual content that needs to be segmented, which is more universal than the description in text form, and can still effectively represent the visual content that is not easy to describe in text; on the other hand, the reference image is used to guide the query image to perform segmentation processing, so that the query image can be segmented in accordance with the reference image prompt, specifically including primary target segmentation based on the reference image features and target segmentation result selection based on the reference classification features, so that the target segmentation result consistent with the segmented visual content prompted by the reference image can be finally obtained, effectively improving the performance of image segmentation, and the processing process does not require full human participation, especially for batch image segmentation, which can greatly improve the segmentation efficiency.

[0094] The specific implementation of this application is described below through specific embodiments.

[0095] Figure 3FIG. 1 is a schematic diagram of a specific process of an image segmentation method in a specific embodiment of the present application. A reference image is used as an example for explanation. Figure 3 As shown, the method includes:

[0096] Step 301: Obtain a reference image and a mask image of the reference image.

[0097] A reference image input by a user can be received, and a mask image of the visual content to be segmented can be determined. In this embodiment, it is assumed that the reference image and its mask image are as follows: Figure 2 As shown, the visual content that needs to be segmented is "bird".

[0098] The specific processing of this step can be implemented in the existing way, and will not be repeated here.

[0099] Next, steps 302 to 304 provide a process of processing the reference image using the pre-trained first neural network, such as Figure 4 shown.

[0100] Step 302: Use a first reference feature extraction module to extract image features from a reference image to obtain primary image features.

[0101] like Figure 4 As shown, the first neural network includes a first reference feature extraction module, a second reference feature extraction module and a reference embedding layer module. The processing of this step is performed in the first reference feature extraction module, which is used to extract image features from the input reference image. This part can be as follows Figure 4 This is achieved by using several cascaded Transformer layers. Of course, other image feature extraction structures can also be used to achieve this.

[0102] Step 303 : Based on the guidance of the mask image of the reference image, a second reference feature extraction module is used to perform secondary feature extraction on the primary image features to obtain reference image features.

[0103] The image features of the entire reference image are obtained through the aforementioned step 302, in which the visual content to be segmented is not particularly focused; in order to specifically describe the visual content A to be segmented, this step further performs secondary feature extraction on the primary image features under the guidance of the mask image to generate image features that can focus on the visual content A for the query image segmentation guidance (hereinafter referred to as reference image features). Generally, several cascaded Transformer layer processes can be used to achieve the feature extraction required above. Of course, other feature extraction structures can also be used in practical applications, as long as feature extraction can be achieved under the guidance of the mask image.

[0104] In this embodiment, the secondary feature extraction process for the primary image features can be as follows: Figure 4 As shown in the second reference feature extraction module in, it specifically includes:

[0105] a. Initialize the reference vector Suppot vector;

[0106] As mentioned above, both the first neural network and the second neural network are neural networks and need to be trained in advance; usually, the first neural network and the second neural network can be jointly trained.

[0107] The reference vector initialized is Figure 4 The gray rectangle in the image. In the actual processing flow of query image segmentation, that is, the reasoning process of the first neural network and the second neural network, the initialized reference vector can be pre-set, usually determined through the training process of the neural network. Accordingly, in the neural network training process, the initialized reference vector can be set to a fixed value or a random value; or, the reference vector Suppot vector initialized in the reasoning process can also be related to the reference image, then correspondingly, in the neural network training process, the initialized reference vector is also related to the reference image. For example, the initialized reference vector can be obtained by performing an average pooling operation after feature extraction on the foreground part of the mask image of the reference image. The initialized support vector under this processing method is more targeted to the visual content A.

[0108] b. Inputting the initialized reference vector, the primary image features and the mask image of the reference image into the second reference feature extraction module for processing to obtain the reference image features;

[0109] The second reference feature extraction module may be composed of several cascaded Transformer layer processings. Since the initial input of these Transformer layers includes a mask image, image features focusing on the visual content A indicated by the mask image may be obtained based on the multi-level Transformer layer processing. The specific structure for performing secondary image feature extraction may adopt a structure of several cascaded Transformer layers as described above, and the more cascaded Transformer layers there are, the deeper and more universal image features may be obtained. Of course, in practical applications, other feature extraction structures that can introduce a mask image may also be adopted, and this application does not limit this.

[0110] After processing a and b, the reference image features can be obtained, that is, the updated Suppot vector, such as Figure 4 The small black rectangle in the

[0111] Step 304: Use a reference embedding layer module to perform embedding layer processing on the reference image features and the primary image features to obtain reference classification features.

[0112] The input of the reference embedding layer module includes reference image features and primary image features. After being processed by the embedding layer, feature information used to classify the visual content A is obtained, which is referred to as reference classification features below. Figure 4 The small grid rectangle in the .

[0113] After the processing of the aforementioned steps 302 to 304, on the one hand, the reference image features are determined by the second reference feature extraction module, which are used to describe the image features of the reference image at the image pixel level, especially focusing on the image feature information of the visual content A at the image pixel level; on the other hand, the reference classification information is determined by the reference embedding layer module, which is used to describe the feature information for classifying the reference image at the level of overall image classification, especially focusing on the feature information of the visual content A at the image classification level.

[0114] Next, steps 305 to 309 are used to describe the process of segmenting the query image using the pre-trained second neural network. Figure 5 When performing query image segmentation, the visual information of the reference image is directly combined, which is conducive to making full use of the features of the reference image and making the segmentation result more consistent with the visual content specified in the reference image.

[0115] Step 305 : Use the first query feature extraction module to extract image features from the query image to obtain primary query image features and high-resolution query image features.

[0116] The query image is the target image that needs to be segmented. This step is used to extract image features from the query image, and can be implemented using various existing image feature extraction structures. In this embodiment, the structure of the first query feature extraction module is the same as the first reference feature extraction module. In order to perform image segmentation more finely, in the image feature extraction process, the high-resolution query image features are extracted separately. The specific high-resolution standard can be pre-set, for example, it can be 1 / 4 of the basic resolution.

[0117] Step 306: Based on the reference image features, a second query feature extraction module is used to perform secondary feature extraction on the primary query image features to obtain at least one conditional query image feature.

[0118] This step performs secondary feature extraction on the primary query image features based on the reference image features obtained in the aforementioned reference image processing. Since the reference image features reflect the pixel-level feature information of the reference image, especially the visual content A, secondary feature extraction based on the reference image features can obtain image features consistent with the features of the visual content A in the query image, i.e., the query image features. The conditional query image features obtained after the secondary feature extraction are one or more, where the number of conditional query image features is the maximum number of segmentation targets that can be obtained in the instance segmentation requirements.

[0119] In this embodiment, the process of performing secondary feature extraction on the primary query image features can be as follows: Figure 5 As shown in the second query feature extraction module in, it specifically includes:

[0120] A. Initialize at least one query vector;

[0121] The number of query vectors can be one or more. Usually, for semantic segmentation applications, one query vector can be used; and for instance segmentation and semantic segmentation applications, multiple query vectors can be used. The number of query vectors usually indicates the maximum number of segmentation targets that can be obtained in instance segmentation.

[0122] During the inference process, the query vector initialized is obtained in advance through the joint training process of the first neural network and the second neural network; during the training process, the initialized query vector can be a specified special value (such as all zeros) or can be randomly initialized.

[0123] B. After concatenating the query vector with the reference image features, the query vector is input into the second query feature extraction module together with the primary query image features. After being processed by multiple cascaded Transformer layers in the second query feature extraction module, secondary feature extraction is performed to obtain N conditional query image features.

[0124] In this embodiment, the second query feature extraction module adopts the same structure as the first query feature extraction module, that is, it is composed of several cascaded Transformer layers. Figure 5 The gray rectangle in the figure) and the reference image features (such as Figure 5If there are multiple reference images, as mentioned above, the reference image features corresponding to the multiple reference images can be averaged as the updated reference image features, and the query vector and the updated reference image features are spliced ​​in this step; or, if there are multiple reference images, the reference image features corresponding to each reference image can be input into the second neural network, and the query vector and all the reference image features are spliced ​​together and input into the second query feature extraction module in this step. After the secondary feature extraction process, N conditional query image features (such as Figure 5 The small rectangle composed of medium gray and black). Since the reference image features are introduced, the obtained conditional query image features can reflect the feature information consistent with the visual content A.

[0125] Step 307: In the image segmentation module, the high-resolution query image features and the conditional query image features are convolved to obtain a response map of the visual specified content corresponding to each conditional query image feature as a target segmentation result.

[0126] In this step, the high-resolution query image features and the conditional query image features are convolved, and the convolution result is used to represent the position response map of each conditional query image feature for the visual content A. The number of response maps obtained here is N, and each response map corresponds to a conditional query image feature Conditional Query vector, which is actually a target segmentation result. The target segmentation result here gives the segmentation result whose features are consistent with the reference image features, but this segmentation result may also be inaccurate and needs to be screened through subsequent processing.

[0127] Step 308: Use the query embedding layer module to perform embedding layer processing on the primary query image feature and each conditional query image feature to obtain a query classification feature corresponding to each conditional query image feature.

[0128] In this step, when embedding layer processing is performed, any conditional query image feature and the query image feature are embedded in the layer to obtain the query classification feature corresponding to the conditional query image feature (for example Figure 5 ), is used to represent the feature information related to the specified visual content A classification under the corresponding conditional query image feature (that is, the target segmentation result corresponding to the conditional query image feature). In this embodiment, the query embedding layer module can use the same network architecture as the reference embedding layer module, or other network architectures for embedding layer processing can be used, as long as the embedding layer processing can be implemented.

[0129] Step 309 : For each query classification feature, calculate the similarity between it and the reference classification feature as the confidence of the target segmentation result corresponding to the query classification feature.

[0130] Since when segmenting the query image, the requirement is to segment the visual content A specified by the reference image, the target segmentation result corresponding to each conditional query image feature obtained in step 307, if its classification feature is closer to the classification feature of the visual content A, then the credibility of the target segmentation result is higher. Based on this, in this step, the query classification feature corresponding to each conditional query image feature is calculated with the reference classification feature for similarity, and the similarity calculation result is used as the confidence of the target segmentation result corresponding to the corresponding conditional query image feature. Here, a correspondence is also established between the target segmentation result corresponding to the conditional query image feature and the query classification feature corresponding to the conditional query image feature, that is, the target segmentation result corresponding to the query classification feature.

[0131] In addition, if there are multiple reference images and the reference classification features corresponding to each reference image are directly input into the second neural network, then when this step determines the confidence, for any query classification feature B, the similarity is calculated with each reference classification feature respectively, and then multiple similarity results corresponding to the query classification feature B can be obtained. These multiple similarity results can be statistically analyzed to obtain a statistical analysis result as the confidence of the target segmentation result corresponding to the query classification feature B. The specific statistical analysis result can be, for example, the maximum value of all similarity results.

[0132] The processing of steps 308 to 309 can be performed at Figure 5 The classification header module implementation shown.

[0133] As described above, through steps 305 to 309, multiple target segmentation results and the confidence levels of the target segmentation results can be obtained. Figure 5 Only one target segmentation result is given as an example. So far, the processing of the second neural network is completed.

[0134] Step 310: Based on the confidence level, determine the final segmentation result of the image segmentation and output it.

[0135] In this embodiment, corresponding methods for determining the final segmentation result are provided according to different segmentation requirements.

[0136] For semantic segmentation, each confidence level is multiplied by the corresponding target segmentation result, and then the multiplication results are summed up for each image element to obtain the summed result; when the summed result is greater than the set summed threshold, the summed result is output as the final segmentation result of the image segmentation. In the semantic segmentation mode, a segmented target is finally obtained. For example, given Figure 6The reference image and its mask shown in the figure require the strawberries in the image to be segmented out. The semantic segmentation results are as follows Figure 7 As shown, all strawberries are segmented as one target.

[0137] For instance segmentation requirements, the target segmentation result with a confidence level greater than a set confidence level threshold (e.g., 0.5) is selected as the final segmentation result of the image segmentation. Figure 6 The reference image and its mask shown in the figure require the strawberries in the image to be segmented out. The example segmentation result is as follows Figure 8 As shown, each strawberry is segmented as a separate object.

[0138] At this point, the image segmentation method process in the specific embodiment ends. The above embodiment is completed on the basis of the pre-trained first neural network and the second neural network, and the training process of the first neural network and the second neural network is similar to the above reasoning process, specifically including:

[0139] A', inputting the acquired reference image and its mask image into the first neural network, and after being processed by the first neural network, inputting the obtained reference image features and reference classification features into the second neural network;

[0140] B' and the second neural network obtain various target segmentation results and their confidences of this training after processing, compare these target segmentation results and their confidences with the standard comparison samples, so as to determine the loss function of this training, and adjust the parameters of the first neural network and the second neural network accordingly; return to step A' for the next training until the training end condition is met, that is, the trained first neural network and the second neural network can be obtained.

[0141] In the above single training, a single reference image or multiple reference images can be input. The processing of multiple reference images is the same as the above Figure 3 The processing in the shown flow is the same.

[0142] In the above-mentioned image segmentation method of the present application, a complete set of unified methods is used to directly couple the user-specified visual content to perform segmentation prediction during the segmentation process, and it can support the input of multiple reference images and multiple query images, and is suitable for batch calibration of user-specified visual content. The segmentation results are more in line with user expectations, and instance-level segmentation annotation and semantic segmentation annotation are supported. At the same time, the present application directly implements the segmentation of the image to be segmented based on the visual content of the reference image, avoiding the problem of difficult naming of visual content when segmentation is based on text prompts. The visual content information of the reference image is used in the segmentation process, so that the segmentation result is the desired purpose, saving a lot of manpower costs. Furthermore, the present application combines the visual content of the reference image, supports the use of only a single reference image, and also supports the use of multiple reference images. The use of multiple reference images can better capture the characteristics of the required segmentation visual content, and usually better segmentation results can be obtained, which can improve the performance of the image segmentation method from multiple aspects and angles.

[0143] The above is the specific implementation of the image segmentation method in this application. This application also provides an image segmentation device, which can be used to implement the above image segmentation method. Fig. 9 FIG. 1 is a schematic diagram of the basic structure of the image segmentation device in this application. Fig. 9 As shown, the device includes: a reference image acquisition unit, a first neural network unit, a second neural network unit and a post-processing output unit.

[0144] Wherein, the reference image acquisition unit is used to acquire at least one reference image and a mask image of the reference image, wherein the mask image is used to mark the visual content to be segmented;

[0145] A first neural network unit is used to input the reference image and the mask image of the reference image into the first neural network for processing, so as to obtain reference classification features for indicating visual content classification information and reference image features for indicating feature information of the visual content at various positions in the image;

[0146] A second neural network unit is used to input the query image, the reference classification feature and the reference image feature into the second neural network, and under the guidance of the reference image feature, perform target segmentation on the query image to obtain at least one target segmentation result, and determine the confidence of each target segmentation result based on the reference classification feature;

[0147] The post-processing output unit is used to determine the final segmentation result of the image segmentation based on the confidence level and output it.

[0148] Optionally, in the first neural network unit, the reference image and the mask image of the reference image are input into the first neural network for processing, which may specifically include:

[0149] In the first neural network,

[0150] Using a first reference feature extraction module to extract image features from the reference image to obtain primary image features;

[0151] Based on the guidance of the mask image of the reference image, a second reference feature extraction module is used to perform secondary feature extraction on the primary image features to obtain reference image features;

[0152] The reference embedding layer module is used to perform embedding layer processing on the reference image features and the first-level image features to obtain the reference classification features.

[0153] Optionally, in the second neural network unit, the processing of object segmentation on the query image may specifically include:

[0154] In the second neural network,

[0155] Using the first query feature extraction module to extract image features from the query image, to obtain primary query image features and high-resolution query image features;

[0156] Based on the reference image features, using a second query feature extraction module to perform secondary feature extraction on the primary query image features to obtain at least one conditional query image feature;

[0157] The high-resolution query image features and the conditional query image features are convolved to obtain a response map of the visually specified content corresponding to each conditional query image feature as the target segmentation result.

[0158] Optionally, in the second neural network, the process of determining the confidence of each target segmentation result based on the reference classification feature may specifically include:

[0159] In the second neural network, the query embedding layer module is used to perform embedding layer processing on the primary query image feature and each conditional query image feature to obtain a query classification feature corresponding to each conditional query image feature;

[0160] For each query classification feature, the similarity between it and the reference classification feature is calculated as the confidence of the target segmentation result corresponding to the query classification feature; wherein the target segmentation result corresponding to the query classification feature is the target segmentation result corresponding to the conditional query image feature corresponding to the query classification feature.

[0161] Optionally, in the post-processing output unit, the final segmentation result of the image segmentation is determined to be output, which may specifically include:

[0162] For instance segmentation requirements, the target segmentation result corresponding to the query classification feature with a confidence greater than the set confidence threshold is selected and output as the final segmentation result of the image segmentation;

[0163] And / or, for semantic segmentation requirements, each confidence level is multiplied by the corresponding target segmentation result, and then the multiplication results are summed up image element by image element to obtain a summed result; when the summed result is greater than a set summed threshold, the summed result is output as the final segmentation result of the image segmentation.

[0164] Optionally, in the first neural network unit, a second reference feature extraction module is used to perform a secondary feature extraction process on the primary image features, specifically including:

[0165] The preset reference vector, the primary image feature and the mask image of the reference image are input into the second reference feature extraction module, and are processed by multiple cascaded Transformer layers in the second reference feature extraction module to perform secondary feature extraction.

[0166] Optionally, in the second neural network unit, a second query feature extraction module is used to perform a secondary feature extraction process on the primary query image feature, which may specifically include:

[0167] After the preset N query vectors are concatenated with the reference image features, they are input into the second query feature extraction module together with the primary query image features. After being processed by multiple cascaded Transformer layers in the second query feature extraction module, secondary feature extraction is performed to obtain N conditional query image features; where N is 1 or the maximum number of targets output under instance segmentation requirements.

[0168] Optionally, when there are multiple reference images, the first neural network unit can be further used to average the multiple reference image features obtained by the second reference feature extraction module as updated reference image features, and input them into the second neural network; and can also be used to average the multiple reference classification features obtained by the reference embedding layer module as updated reference classification features, and input them into the second neural network.

[0169] Optionally, the reference vector and the query vector may be determined in advance by jointly training the first neural network and the second neural network.

[0170] Optionally, when there are multiple reference images,

[0171] In the second neural network unit, N query vectors are concatenated with reference image features corresponding to all reference images; when calculating the similarity, the similarity between the query classification features and the reference classification features corresponding to each reference image is calculated, and the maximum value between the similarities is selected as the confidence of the target segmentation result corresponding to the query classification feature.

[0172] The present application also provides a computer-readable storage medium, which stores instructions, and when the instructions are executed by a processor, the steps in the image segmentation method described above can be executed. In practical applications, the computer-readable medium can be included in each device / apparatus / system in the above embodiments, or it can exist independently without being assembled into the device / apparatus / system. Instructions are stored in the computer-readable storage medium, and when the instructions stored in the computer-readable storage medium are executed by a processor, the steps in the image segmentation method described above can be executed.

[0173] According to the embodiments disclosed in the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof, but it is not used to limit the scope of protection of the present application. In the embodiments disclosed in the present application, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0174] Fig.10 This application also provides an electronic device. Fig.10 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0175] The electronic device may include a processor 1001 with one or more processing cores, a memory 1002 with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When the program in the memory 1002 is executed, the image segmentation method may be implemented.

[0176] Specifically, in practical applications, the electronic device may further include components such as a power supply 1003, an input / output unit 1004, etc. Those skilled in the art may understand that Fig.10 The structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0177] Processor 1001 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the server and processes data by running or executing software programs and / or modules stored in memory 1002 and calling data stored in memory 1002, thereby monitoring the electronic device as a whole.

[0178] The memory 1002 can be used to store software programs and modules, that is, the above-mentioned computer-readable storage medium. The processor 1001 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002. The memory 1002 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 1002 may also include a memory controller to provide the processor 1001 with access to the memory 1002.

[0179] The electronic device also includes a power supply 1003 for supplying power to various components, which can be logically connected to the processor 1001 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 1003 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0180] The electronic device may also include an input and output unit 1004, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, and optical signal inputs related to user settings and function control. The input unit output 1004 may also be used to display information input by the user or information provided to the user and various graphical user interfaces, which may be composed of graphics, text, icons, videos, and any combination thereof.

[0181] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image segmentation method, characterized in that: include: Acquire at least one reference image and a mask image of the reference image, wherein the mask image is used to mark visual content that needs to be segmented; Inputting the reference image and the mask image of the reference image into a first neural network for processing, to obtain reference classification features for indicating classification information of the visual content and reference image features for indicating feature information of the visual content at various positions in the image; Inputting the query image, the reference classification feature and the reference image feature into a second neural network, performing target segmentation on the query image under the guidance of the reference image feature to obtain at least one target segmentation result, and determining the confidence of each target segmentation result based on the reference classification feature; Based on the confidence level, determining a final segmentation result of the image segmentation for output; Wherein, performing object segmentation on the query image includes: In the second neural network, Using a first query feature extraction module to extract image features from the query image to obtain primary query image features and high-resolution query image features; Based on the reference image feature, using a second query feature extraction module to perform secondary feature extraction on the primary query image feature to obtain at least one conditional query image feature; The high-resolution query image features and the conditional query image features are convolved to obtain a response map of the visual specified content corresponding to each of the conditional query image features as a target segmentation result.

2. The method according to claim 1, characterized in that The step of inputting the reference image and the mask image of the reference image into a first neural network for processing comprises: In the first neural network, Using a first reference feature extraction module to extract image features from the reference image to obtain primary image features; Based on the guidance of the mask image of the reference image, using a second reference feature extraction module to perform secondary feature extraction on the primary image features to obtain the reference image features; The reference image features and the primary image features are subjected to embedding layer processing using a reference embedding layer module to obtain the reference classification features.

3. The method according to claim 1, characterized in that The step of determining the confidence level of each target segmentation result based on the reference classification feature comprises: In the second neural network, using a query embedding layer module, embedding layer processing is performed on the primary query image feature and each of the conditional query image features to obtain a query classification feature corresponding to each of the conditional query image features; For each of the query classification features, the similarity between it and the reference classification feature is calculated as the confidence of the target segmentation result corresponding to the query classification feature; wherein the target segmentation result corresponding to the query classification feature is the target segmentation result corresponding to the conditional query image feature corresponding to the query classification feature.

4. The method according to claim 3, characterized in that The step of determining the final segmentation result of the image segmentation and outputting it comprises: For instance segmentation requirements, the target segmentation result corresponding to the query classification feature whose confidence is greater than the set confidence threshold is selected and output as the final segmentation result of the image segmentation; And / or, for semantic segmentation requirements, each confidence level is multiplied by the corresponding target segmentation result, and then the multiplication results are summed up for each image element to obtain a summed result; when the summed result is greater than a set summed threshold, the summed result is output as the final segmentation result of the image segmentation.

5. The method according to claim 2, characterized in that: The using the second reference feature extraction module to perform secondary feature extraction on the primary image features includes: The preset reference vector, the primary image features and the mask image of the reference image are input into the second reference feature extraction module, and are processed by multiple cascaded Transformer layers in the second reference feature extraction module to perform secondary feature extraction.

6. The method according to claim 3, characterized in that The using the second query feature extraction module to perform secondary feature extraction on the primary query image feature includes: After the preset N query vectors are spliced ​​with the reference image features, they are input into the second query feature extraction module together with the primary query image features, and are processed by multiple cascaded Transformer layers in the second query feature extraction module to perform secondary feature extraction to obtain N conditional query image features; wherein N is 1 or the maximum number of targets output under instance segmentation requirements.

7. The method according to claim 5, characterized in that The reference vector and the query vector are determined in advance by jointly training the first neural network and the second neural network.

8. The method according to claim 2, characterized in that: When there are multiple reference images, the method further comprises: Averaging the multiple reference image features obtained by the second reference feature extraction module as updated reference image features, and inputting the updated reference image features into the second neural network; The multiple reference classification features obtained by the reference embedding layer module are averaged as updated reference classification features and input into the second neural network.

9. The method according to claim 6, characterized in that When there are multiple reference images, Performing the splicing of the N query vectors with reference image features determined corresponding to all reference images; When calculating the similarity, the similarity between the query classification feature and the reference classification feature determined corresponding to each reference image is calculated, and the maximum value between the similarities is selected as the confidence of the target segmentation result corresponding to the query classification feature.

10. An image segmentation device, characterized in that: include: A reference image acquisition unit, a first neural network unit, a second neural network unit and a post-processing output unit; The reference image acquisition unit is used to acquire at least one reference image and a mask image of the reference image, wherein the mask image is used to mark visual content that needs to be segmented; The first neural network unit is used to input the reference image and the mask image of the reference image into a first neural network for processing, so as to obtain reference classification features for indicating the classification information of the visual content and reference image features for indicating feature information of the visual content at various positions in the image; The second neural network unit is used to input the query image, the reference classification feature and the reference image feature into a second neural network, perform target segmentation on the query image under the guidance of the reference image feature, obtain at least one target segmentation result, and determine the confidence of each target segmentation result based on the reference classification feature; The post-processing output unit is used to determine and output a final segmentation result of the image segmentation based on the confidence level; Wherein, in the second neural network unit, performing target segmentation on the query image includes: In the second neural network, Using a first query feature extraction module to extract image features from the query image to obtain primary query image features and high-resolution query image features; Based on the reference image feature, using a second query feature extraction module to perform secondary feature extraction on the primary query image feature to obtain at least one conditional query image feature; The high-resolution query image features and the conditional query image features are convolved to obtain a response map of the visual specified content corresponding to each of the conditional query image features as a target segmentation result.

11. The device according to claim 10, characterized in that In the first neural network unit, inputting the reference image and the mask image of the reference image into the first neural network for processing includes: In the first neural network, Using a first reference feature extraction module to extract image features from the reference image to obtain primary image features; Based on the guidance of the mask image of the reference image, using a second reference feature extraction module to perform secondary feature extraction on the primary image features to obtain the reference image features; The reference image features and the primary image features are subjected to embedding layer processing using a reference embedding layer module to obtain the reference classification features.

12. The device according to claim 10, characterized in that In the second neural network, determining the confidence of each target segmentation result based on the reference classification feature includes: In the second neural network, using a query embedding layer module, embedding layer processing is performed on the primary query image feature and each of the conditional query image features to obtain a query classification feature corresponding to each of the conditional query image features; For each of the query classification features, the similarity between it and the reference classification feature is calculated as the confidence of the target segmentation result corresponding to the query classification feature; wherein the target segmentation result corresponding to the query classification feature is the target segmentation result corresponding to the conditional query image feature corresponding to the query classification feature.

13. The device according to claim 12, characterized in that In the post-processing output unit, the determining and outputting of the final segmentation result of the image segmentation includes: For instance segmentation requirements, the target segmentation result corresponding to the query classification feature whose confidence is greater than the set confidence threshold is selected and output as the final segmentation result of the image segmentation; And / or, for semantic segmentation requirements, each confidence level is multiplied by the corresponding target segmentation result, and then the multiplication results are summed up for each image element to obtain a summed result; when the summed result is greater than a set summed threshold, the summed result is output as the final segmentation result of the image segmentation.

14. The device according to claim 11, characterized in that In the first neural network unit, the second reference feature extraction module is used to perform secondary feature extraction on the primary image features, including: The preset reference vector, the primary image features and the mask image of the reference image are input into the second reference feature extraction module, and are processed by multiple cascaded Transformer layers in the second reference feature extraction module to perform secondary feature extraction.

15. The device according to claim 12, characterized in that In the second neural network unit, the second query feature extraction module is used to perform secondary feature extraction on the primary query image feature, including: After the preset N query vectors are spliced ​​with the reference image features, they are input into the second query feature extraction module together with the primary query image features, and are processed by multiple cascaded Transformer layers in the second query feature extraction module to perform secondary feature extraction to obtain N conditional query image features; wherein N is 1 or the maximum number of targets output under instance segmentation requirements.

16. The device according to claim 11, characterized in that When there are multiple reference images, the first neural network unit is further used to average the multiple reference image features obtained by the second reference feature extraction module as updated reference image features, and input them into the second neural network; and is also used to average the multiple reference classification features obtained by the reference embedding layer module as updated reference classification features, and input them into the second neural network.

17. The device according to claim 15, characterized in that When there are multiple reference images, In the second neural network unit, the N query vectors are spliced ​​with the reference image features determined corresponding to all reference images; when calculating the similarity, the similarity between the query classification features and the reference classification features determined corresponding to each reference image is calculated, and the maximum value between the similarities is selected as the confidence of the target segmentation result corresponding to the query classification feature.

18. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by a processor, the image segmentation method described in any one of claims 1 to 9 can be implemented.

19. An electronic device, characterized in that: The electronic device includes at least a computer-readable storage medium and also includes a processor; The processor is used to read executable instructions from the computer-readable storage medium and execute the instructions to implement the image segmentation method according to any one of claims 1 to 9.

Images