Image instance segmentation method, device and medium

Abstract

The present invention discloses an image instance segmentation method, device and medium, which utilize an alignment query learning module to generate an alignment query based on a ground truth mask and a pixel embedding feature map, obtain a predicted segmentation mask and a predicted classification based on the alignment query through a self-attention transform decoder, calculate a first error between the predicted segmentation mask and a preset standard segmentation mask based on a first preset learning loss function, and calculate a second error between the predicted classification and the preset standard classification based on a second preset learning loss function, perform back propagation based on the first error and the second error, adjust the classification segmentation task-associated parameters of the alignment query to obtain an adjusted alignment query, obtain the classification confidence in the instance segmentation result based on the adjusted alignment query and the pixel embedding feature map, and accurately reflect the segmentation quality, so that the user can intuitively and vividly determine the quality of the segmented instance, avoiding additional quality analysis and determination labor.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to an image instance segmentation method, device and medium. Background Art

[0002] Image instance segmentation, as a pixel-level visual technology, plays an important role in supporting applications in medical imaging, autonomous driving, robotics, geographic information systems, and other fields. However, current image instance segmentation technology suffers from a problem in which the classification confidence cannot accurately reflect the segmentation quality. This makes it difficult for users to quickly determine and judge the image quality based on the classification confidence, which, to a certain extent, increases the labor of manual analysis. Summary of the Invention

[0003] The main purpose of the present invention is to provide an image instance segmentation method, device and medium, aiming to solve the technical problem that users are unable to quickly determine and judge the quality of an image based on current image instance segmentation technology.

[0004] To achieve the above object, the present invention provides an image instance segmentation method, characterized in that the image instance segmentation method comprises the following steps:

[0005] generating an alignment query using an alignment query learning module based on a ground truth mask and a pixel embedding feature map obtained by processing an initial image;

[0006] Obtaining a predicted segmentation mask and a predicted classification based on the alignment query by a self-attention transform decoder, and calculating a first error between the predicted segmentation mask and a preset standard segmentation mask based on a first preset learning loss function, and calculating a second error between the predicted classification and a preset standard classification based on a second preset learning loss function;

[0007] Performing backpropagation based on the first error and the second error to adjust parameters associated with the classification and segmentation tasks of the alignment query to obtain an adjusted alignment query, wherein the segmentation task and the classification task in the adjusted alignment query are consistent;

[0008] An instance segmentation result of the initial image is obtained according to the adjusted alignment query and the pixel embedding feature map, wherein the instance segmentation result includes a classification confidence, a classification result, and a mask result.

[0009] Optionally, the step of generating an alignment query based on the ground truth mask and the pixel embedding feature map using the alignment query learning module comprises:

[0010] Performing random point sampling on the ground truth mask and the pixel embedding feature map using the alignment query learning module to obtain a point-sampled ground truth mask and a point-sampled pixel embedding;

[0011] The point sampling ground truth mask and the point sampling pixel embedding are multiplied to obtain the masked point sampling features, global average pooling and layer normalization are used to aggregate the masked point sampling features, and then the alignment query is obtained through a multi-layer perceptron.

[0012] Optionally, the step of obtaining an instance segmentation result of the initial image according to the adjusted alignment query and the pixel embedding feature map includes:

[0013] Performing vector dot product of the adjusted alignment query and the instance query with the feature vector corresponding to the pixel embedding feature map through the self-attention transform decoder to obtain a first-category feature vector and a second-category feature vector;

[0014] Performing vector concatenation of the first-category feature vector and the second-category feature vector to obtain an instance segmentation result of a current decoding layer, wherein the self-attention transform decoder has several decoding layers;

[0015] An instance segmentation result of the initial image is determined according to the instance segmentation result of the current decoding layer.

[0016] Optionally, the step of determining the instance segmentation result of the initial image according to the instance segmentation result of the current decoding layer includes:

[0017] Updating the adjusted alignment query based on the instance segmentation result of the current decoding layer to obtain an updated alignment query;

[0018] Inputting the mask result included in the instance segmentation result of the current decoding layer into the next decoding layer, and calculating a new instance segmentation result of the current decoding layer based on the mask result, the updated alignment query and the instance query by the self-attention transform decoder;

[0019] Determine whether the new current decoding layer is the final decoding layer of the self-attention transform decoder;

[0020] If so, the instance segmentation result of the new current decoding layer is output as the instance segmentation result of the initial image.

[0021] Optionally, after the step of determining whether the new current decoding layer is the final decoding layer of the self-attention transform decoder, the method further includes:

[0022] If the current decoding layer is not the final decoder layer of the self-attention transform decoder, updating the updated alignment query based on the instance segmentation result of the new current decoding layer to obtain a further updated alignment query.

[0023] Optionally, before the step of inputting the mask result included in the instance segmentation result of the current decoding layer into the next decoding layer, the method further includes:

[0024] Alignment label assignment learning is performed on the updated alignment query and the instance query, and the instance segmentation result is generated based on the alignment query and the instance query after the alignment label assignment learning.

[0025] Optionally, the step of performing alignment label assignment learning on the updated alignment query and the instance query includes:

[0026] After matching the instance query and the alignment query into positive samples and negative samples based on a bipartite graph, a preset mask quality is used as a classification optimization target to guide the positive samples, and the sample weights of the negative samples are adjusted.

[0027] Optionally, the steps of the image instance segmentation method further include:

[0028] Performing layer-by-layer feature extraction and downsampling on the initial image through a feature extraction backbone network to obtain a multi-scale feature map;

[0029] The multi-scale feature map is upsampled and fused through a pixel decoder to obtain the pixel embedding feature map.

[0030] To achieve the above objectives, the present invention also provides an electronic device, which includes: a memory, a processor, and a computer processing program stored in the memory and executable on the processor. When the computer processing program is executed by the processor, the steps of the above-mentioned image instance segmentation method are implemented.

[0031] In addition, to achieve the above-mentioned purpose, the present invention also proposes a computer-readable storage medium, on which a computer processing program is stored. When the computer processing program is executed by a processor, the steps of the above-mentioned image instance segmentation method are implemented.

[0032] In the present invention, an alignment query learning module is used to generate an alignment query based on a ground truth mask and a pixel embedding feature map, wherein the pixel embedding feature map is obtained by processing an initial image, a predicted segmentation mask and a predicted classification are obtained based on the alignment query by a self-attention transform decoder, a first error between the predicted segmentation mask and a preset standard segmentation mask is calculated based on a first preset learning loss function, and a second error between the predicted classification and the preset standard classification is calculated based on a second preset learning loss function, backpropagation is performed based on the first error and the second error, and the classification segmentation task-associated parameters of the alignment query are adjusted to obtain an adjusted alignment query, wherein the segmentation task and the classification task in the adjusted alignment query are consistent, and an instance segmentation result of the initial image is obtained based on the adjusted alignment query and the pixel embedding feature map, wherein the instance segmentation result includes a classification confidence, a classification result, and a mask result. The classification confidence in the instance segmentation result output by the adjusted alignment query can accurately reflect the segmentation quality, allowing users to intuitively and visually determine the quality of the segmented instance, avoiding additional quality analysis and determination labor. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present invention;

[0034] Figure 2 1. A schematic diagram of a flow chart of an embodiment of an image instance segmentation method according to the present invention;

[0035] Figure 3 A schematic diagram of the process of generating and learning alignment queries in the present invention;

[0036] Figure 4 Schematic diagram of the framework of image instance segmentation of the present invention;

[0037] Figure 5 Schematic diagram of the flowchart of the image instance segmentation method of the present invention.

[0038] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0039] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] like Figure 1 As shown, Figure 1 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present invention.

[0041] The image instance segmentation method of the embodiment of the present invention is applied to an electronic device, such as Figure 1As shown, the electronic device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display area (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0042] Optionally, the electronic device may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among them, the sensors include light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; of course, the mobile terminal can also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which will not be repeated here.

[0043] Those skilled in the art will understand that Figure 1 The electronic device structure shown in the figure does not constitute a limitation to the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0044] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a computer processing program.

[0045] exist Figure 1In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the computer processing program stored in the memory 1005 and perform the following operations:

[0046] generating an alignment query using an alignment query learning module based on a ground truth mask and a pixel embedding feature map obtained by processing an initial image;

[0047] Obtaining a predicted segmentation mask and a predicted classification based on the alignment query by a self-attention transform decoder, and calculating a first error between the predicted segmentation mask and a preset standard segmentation mask based on a first preset learning loss function, and calculating a second error between the predicted classification and a preset standard classification based on a second preset learning loss function;

[0048] Performing backpropagation based on the first error and the second error to adjust parameters associated with the classification and segmentation tasks of the alignment query to obtain an adjusted alignment query, wherein the segmentation task and the classification task in the adjusted alignment query are consistent;

[0049] An instance segmentation result of the initial image is obtained according to the adjusted alignment query and the pixel embedding feature map, wherein the instance segmentation result includes a classification confidence, a classification result, and a mask result.

[0050] Furthermore, the processor 1001 may call a computer program stored in the memory 1005 and perform the following operations:

[0051] The step of generating an alignment query based on the ground truth mask and the pixel embedding feature map using an alignment query learning module comprises: randomly sampling the ground truth mask and the pixel embedding feature map using the alignment query learning module to obtain a point-sampled ground truth mask and a point-sampled pixel embedding;

[0052] The point sampling ground truth mask and the point sampling pixel embedding are multiplied to obtain the masked point sampling features, global average pooling and layer normalization are used to aggregate the masked point sampling features, and then the alignment query is obtained through a multi-layer perceptron.

[0053] Furthermore, the processor 1001 may call a computer program stored in the memory 1005 and perform the following operations:

[0054] The step of obtaining an instance segmentation result of the initial image according to the adjusted alignment query and the pixel embedding feature map comprises: performing vector dot product of the adjusted alignment query and the instance query with the feature vector corresponding to the pixel embedding feature map by the self-attention transform decoder to obtain a first type of feature vector and a second type of feature vector;

[0055] Performing vector concatenation of the first-category feature vector and the second-category feature vector to obtain an instance segmentation result of a current decoding layer, wherein the self-attention transform decoder has several decoding layers;

[0056] An instance segmentation result of the initial image is determined according to the instance segmentation result of the current decoding layer.

[0057] Furthermore, the processor 1001 may call a computer program stored in the memory 1005 and perform the following operations:

[0058] The step of determining the instance segmentation result of the initial image according to the instance segmentation result of the current decoding layer comprises: updating the adjusted alignment query based on the instance segmentation result of the current decoding layer to obtain an updated alignment query;

[0059] Inputting the mask result included in the instance segmentation result of the current decoding layer into the next decoding layer, and calculating a new instance segmentation result of the current decoding layer based on the mask result, the updated alignment query and the instance query by the self-attention transform decoder;

[0060] Determine whether the new current decoding layer is the final decoding layer of the self-attention transform decoder;

[0061] If so, the instance segmentation result of the new current decoding layer is output as the instance segmentation result of the initial image.

[0062] Furthermore, the processor 1001 may call a computer program stored in the memory 1005 and perform the following operations:

[0063] After the step of determining whether the new current decoding layer is the final decoding layer of the self-attention transform decoder, it also includes: if the current decoding layer is not the final decoder layer of the self-attention transform decoder, updating the updated alignment query based on the instance segmentation result of the new current decoding layer to obtain a further updated alignment query.

[0064] Furthermore, the processor 1001 may call a computer program stored in the memory 1005 and perform the following operations:

[0065] Before the step of inputting the mask result included in the instance segmentation result of the current decoding layer into the next decoding layer, it also includes: performing alignment label assignment learning on the updated alignment query and the instance query, and generating the instance segmentation result based on the alignment query and the instance query after the alignment label assignment learning.

[0066] Furthermore, the processor 1001 may call a computer program stored in the memory 1005 and perform the following operations:

[0067] The step of performing alignment label assignment learning on the updated alignment query and the instance query includes: matching the instance query and the alignment query into positive samples and negative samples based on a bipartite graph, using a preset mask quality as a classification optimization target to guide the positive samples, and adjusting the sample weights of the negative samples.

[0068] Furthermore, the processor 1001 may call a computer program stored in the memory 1005 and perform the following operations:

[0069] Performing layer-by-layer feature extraction and downsampling on the initial image through a feature extraction backbone network to obtain a multi-scale feature map;

[0070] The multi-scale feature map is upsampled and fused through a pixel decoder to obtain the pixel embedding feature map.

[0071] like Figure 2 As shown, Figure 2 FIG. 1 is a flow chart of an embodiment of an image instance segmentation method according to the present invention. In this embodiment, the image instance segmentation method includes the following steps:

[0072] In step A10, an alignment query learning module is used to generate an alignment query based on the ground truth mask and the pixel embedding feature map, wherein the pixel embedding feature map is obtained by processing the initial image.

[0073] In the present invention, the newly added alignment query learning module generates alignment queries through the ground truth mask and the pixel embedded feature map. The generation method is to use the two to perform vector dot product after random point sampling, and divide them into different alignment queries according to different sampling point sets. These alignment queries learn to establish consistency between the segmentation task and the classification task after passing through the self-attention transform decoder, so that the classification confidence in the instance segmentation result based on the alignment query output can accurately reflect the segmentation quality.

[0074] Optionally, the step of generating an alignment query based on the ground truth mask and the pixel embedding feature map using the alignment query learning module in step A10 includes:

[0075] Step A101, using the alignment query learning module to perform random point sampling on the ground truth mask and the pixel embedding feature map to obtain a point sampling ground truth mask and a point sampling pixel embedding;

[0076] In step A102, the point sampling ground truth mask and the point sampling pixel embedding are multiplied to obtain masked point sampling features, global average pooling and layer normalization are used to aggregate the masked point sampling features, and the alignment query is obtained through a multi-layer perceptron.

[0077] Specific reference Figure 3 and Figure 5 As shown in , the alignment query is generated by the pixel embedding feature map and the ground truth mask. As shown in Formula 1, the pixel embedding feature map and the ground truth mask are first sampled using the same uniformly distributed random points to obtain T groups of sampling groups. Based on the T groups of sampling groups, the corresponding T point sampling pixel embeddings and T*M point sampling ground truth masks are obtained respectively. Then, the T point sampling pixel embeddings and T*M point sampling ground truth masks are multiplied to obtain N AQ Finally, global average pooling and layer normalization are used to aggregate N AQ Masks are used to block the sampling features of points, and then different alignment queries are obtained through a multi-layer perceptron, where N AQ Indicates the number of queries.

[0078]

[0079] Among them, PE represents the pixel embedding feature map, m j is the ground truth mask, PPE i is the point sampling pixel embedding, Pm j is the ground truth mask representing the point sampling, MPf i Indicates the mask shielding point sampling feature, aq i represents the initial value of the alignment query, MLP represents a multi-layer perceptron with 2 hidden layers, LN represents layer normalization, and GAP represents global average pooling.

[0080] like Figure 3 As shown in , for each image with M instances, we generate T sampling point sets, each of which can generate M alignment queries. Due to the use of uniformly distributed random point sampling, the alignment queries generated by the same ground truth mask in different groups are different. The robustness of mask quality learning is enhanced by different alignment queries. We retain the top N generated by all T*M queries. AQ queries, if the number of queries is less than N AQ , then fill the zero vector. Here we set N AQ =100.

[0081] Step A20, obtaining a predicted segmentation mask and a predicted classification based on the alignment query through a self-attention transform decoder, and calculating a first error between the predicted segmentation mask and a preset standard segmentation mask based on a first preset learning loss function, and calculating a second error between the predicted classification and the preset standard classification based on a second preset learning loss function.

[0082] Step A30 , performing back propagation based on the first error and the second error, and adjusting the classification and segmentation task-related parameters of the alignment query to obtain an adjusted alignment query, wherein the segmentation task and the classification task in the adjusted alignment query are consistent.

[0083] After the generated alignment query obtains the predicted segmentation mask and predicted classification through the self-attention transform decoder, the first error between the predicted segmentation mask and the preset standard segmentation mask is calculated based on the first preset learning loss function. Specifically, in this embodiment, the binary cross entropy loss function (BCE) and the DICE loss function are weighted as the learning loss function for predicting the segmentation mask, and the second error between the predicted classification and the preset standard classification is calculated based on the second preset learning loss function. Specifically, the quality focus loss function is used as the learning loss function for predicting the classification. The error obtained according to the above learning loss function is backpropagated, thereby continuously adjusting and optimizing the classification segmentation task-associated parameters of the alignment query until they are adjusted and optimized to the optimal classification segmentation task-associated parameters, thereby obtaining an adjusted alignment query. The segmentation task and the classification task in the adjusted alignment query are consistent, so that the classification confidence in the instance segmentation result output based on the adjusted alignment query is consistent with the segmentation quality, allowing users to intuitively and vividly determine the quality of the segmented instance, avoiding additional quality analysis and determination labor.

[0084] where the label of an alignment query is the ground truth mask that generated it, and the intersection-over-union between the predicted segmentation mask and the ground truth mask.

[0085] Step A40: Obtain an instance segmentation result of the initial image according to the adjusted alignment query and the pixel embedding feature map, wherein the instance segmentation result includes a classification confidence, a classification result, and a mask result.

[0086] In this embodiment, the Figure 4 The image instance segmentation framework outputs instance segmentation results. Specifically, the initial image is input into the feature extraction backbone network and the pixel decoder. After obtaining the pixel embedding feature map, the pixel embedding feature map is vector-multiplied with the adjusted alignment query and instance query to obtain the instance segmentation result.

[0087] The generated pixel embedding feature map is vector-wise multiplied with the alignment query and instance query, and then fed into the self-attention transform decoder to generate and learn the aligned query learning module to obtain the adjusted aligned query. At the same time, the adjusted aligned query is also subjected to alignment label assignment learning with the instance query. This aims to explicitly measure the classification confidence score and segmentation quality of the aligned query, thereby improving the accuracy of the segmentation quality reflected by the classification confidence.

[0088] Optionally, the steps of the image instance segmentation method further include:

[0089] Step A401: Perform layer-by-layer feature extraction and downsampling on the initial image through a feature extraction backbone network to obtain a multi-scale feature map.

[0090] In this embodiment, two typical feature extraction backbone networks are used: the first is a standard convolution-based residual network backbone, and the other is a moving window-based hierarchical visual self-attention model backbone. The feature extraction backbone network will provide multi-scale feature maps for subsequent steps. Specifically, refer to Figure 5 The feature extraction backbone network performs feature extraction operations and downsampling on the input initial image layer by layer to obtain a multi-scale feature map with a resolution smaller than the initial image, for example, a multi-scale feature map with a resolution of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the initial image.

[0091] Step A402: upsampling and fusing the multi-scale feature map through a pixel decoder to obtain the pixel embedding feature map.

[0092] In this embodiment, the pixel decoder upsamples and fuses the multi-scale feature maps obtained by the previous feature extraction backbone network layer by layer to obtain a pixel embedded feature map. The pixel decoder consists of two parts, namely a feature pyramid and a multi-scale deformable attention transformer. Specifically. Assuming that there are multi-scale feature maps with resolutions of 1 / 8, 1 / 16 and 1 / 32 of the initial image, the multi-scale feature maps with resolutions of 1 / 8, 1 / 16 and 1 / 32 of the initial image are first connected, and then the multi-scale feature maps are passed through 6 layers of multi-scale deformable attention transformers. This operation allows the information between feature maps of different scales to fully interact. The connected multi-scale feature maps are then separated into multi-scale feature maps with resolutions of 1 / 8, 1 / 16 and 1 / 32 of the initial image. After that, the feature maps of different scales are fused into a pixel embedded feature map using a feature pyramid. We start with a feature map with a resolution of 1 / 32 of the initial image, upsample it and fuse it through a convolutional layer with a feature map with twice the resolution, that is, a feature map with a resolution of 1 / 16 of the initial image, to generate a feature map with a resolution of 1 / 8 of the initial image. After upsampling the feature map, we also pass it through a convolutional layer with a feature map with twice the resolution, that is, a feature map with a resolution of 1 / 4 of the initial image, to generate a feature map with a resolution of 1 / 4 of the initial image, and then embed the feature map as a pixel-embedded feature map.

[0093] Among them, the pixel embedded feature map will interact with the subsequent alignment query and instance query to obtain the mask result in the instance segmentation result. The classification result is obtained by the alignment query and instance query through the classification layer. In addition to being used to characterize the reliability of the classification result, the classification confidence in this embodiment is also used to characterize the reliability of the segmentation result because the consistency of the classification task and the segmentation task is established in the alignment query. This allows users to determine the quality of the segmentation instance directly based on the classification confidence without any calculation.

[0094] Optionally, the step of obtaining an instance segmentation result of the initial image according to the adjusted alignment query and the pixel embedding feature map in step A40 includes:

[0095] Step A403, performing vector dot product on the feature vectors corresponding to the pixel embedding feature map based on the adjusted alignment query and the instance query by the self-attention transform decoder to obtain a first-category feature vector and a second-category feature vector;

[0096] Step A404: concatenate the first-category feature vector and the second-category feature vector to obtain an instance segmentation result of the current decoding layer, wherein the self-attention transform decoder has several decoding layers.

[0097] In this embodiment, a masked self-attention transform decoder is used instead of a standard decoder. Typically, a self-attention transform decoder has nine decoding layers and 100 queries. Each decoding layer updates the alignment query and instance query, meaning each decoding layer generates a complete instance segmentation result. Each decoding layer uses the masked result from the instance segmentation result of the previous decoding layer to limit the current layer's attention range, improving focus.

[0098] Specifically, the initial decoding layer of the self-attention transform decoder has three inputs, namely the adjusted alignment query, instance query and pixel embedding feature map. After inputting the adjusted alignment query and instance query, the self-attention transform decoder calculates the cross attention between the vector of the alignment query, the vector of the instance query and the feature vector corresponding to the pixel embedding feature map, and obtains the first-category feature vector and the second-category feature vector. The first-category feature vector and the second-category feature vector are spliced ​​to obtain the instance segmentation result of the current decoding layer.

[0099] It should be noted that because there are several decoding layers in the self-attention transform decoder, it is necessary to generate instance segmentation results for each layer in turn. In this way, the focus range when calculating the cross-attention is limited by the mask result of the instance segmentation result of the previous decoding layer. Only the area where the binary mask is 1 is calculated, and the area where it is 0 is ignored, so as to improve the focus and enhance the accuracy of the instance segmentation results.

[0100] The loss function is calculated for the alignment query of the initial decoding layer and the instance segmentation results generated by each decoding layer as an auxiliary loss, and the loss function of the auxiliary loss is determined by the alignment label assignment method.

[0101] Step A405 : Determine the instance segmentation result of the initial image according to the instance segmentation result of the current decoding layer.

[0102] When the current decoding layer is the final decoding layer of the self-attention transform decoder, the instance segmentation result of the current decoding layer can be determined as the instance segmentation result of the initial image.

[0103] Optionally, the step of determining the instance segmentation result of the initial image according to the instance segmentation result of the current decoding layer in step A405 includes:

[0104] Step B10, updating the adjusted alignment query based on the instance segmentation result of the current decoding layer to obtain an updated alignment query;

[0105] Step B20: input the mask result included in the instance segmentation result of the current decoding layer into the next decoding layer, and calculate the new instance segmentation result of the current decoding layer based on the mask result, the updated alignment query and the instance query through the self-attention transform decoder.

[0106] Assume that there are 9 decoding layers in the self-attention decoder. At this time, the adjusted alignment query is updated. After obtaining the updated alignment query, the updated alignment query and the mask result included in the instance segmentation result of the current decoding layer are input into the next decoding layer, so that the next decoding layer generates the instance segmentation result through the updated alignment query, instance query, pixel embedding feature map, and the mask result of the instance segmentation result of the previous decoding layer. The mask result of the instance segmentation result of the previous decoding layer will limit the focus range when calculating the cross attention. Only the area where the binary mask is 1 is calculated, and the area where it is 0 is ignored, thereby improving the focus and the accuracy of the instance segmentation result.

[0107] Optionally, before the step of inputting the mask result included in the instance segmentation result of the current decoding layer into the next decoding layer in step B20, the method further includes:

[0108] Step B201 : performing alignment label assignment learning on the updated alignment query and the instance query, and generating the instance segmentation result based on the alignment query and the instance query after the alignment label assignment learning.

[0109] Optionally, the step of performing alignment label assignment learning on the updated alignment query and the instance query in step B201 includes:

[0110] Step B2011, after matching the instance query and the alignment query into positive samples and negative samples based on a bipartite graph, use the preset mask quality as a classification optimization target to guide the positive samples, and adjust the sample weights of the negative samples.

[0111] Furthermore, we introduce an aligned label assignment learning method to explicitly align the classification confidence and segmentation quality of the query.

[0112] Reference Figure 5 As shown, a one-to-one label assignment method based on bipartite graph matching is usually used. On the basis of one-to-one matching, it is assumed that part of the matched updated alignment queries and instance queries are used as positive samples, and the other part of the queries are used as negative samples, wherein the mask quality is used to guide the classification optimization target of the positive samples.

[0113] Among the positive samples, for the positive samples selected by bipartite graph matching, following the recent single-stage detection method, the quality focus loss is adopted, and the intersection-over-union ratio between the predicted segmentation mask and the matched ground truth mask (i.e., mask quality) is used as the classification optimization target. As shown in Formula 2, the loss function of the positive sample is expressed, which further aligns the classification task and segmentation task of the updated alignment query. At the same time, it can also reduce the negative impact of selecting samples with high classification confidence scores but low mask quality as positive samples in bipartite graph matching.

[0114]

[0115] Where i represents the i-th instance query matched in the bipartite graph matching, t i represents the intersection-over-union ratio between the predicted segmentation mask and its matching ground truth mask, s represents the classification confidence, γ is a hyperparameter, in this embodiment, it is set to 2, BCE represents the binary cross entropy loss function, L cls_pos Represents the loss function of the positive sample, N pos Indicates the number of categories.

[0116] In the negative samples, the deviation between classification confidence and segmentation quality is reduced by reducing the weight of samples with high segmentation quality but regarded as negative samples. Specifically, in order not to destroy the one-to-one label assignment and generate too many repeated predictions, only queries with IoU greater than θ are reduced, where IoU represents the maximum IoU between the predicted segmentation mask and all GT masks. According to the evaluation index of COCO, setting it below 0.5 is a sufficient condition for incorrect prediction, that is, θ = 0.5, and the negative weight W neg It should be a monotonically decreasing function defined in the interval [0.5, 1], so the loss function of the negative sample is shown in Formula 3:

[0117]

[0118] Among them, γ is a hyperparameter, which is set to 2, W neg The function passes through the points (0.5, 1) and (1, 0), so k and b are fixed constants.

[0119] The final alignment label assignment loss function is the sum of the loss functions of positive and negative samples, as shown in Formula 4:

[0120]

[0121] Where i represents the i-th instance query that is matched in the bipartite graph matching, j represents the j-th instance query that is not matched in the bipartite graph matching, and t i represents the intersection-over-union ratio between the predicted segmentation mask and its matched ground truth mask, and s represents the classification confidence.

[0122] In this embodiment, mask quality is used to guide the classification optimization target of positive samples, and the sample weights of negative samples are adjusted to reduce the deviation between classification confidence and segmentation quality, thereby further improving the accuracy of segmentation quality reflected based on classification confidence.

[0123] Step B30, determining whether the new current decoding layer is the final decoding layer of the self-attention transform decoder;

[0124] Optionally, after determining in step B30 whether the new current decoding layer is the final decoding layer of the self-attention transform decoder, the method further includes:

[0125] Step B40: If the current decoding layer is not the final decoder layer of the self-attention transform decoder, the updated alignment query is updated based on the instance segmentation result of the new current decoding layer to obtain a further updated alignment query.

[0126] The specific implementation of step B40 can refer to the specific implementation of the above steps B10 to B20, which will not be described in detail here.

[0127] Step B50: If yes, output the instance segmentation result of the new current decoding layer as the instance segmentation result of the initial image.

[0128] The specific implementation of step B50 can refer to the specific implementation of step A405 above, and will not be repeated here.

[0129] In this embodiment, an alignment query learning module is used to generate an alignment query based on a ground truth mask and a pixel embedding feature map. The pixel embedding feature map is obtained by processing an initial image. A self-attention transform decoder is used to obtain a predicted segmentation mask and a predicted classification based on the alignment query. A first error between the predicted segmentation mask and a preset standard segmentation mask is calculated based on a first preset learning loss function, and a second error between the predicted classification and a preset standard classification is calculated based on a second preset learning loss function. Backpropagation is performed based on the first and second errors, and parameters associated with the classification segmentation task of the alignment query are adjusted to obtain an adjusted alignment query. The segmentation task and the classification task in the adjusted alignment query are consistent. An instance segmentation result of the initial image is obtained based on the adjusted alignment query and the pixel embedding feature map. The instance segmentation result includes a classification confidence, a classification result, and a mask result. The classification confidence in the instance segmentation result output by the adjusted alignment query accurately reflects the segmentation quality, allowing users to intuitively and visually determine the quality of the segmented instance, avoiding additional quality analysis and determination labor.

[0130] In addition, an embodiment of the present invention further proposes an electronic device, which includes a memory, a processor, and a computer processing program stored in the memory and executable on the processor. When the processor executes the computer processing program, the steps of the above-mentioned image instance segmentation method are implemented.

[0131] In addition, the present invention also proposes a computer-readable storage medium, on which a computer processing program is stored. When the modeling simulation program is executed by a processor, the steps of the above-mentioned image instance segmentation method are implemented.

[0132] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0133] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0134] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0135] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structures or equivalent process changes made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for image instance segmentation, characterized in that: The image instance segmentation method comprises the following steps: generating an alignment query using an alignment query learning module based on a ground truth mask and a pixel embedding feature map obtained by processing an initial image; Obtaining a predicted segmentation mask and a predicted classification based on the alignment query by a self-attention transform decoder, and calculating a first error between the predicted segmentation mask and a preset standard segmentation mask based on a first preset learning loss function, and calculating a second error between the predicted classification and a preset standard classification based on a second preset learning loss function; Performing backpropagation based on the first error and the second error to adjust parameters associated with the classification and segmentation tasks of the alignment query to obtain an adjusted alignment query, wherein the segmentation task and the classification task in the adjusted alignment query are consistent; An instance segmentation result of the initial image is obtained according to the adjusted alignment query and the pixel embedding feature map, wherein the instance segmentation result includes a classification confidence, a classification result, and a mask result.

2. The image instance segmentation method according to claim 1, wherein: The step of generating an alignment query based on the ground truth mask and the pixel embedding feature map using the alignment query learning module comprises: Performing random point sampling on the ground truth mask and the pixel embedding feature map using the alignment query learning module to obtain a point-sampled ground truth mask and a point-sampled pixel embedding; The point sampling ground truth mask and the point sampling pixel embedding are multiplied to obtain the masked point sampling features, global average pooling and layer normalization are used to aggregate the masked point sampling features, and then the alignment query is obtained through a multi-layer perceptron.

3. The image instance segmentation method according to claim 2, wherein: The step of obtaining an instance segmentation result of the initial image according to the adjusted alignment query and the pixel embedding feature map comprises: Performing vector dot product of the adjusted alignment query and the instance query with the feature vector corresponding to the pixel embedding feature map through the self-attention transform decoder to obtain a first-category feature vector and a second-category feature vector; Performing vector concatenation of the first-category feature vector and the second-category feature vector to obtain an instance segmentation result of a current decoding layer, wherein the self-attention transform decoder has several decoding layers; An instance segmentation result of the initial image is determined according to the instance segmentation result of the current decoding layer.

4. The image instance segmentation method according to claim 3, wherein: The step of determining the instance segmentation result of the initial image according to the instance segmentation result of the current decoding layer comprises: Updating the adjusted alignment query based on the instance segmentation result of the current decoding layer to obtain an updated alignment query; Inputting the mask result included in the instance segmentation result of the current decoding layer into the next decoding layer, and calculating a new instance segmentation result of the current decoding layer based on the mask result, the updated alignment query and the instance query by the self-attention transform decoder; Determine whether the new current decoding layer is the final decoding layer of the self-attention transform decoder; If so, the instance segmentation result of the new current decoding layer is output as the instance segmentation result of the initial image.

5. The image instance segmentation method according to claim 4, wherein: After the step of determining whether the new current decoding layer is the final decoding layer of the self-attention transform decoder, the method further includes: If the current decoding layer is not the final decoder layer of the self-attention transform decoder, updating the updated alignment query based on the instance segmentation result of the new current decoding layer to obtain a further updated alignment query.

6. The image instance segmentation method according to claim 4, wherein: Before the step of inputting the mask result included in the instance segmentation result of the current decoding layer into the next decoding layer, the method further includes: Alignment label assignment learning is performed on the updated alignment query and the instance query, and the instance segmentation result is generated based on the alignment query and the instance query after the alignment label assignment learning.

7. The image instance segmentation method according to claim 6, wherein: The step of performing alignment label assignment learning on the updated alignment query and the instance query includes: After matching the instance query and the alignment query into positive samples and negative samples based on a bipartite graph, a preset mask quality is used as a classification optimization target to guide the positive samples, and the sample weights of the negative samples are adjusted.

8. The image instance segmentation method according to any one of claims 1 to 7, wherein: The steps of the image instance segmentation method further include: Performing layer-by-layer feature extraction and downsampling on the initial image through a feature extraction backbone network to obtain a multi-scale feature map; The multi-scale feature map is upsampled and fused through a pixel decoder to obtain the pixel embedding feature map.

9. An electronic device, characterized in that: The electronic device comprises: a memory, a processor, and a computer processing program stored in the memory and executable on the processor, wherein the computer processing program is configured to implement the steps of the image instance segmentation method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer processing program, which, when executed by a processor, implements the steps of the image instance segmentation method according to any one of claims 1 to 8.

Images