Image segmentation method, image segmentation device, storage medium and electronic equipment

Abstract

The invention provides an image segmentation method, an image segmentation device, a storage medium and electronic equipment, and relates to the technical field of computer vision. The method comprises the following steps: acquiring a to-be-processed image, and acquiring query information; extracting image features of multiple scales from the to-be-processed image by using an image feature extraction network; processing the image features of the plurality of scales by using a semantic feature processing network to obtain regional distribution features of the to-be-processed image; fusing the query information and the regional distribution features to obtain an attention mask; a transformer decoder is used for processing the image features, the query information and the attention mask, and the updated query information is obtained; and obtaining an image segmentation result of the to-be-processed image based on the updated query information. The calculation amount required by image segmentation can be reduced, and the accuracy of image segmentation can be ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to an image segmentation method, an image segmentation device, a computer-readable storage medium, and an electronic device. Background Art

[0002] Image segmentation refers to dividing an image into several regions with specific properties and identifying regional information. Image segmentation algorithms are mostly implemented based on machine learning models. In related technologies, in order to ensure the accuracy of image segmentation, the model is designed to be more complex, thereby increasing the number of model parameters and the amount of calculation, limiting the application of image segmentation in lightweight scenarios such as mobile terminals. Summary of the invention

[0003] The present disclosure provides an image segmentation method, an image segmentation device, a computer-readable storage medium and an electronic device, so as to reduce the amount of calculation required for image segmentation at least to a certain extent and ensure accuracy.

[0004] According to a first aspect of the present disclosure, there is provided an image segmentation method, comprising: obtaining an image to be processed and obtaining query information; extracting image features of multiple scales from the image to be processed using an image feature extraction network; processing the image features of multiple scales using a semantic feature processing network to obtain regional distribution features of the image to be processed; obtaining an attention mask by fusing the query information and the regional distribution features; processing the image features, the query information, and the attention mask using a transformer (a neural network model) decoder to obtain updated query information; and obtaining an image segmentation result of the image to be processed based on the updated query information.

[0005] According to a second aspect of the present disclosure, an image segmentation device is provided, comprising: a data acquisition unit, configured to acquire an image to be processed and query information; an image feature extraction unit, configured to extract image features of multiple scales from the image to be processed using an image feature extraction network; a regional distribution feature extraction unit, configured to process the image features of multiple scales using a semantic feature processing network to obtain regional distribution features of the image to be processed; an attention mask processing unit, configured to obtain an attention mask by fusing the query information and the regional distribution features; an information updating unit, configured to process the image features, the query information, and the attention mask using a transformer decoder to obtain updated query information; and an image segmentation result output unit, configured to obtain an image segmentation result of the image to be processed based on the updated query information.

[0006] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image segmentation method of the first aspect and possible implementation methods thereof are implemented.

[0007] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the image segmentation method of the first aspect and its possible implementation method by executing the executable instructions.

[0008] The technical solution disclosed in this disclosure has the following beneficial effects:

[0009] The use of a lightweight image feature extraction network to extract image features is conducive to reducing the number of network parameters and the amount of calculation, and by updating the query information, the information of the image features is passed to the query information, and finally the image segmentation result is obtained based on the updated query information, which makes the image segmentation process more stable and reliable. Therefore, this solution can ensure the accuracy of image segmentation while reducing the number of network parameters and the amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of an image segmentation method in this exemplary embodiment is shown;

[0011] Figure 2 A flowchart of extracting image features at multiple scales in this exemplary embodiment is shown;

[0012] Figure 3 A flow chart of obtaining regional distribution features in this exemplary embodiment is shown;

[0013] Figure 4 A schematic diagram showing the structure and processing process of a transformer decoder in this exemplary embodiment is shown;

[0014] Figure 5 A flowchart showing updating query information in this exemplary embodiment is shown;

[0015] Figure 6 A schematic diagram showing updating query information in this exemplary embodiment;

[0016] Figure 7 A schematic flowchart of an image segmentation method in this exemplary embodiment is shown;

[0017] Figure 8 A flowchart of obtaining a semantic segmentation result and an instance segmentation result in this exemplary embodiment is shown;

[0018] Fig. 9A schematic diagram showing the structure of an image segmentation device in this exemplary embodiment is shown;

[0019] Fig.10 A schematic structural diagram of an electronic device in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0020] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings.

[0021] The accompanying drawings are schematic diagrams of the present disclosure and are not necessarily drawn to scale. Some of the block diagrams shown in the accompanying drawings may be functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in hardware modules or integrated circuits, or in networks, processors or microcontrollers. The embodiments can be implemented in various forms and should not be construed as being limited to the examples set forth herein. The features, structures or characteristics described in the present disclosure may be combined in one or more embodiments in any suitable manner. In the description below, many specific details are provided to provide a full description of the embodiments of the present disclosure. However, those skilled in the art should appreciate that one or more specific details may be omitted when implementing the technical solution of the present disclosure, or one or more specific details may be replaced by other methods, components, devices, steps, etc.

[0022] Some general image segmentation algorithms use relatively complex machine learning models. Although they have good performance, the relatively large number of parameters and computational complexity make them difficult to apply in lightweight scenarios, such as running on mobile devices with limited hardware resources or applying to tasks with high real-time requirements.

[0023] However, most of the current image segmentation algorithm models for mobile terminals have low accuracy and do not support simultaneous processing of multiple tasks such as instance segmentation and semantic segmentation. This affects the versatility and application scope of image segmentation and cannot meet the needs of complex scenarios and multi-tasks.

[0024] In view of one or more of the above problems, an exemplary embodiment of the present disclosure provides an image segmentation method, which can reduce the amount of calculation required for image segmentation and ensure accuracy.

[0025] Figure 1 An exemplary process of the image segmentation method is shown, which may include the following steps S110 to S160:

[0026] Step S110, obtaining an image to be processed and obtaining query information;

[0027] Step S120, extracting image features of multiple scales from the image to be processed using an image feature extraction network;

[0028] Step S130, using a semantic feature processing network to process image features at multiple scales to obtain regional distribution features of the image to be processed;

[0029] Step S140, obtaining an attention mask by fusing the query information and the regional distribution features;

[0030] Step S150, using a transformer decoder to process the image features, query information, and attention mask to obtain updated query information;

[0031] Step S160: obtaining an image segmentation result of the image to be processed based on the updated query information.

[0032] Based on the above method, a lightweight image feature extraction network is used to extract image features, which is conducive to reducing the number of network parameters and the amount of calculation. By updating the query information, the information of the image features is passed to the query information, and finally the image segmentation result is obtained based on the updated query information, which makes the image segmentation process more stable and reliable. Therefore, this solution can ensure the accuracy of image segmentation while reducing the number of network parameters and the amount of calculation.

[0033] Below Figure 1 Provide detailed instructions for each step.

[0034] refer to Figure 1 In step S110, the image to be processed is obtained, and query information is obtained.

[0035] The image to be processed is an image that needs to be segmented, which can be an image of any type and acquired by any method. In one embodiment, the acquired original image can be preprocessed, such as sampling to a specific size, image noise reduction, etc., and the preprocessed image is used as the image to be processed. The image to be processed may include one or more targets, and the target may be a recognizable person or object. For example, image segmentation of the image to be processed may refer to identifying and segmenting the target in the image to obtain the corresponding result.

[0036] The query information is used to learn the semantic information in the image to be processed. The query information may be in the form of a query vector or a query matrix. In this exemplary embodiment, any amount of query information may be obtained, and different query information may be used to learn the semantic information of different objects in the image to be processed.

[0037] In step S110, initial query information may be obtained. For example, the number and dimension of query information may be preset, and query information of corresponding number and dimension may be obtained by random initialization. Alternatively, a query vector with certain generality may be obtained during training, and used as the initial query vector in step S110.

[0038] Continue to refer Figure 1 In step S120, an image feature extraction network is used to extract image features of multiple scales from the image to be processed.

[0039] Among them, multiple scales refer to feature extraction of local areas (or global areas) of different sizes in the processed image. Extracting image features at different scales can capture the structure and detail information of different scales in the image.

[0040] In one embodiment, the image feature extraction network may include a first feature extraction layer, a second feature extraction layer, ..., an Mth feature extraction layer connected in sequence, where M is a positive integer not less than 2. Figure 2 As shown, the above-mentioned use of the image feature extraction network to extract image features of multiple scales from the image to be processed may include the following steps S210 and S220:

[0041] Step S210, inputting the image to be processed into an image feature extraction network, and performing feature extraction processing on the image to be processed by a first feature extraction layer to obtain image features of a first scale;

[0042] Step S220: The i-th feature extraction layer performs downsampling and feature extraction processing on the image features output by the i-1-th feature extraction layer to obtain image features of the i-th scale, where i is any positive integer in [2, M].

[0043] Among them, the structures of different feature extraction layers can be the same or different. Exemplarily, each feature extraction layer can include a pooling layer and a convolution layer, the pooling layer is used for downsampling, such as 2 times downsampling can be achieved through a 2×2 pooling window, and the convolution layer is used to extract image features. After the image to be processed is input into the image feature extraction network, the first feature extraction layer, the second feature extraction layer, ..., the Mth feature extraction layer can extract image features of the first scale, image features of the second scale, ..., image features of the Mth scale in sequence. From the first scale to the Mth scale, the size of the image features can decrease. Exemplarily, M is 4, and the size of the image to be processed is H×W, H represents the height, and W represents the width. The image to be processed is input into the image feature extraction network, and the first feature extraction layer extracts the image features of the first scale, and its size can be H / 4×W / 4. The second feature extraction layer performs 2 times downsampling and convolution processing on the image features of the first scale to obtain image features of the second scale, and its size can be H / 8×W / 8. The third feature extraction layer performs 2-fold downsampling and convolution processing on the image features of the second scale to obtain image features of the third scale, whose size may be H / 16×W / 16. The fourth feature extraction layer performs 2-fold downsampling and convolution processing on the image features of the third scale to obtain image features of the fourth scale, whose size may be H / 32×W / 32. Thus, feature extraction of the pyramid structure is achieved, which is conducive to simplifying the image feature extraction network and reducing the amount of computation for image feature extraction.

[0044] Continue to refer Figure 1 In step S130, the image features of multiple scales are processed using a semantic feature processing network to obtain regional distribution features of the image to be processed.

[0045] The regional distribution feature is a feature that characterizes the distribution of objects in the image to be processed, such as a mask feature. Based on the extraction of image features at multiple scales, the semantic feature processing network can be used to further learn image semantic information and obtain regional distribution features.

[0046] In one embodiment, the semantic feature processing network may include a semantic extraction module, a feature optimization module, and a regional distribution extraction module. Figure 3 As shown, the above-mentioned processing of image features at multiple scales by using a semantic feature processing network to obtain regional distribution features of the image to be processed may include the following steps S310 to S330:

[0047] Step S310: input the image features of multiple scales into the semantic feature processing network, and the semantic extraction module fuses and processes the image features of multiple scales to obtain global semantic features.

[0048] In one embodiment, if the sizes of image features at multiple scales are different, in order to fuse them, the image features at multiple scales can be converted to the same size through sampling processing. For example, the sizes of image features at four scales are H / 4×W / 4, H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32, respectively. They can be downsampled to H / 64×W / 64 through an average pooling operation, and then the downsampled image features are spliced ​​according to the channel dimensions of the image features to achieve the fusion of image features at different scales.

[0049] In one embodiment, the semantic extraction module may include a transformer block composed of a multi-head self-attention mechanism and a feedforward network, and the fused image features may be input into the transformer block, and global semantic features may be obtained after processing. The transformer block may obtain the receptive field and rich semantic information of the entire image to be processed, thereby improving the accuracy and robustness of image segmentation.

[0050] Step S320: The feature optimization module optimizes the image features at multiple scales based on the global semantic features to obtain semantic injection features at multiple scales.

[0051] Among them, there may be semantic gaps in image features of different scales. After obtaining the global semantic features, the image features are optimized based on the global semantic features. Part of the information of the global semantic features can be integrated into the image features, which is equivalent to injecting global semantics into the image features to obtain semantic injection features, thereby alleviating the semantic gap.

[0052] In one embodiment, the feature optimization module optimizes the image features at multiple scales based on the global semantic features to obtain the semantic injection features at multiple scales, which may include the following steps:

[0053] Perform dimension conversion and batch normalization on image features of multiple scales to obtain normalized image features of multiple scales;

[0054] The global semantic features are dimensionally transformed and batch normalized to obtain normalized global semantic features, and the global semantic features are dimensionally transformed, batch normalized, and activated to obtain semantic optimization weights;

[0055] The normalized image features of each scale are weighted based on the semantic optimization weights and then added to the normalized global semantic features to obtain the semantic injection features of each scale.

[0056] Among them, dimension conversion is used to convert image features or global semantic features to the required dimensions, such as aligning the dimensions of image features with global semantic features. Dimension conversion can be achieved through 1×1 convolutional layers, feature rearrangement layers, etc. Batch normalization can control the numerical range of image features or global semantic features, such as unifying the numerical range of image features with global semantic features. Activation processing can use activation functions such as sigmoid (an S-shaped function) and ReLU (Rectified Linear Unit).

[0057] Exemplarily, the image features of each scale are processed by a 1×1 convolution layer and a batch normalization layer to obtain normalized image features. The global semantic features are processed by a 1×1 convolution layer and a batch normalization layer to obtain normalized global semantic features. The global semantic features are also processed by a 1×1 convolution layer, a batch normalization layer (the parameters used may be the same or different from those of the 1×1 convolution layer and batch normalization layer in the previous step), and an activation process to obtain semantic optimization weights. The normalized image features of each scale are weighted using the semantic optimization weights, and the weighted results are added to the normalized global semantic features to obtain semantic injection features of each scale. The semantic injection feature fusion contains rich semantic information and detail information, which helps to generate more accurate image segmentation results later.

[0058] Step S330: The region distribution extraction module fuses and processes the semantic injection features of multiple scales to obtain the region distribution features of the image to be processed.

[0059] Semantic injection features of different scales can reflect the target distribution information at different scales in the image to be processed. The semantic injection features of different scales are fused and recorded as semantic injection fusion features. The target distribution information in the semantic injection fusion features is more comprehensive and complete. By further processing the semantic injection fusion features, the regional distribution features of the image to be processed can be obtained. Compared with the semantic injection fusion features, the feature space of the regional distribution features is closer to the output space, and its information is more representational.

[0060] In one embodiment, the above-mentioned regional distribution extraction module fuses and processes the semantic injection features of multiple scales to obtain the regional distribution features of the image to be processed, which may include the following steps:

[0061] The semantic injection features of multiple scales are sampled to the same size and then fused to obtain the semantic injection fusion features;

[0062] The semantic injection fusion features are convolved to obtain the regional distribution features of the image to be processed.

[0063] Among them, the sizes of semantic injection features of different scales may be the same or different. If the sizes are different, they are sampled to the same size and then fused. For example, the sizes of semantic injection features of 4 scales are H / 4×W / 4, H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32, respectively. The semantic injection features of the latter three scales can be upsampled to H / 4×W / 4, so that the sizes of the semantic injection features of the 4 scales are the same, and they can be fused by adding, splicing along the channel, etc. to obtain semantic injection fusion features. In one embodiment, when the normalized image features are weighted by the global semantic weight, in order to align the dimensions of the normalized image features of different scales with the global semantic weight and the normalized global semantic features, the image features or the normalized image features can be sampled to the same size as the global semantic weight and the normalized global semantic features (such as H / 4×W / 4), so that the semantic injection features of different scales are the same size, and when the fusion processing is performed through the regional distribution extraction module, there is no need to sample the semantic injection features of different scales.

[0064] After obtaining the semantic injection fusion feature, it can be convolved through the regional distribution extraction module to achieve dimensionality conversion or feature space mapping to obtain regional distribution features.

[0065] Exemplarily, the size of the semantic injection features of the four scales is H / 4×W / 4, which are input into the regional distribution extraction module, and the element-by-element camera is first performed to achieve feature fusion to obtain the semantic injection fusion feature, and then convolution processing is performed through two consecutive convolution layers to obtain the regional distribution feature of H / 4×W / 4. Of course, the present disclosure does not limit the size of the regional distribution feature. The larger the size, the finer the segmentation of the image to be processed, and the greater the corresponding amount of calculation.

[0066] Continue to refer Figure 1 ,In step S140, an attention mask is obtained by fusing the query information and the regional distribution features.

[0067] Among them, the query information may include pointing or guiding information for a specific target. By fusing it with the distribution features, the regional distribution information for the specific target can be obtained, which is represented as an attention mask, which is used to make the network focus on the specified area in the image when identifying a specific target, which is conducive to improving the accuracy of the image segmentation results.

[0068] In one implementation, the above-mentioned obtaining of the attention mask by fusing the query information and the regional distribution features may include the following steps:

[0069] The query information and the regional distribution features are multiplied, and the multiplied value is binarized based on the attention threshold to obtain the attention mask.

[0070] Among them, the query information and the regional distribution features can be dot-multiplied to obtain a numerical visualization attention value, and then the attention value is binarized based on the attention threshold (which can be determined based on experience or specific needs, such as the range of attention values ​​is generally 0 to 1, and the attention threshold can be 0.5), that is, the attention value greater than or equal to the attention threshold is binarized to 1, and the attention value less than the attention threshold is binarized to 0, thereby obtaining an attention mask. This utilizes the correlation between the query vector and the regional distribution features, which can improve the accuracy of regional distribution recognition.

[0071] Continue to refer Figure 1 ,In step S150, the image features, query information, and attention mask are processed using a transformer decoder to obtain updated query information.

[0072] Figure 4 The exemplary structure of the transformer decoder is shown, which may include a masked cross attention layer, an addition and normalization layer, a self-attention layer, an addition and normalization layer, a feedforward network, and an addition and normalization layer connected in sequence. The addition and normalization layers at different positions may be directly connected to increase the comprehensiveness of the information and improve the nonlinear capability of the transformer decoder.

[0073] The image features, query information, and attention mask are input into the transformer decoder. In the masked cross attention layer, the image features can form a key matrix and a value matrix, and the query information can form a query matrix. The query matrix and the key matrix further form attention weights, and the attention mask can be superimposed on the attention weights to enhance the use of regional distribution information in the image to be processed. Based on the attention mechanism, the value matrix of the image features is weighted by the attention weights, so that the image features are extracted while redistributing the attention positions of the image, and the attention features are obtained. The attention features carry more sufficient regional distribution information. The attention features are further processed by the addition and normalization layers, the self-attention layers, the feedforward network, etc., and the transformer decoder can output the updated query information. Through the processing of the transformer decoder, with the help of the attention mask, the information in the image features is passed to the query information to update the query information so that the query information is more adapted to the image to be processed.

[0074] In one embodiment, the plurality of scales include a first scale to an Mth scale of decreasing size, where M is a positive integer not less than 2. Figure 5As shown, the above-mentioned processing of the image features, the query information, and the attention mask using the transformer decoder to obtain the updated query information may include the following steps S510 and S520:

[0075] Step S510, inputting the image features, query information, and attention mask of the Mth scale into the transformer decoder to obtain the query information after the first update; and fusing the query information after the first update with the regional distribution features to obtain the attention mask after the first update;

[0076] Step S520, input the image features of the Mj-th scale, the query information updated for the jth time, and the attention mask updated for the jth time into the transformer decoder to obtain the query information updated for the j+1th time; obtain the attention mask updated for the j+1th time by fusing the query information updated for the j+1th time and the regional distribution features; j is any positive integer in [1, M-2].

[0077] Among them, the image feature of the Mth scale is the image feature with the smallest size, which also means that the receptive field of its image feature is the largest, that is, the feature is relatively macroscopic. The image feature of the Mth scale, the query information, and the attention mask are input into the transformer decoder. Through the processing of the transformer decoder, the first update of the query information is completed to obtain the query information after the first update. On the basis of updating the query information, the attention mask can also be updated. The implementation method is the same as the principle of step S140, for example, the query information after the first update and the regional distribution feature are fused, and the attention mask after the first update is obtained based on the fusion result.

[0078] Then, the query information after the first update can be updated for the second time using the image features of the M-1th scale, and the attention mask after the first update can be updated for the second time. Compared with the image features of the Mth scale, the size of the image features of the M-1th scale has increased, which means that the receptive field has decreased, that is, the features have changed from macroscopic to microscopic.

[0079] Next, the query information and the attention mask may be updated for the third time and the fourth time by using the image features of the M-2 scale, the image features of the M-3 scale, etc. For example, when M=4, the query information may be updated three times by using the image features of the 4th scale, the image features of the 3rd scale, and the image features of the 2nd scale.

[0080] It should be noted that after the query information is updated using the image features at the second scale, it can be determined that the update process is completed and the final updated query information is obtained, and the query information can be further updated using the image features at the first scale.

[0081] The attention mask can use the result of the last update of the query vector and the attention mask to limit the attention area to the foreground area in the image to be processed, which can effectively reduce the amount of calculation and improve the convergence speed.

[0082] Through the above-mentioned iterative updating method, image features from macro to micro are gradually transferred to the query information, so that the query information gradually learns from general to refined information in the image to be processed, which is conducive to obtaining high-quality query information and improving the updating efficiency of the query information.

[0083] In one embodiment, in each update, the attention mask can be sampled to the same size as the image feature. For example, in the first update, the attention mask is downsampled to the size of the image feature at the Mth scale (e.g., H / 32×W / 32) to achieve dimensional alignment.

[0084] Continue to refer Figure 1 In step S160, an image segmentation result of the image to be processed is obtained based on the updated query information.

[0085] By updating the query information, the image features are passed to the query information, and the query information is further processed. For example, the query information can be processed by a lightweight fully connected network (a network mainly formed by fully connected layers, and can also include convolutional layers) to obtain an image segmentation result.

[0086] In one embodiment, the image segmentation result includes a category prediction result and a region prediction result of one or more objects in the image to be processed. The above-mentioned image segmentation result of the image to be processed based on the updated query information may include the following steps:

[0087] Obtain the target category prediction result based on the updated query information;

[0088] The regional prediction result of the target is obtained based on the updated query information and regional distribution characteristics.

[0089] Among them, the category prediction result is the result of identifying or predicting "what" the target in the image to be processed is. The updated query information can be processed by the lightweight fully connected network mentioned above. For example, the updated query information is in the form of a query vector. Through the fully connected processing, the query vector is mapped to the category prediction output space to obtain the final category prediction result. The regional prediction result is the result of identifying or predicting "where" the target in the image to be processed is distributed. The updated query information and the regional distribution features can be fused, such as multiplying the two (such as dot multiplication) to obtain the regional prediction result of the target. The regional prediction result can be the mask prediction result of the target, that is, predicting which areas or which pixels in the image to be processed are the target. In this way, the semantic segmentation of the image to be processed can be achieved.

[0090] In one embodiment, reference may be made to Figure 6 Update the query information. First, perform a dot multiplication on the query information and the regional distribution features to obtain a mask prediction result, and binarize the mask prediction result to obtain an attention mask. Input the attention mask, query information, and image features into the transformer decoder, and output the updated query information after being processed by the transformer decoder. Then, the updated query information can be dot multiplied with the regional distribution features to obtain an updated mask prediction result, and then binarized to obtain an updated attention mask, so that the next update can be performed. Through iterative updates, the query information continuously learns image features to obtain high-quality query information.

[0091] Figure 7The schematic flow chart of the image segmentation method is shown. The image to be processed is obtained, whose size is H×W, and the initial query information is obtained. The image to be processed is input into the image feature extraction network, and image features of four scales are extracted in sequence, namely H / 4×W / 4 image features, H / 8×W / 8 image features, H / 16×W / 16 image features, and H / 32×W / 32 image features. Then the image features of the four scales are sampled as H / 64×W / 64 image features and spliced ​​along the channel to achieve the fusion of the image features of the four scales. The fused features are input into the transformer block for processing to obtain global semantic features. The image features of each scale and the global semantic features are respectively input into the semantic optimization module. In the semantic optimization module, the semantic optimization weight is obtained according to the global semantic feature, and the normalized image features are weighted based on the semantic optimization weight and added to the normalized global semantic feature to obtain the semantic optimization features of each scale. The semantic optimization features of the four scales are input into the regional distribution extraction module, and the regional distribution features are obtained after processing. Multiply the region distribution features with the initial query information to obtain the initial region prediction result, binarize it and downsample it to H / 32×W / 32 size to obtain the initial attention mask. Input the initial attention mask, the initial query information, and the H / 32×W / 32 image features into the transformer decoder to obtain the first updated query information. Multiply the first updated query information with the region distribution features to obtain the first updated region prediction result, binarize it and downsample it to H / 16×W / 16 size to obtain the first updated attention mask. Input the first updated attention mask, the first updated query information, and the H / 16×W / 16 image features into the transformer decoder to obtain the second updated query information. Multiply the second updated query information with the region distribution features to obtain the second updated region prediction result, binarize it and downsample it to H / 8×W / 8 size to obtain the second updated attention mask. The attention mask after the second update, the query information after the second update, and the H / 8×W / 8 image features are input into the transformer decoder to obtain the query information after the third update, which can be used as the final updated query information. It is multiplied with the regional distribution features to obtain the final regional prediction result. The final updated query information is processed using a fully connected network to obtain the category prediction result of the target in the image to be processed.

[0092] It should be understood that if multiple initial query information is obtained, the above update process can be performed on each query information, and finally the corresponding region prediction results and category prediction results are obtained. By combining the region prediction results and category prediction results corresponding to different query information, the categories and distribution areas of multiple targets in the image to be processed can be obtained, thereby realizing the semantic segmentation of the image to be processed.

[0093] The one or more networks mentioned above can be trained by supervised learning. One or more of cross entropy loss, binary cross entropy loss, Dice loss, L1 loss, and L2 loss can be used to construct a loss function and calculate a gradient value to update the network parameters, thereby realizing the training process.

[0094] As can be seen from the above, when performing image segmentation, although the query information is updated multiple times using the transformer decoder, the query information updated for the last time is finally used to obtain the image segmentation result. In one embodiment, when training the transformer decoder, in order to obtain better performance, the category prediction result and the region prediction result corresponding to the query information can be respectively compared with the category ground truth and the region ground truth in each update, and the prediction result corresponding to the query information with the smallest loss is selected as the output prediction. This achieves bipartite matching, which helps to improve the accuracy and reliability of the prediction results, thereby improving the performance of the entire network.

[0095] In one implementation, the query information may include first query information and second query information; the transformer decoder may include a first transformer decoder and a second transformer decoder; and the image segmentation result may include a semantic segmentation result and an instance segmentation result. Figure 8 As shown, the attention mask is obtained by fusing the query information and the regional distribution features, which may include the following steps S810:

[0096] Step S810, obtaining a first attention mask by fusing the first query information and the regional distribution features; obtaining a second attention mask by fusing the second query information and the regional distribution features.

[0097] The above-mentioned processing of the image features, the query information, and the attention mask by using the transformer decoder to obtain the updated query information may include the following steps S820:

[0098] Step S820, using the first transformer decoder to process the image features, the first query information, and the first attention mask to obtain updated first query information; using the second transformer decoder to process the image features, the second query information, and the second attention mask to obtain updated second query information;

[0099] The above-mentioned obtaining the image segmentation result of the image to be processed based on the updated query information may include the following steps S830:

[0100] Step S830, obtaining a semantic segmentation result based on the updated first query information; and obtaining an instance segmentation result based on the updated second query information.

[0101] Among them, the first query information and the second query information are two types of query information with different functions. The first query information is used to realize the semantic segmentation of the image, and the second query information is used to realize the instance segmentation of the image. Therefore, the information content that these two types of query information need to learn from the image features is also different. The first transformer decoder and the second transformer decoder can be set to update the first query information and the second query information respectively. The structures or parameters of the two transformer decoders can be different. For example, considering that the instance segmentation corresponding to the second transformer decoder requires relatively more information, more intermediate layers can be set for the second transformer decoder. The first transformer decoder can adopt a relatively lightweight structure to reduce the calculation amount of the first transformer decoder. Correspondingly, the attention masks used in the two transformer decoders are also different. The first query information and the regional distribution features can be fused to obtain the first attention mask, which is used to input into the first transformer decoder to update the first query information, and the second query information and the regional distribution features can be fused to obtain the second attention mask, which is used to input into the second transformer decoder to update the second query information. By using different true value data in network training, the first transformer decoder and the second transformer decoder are trained separately, so that the first transformer decoder learns the image feature transfer processing related to semantic segmentation, and the second transformer decoder learns the image feature transfer processing related to instance segmentation. Therefore, when using two transformer decoders to update query information, the first transformer decoder can make the first query information gradually learn the information related to semantic segmentation in the image features, and the second transformer decoder can make the second query information gradually learn the information related to instance segmentation in the image features. After completing the update of the first query information and the second query information, the semantic segmentation result is finally obtained based on the updated first query information, and the instance segmentation result is obtained based on the updated second query information.

[0102] based on Figure 8 The method can realize multi-task image segmentation processing. It should be understood that Figure 8The case where semantic segmentation and instance segmentation are implemented simultaneously is only an exemplary description. According to specific needs, corresponding types of query information can be set, and a corresponding number of transformer decoders can be set and trained. For example, three types of query information can be set, and three transformer decoders can be trained to perform semantic segmentation, instance segmentation, and panoptic segmentation simultaneously.

[0103] In one embodiment, in addition to updating the query information, the image features may also be updated. Exemplarily, the transformer decoder used in step S150 may be recorded as a third transformer decoder (to distinguish the first transformer decoder and the second transformer decoder mentioned above). A fourth transformer decoder may also be provided, and the image features, query information, and attention mask may be input into the fourth transformer decoder, wherein the image features may form a query matrix correspondingly, the query information may form a key matrix and a value matrix, the query matrix may be multiplied by the key matrix, and the attention mask may be added to obtain the attention weight, the value matrix may be weighted using the attention weight, and then processed by other intermediate layers in the fourth transformer decoder to output the updated image features. This makes the image features more accurate and can reduce the amount of computation of the third transformer decoder. Based on the updated image features, one or more steps in steps S130 to S160 may be performed again to achieve the same or better image segmentation effect while reducing the amount of computation.

[0104] It should be noted that updating the image features can be performed before updating the query information, or after updating the query information, or updating the image features and updating the query information can be performed alternately. For example, steps S110 to S150 can be performed first to obtain updated query information, and then the image features can be updated. Based on the updated image features, step S130 can be performed again to obtain updated regional distribution features, and step S160 can be performed based on the updated regional distribution features to improve the quality of the image segmentation results. Alternatively, steps S110 to S150 can be performed first to obtain updated query information, and then the image features can be updated. Based on the updated image features, steps S130 to S150 can be performed again to obtain updated regional distribution features, and the query information can be updated again, and finally step S160 can be performed to improve the quality of the image segmentation results.

[0105] The exemplary embodiment of the present disclosure also provides an image segmentation device. Fig. 9 As shown, the image segmentation device 900 may include the following program units:

[0106] The data acquisition unit 910 is configured to acquire the image to be processed and acquire the query information;

[0107] An image feature extraction unit 920 is configured to extract image features of multiple scales from the image to be processed using an image feature extraction network;

[0108] The regional distribution feature extraction unit 930 is configured to process the image features of multiple scales using a semantic feature processing network to obtain the regional distribution features of the image to be processed;

[0109] The attention mask processing unit 940 is configured to obtain an attention mask by fusing the query information and the regional distribution feature;

[0110] An information updating unit 950 is configured to process the image features, the query information, and the attention mask using a transformer decoder to obtain updated query information;

[0111] The image segmentation result output unit 960 is configured to obtain the image segmentation result of the image to be processed based on the updated query information.

[0112] In one embodiment, the image feature extraction network includes a first feature extraction layer to an Mth feature extraction layer connected in sequence, where M is a positive integer not less than 2; the above-mentioned use of the image feature extraction network to extract image features of multiple scales from the image to be processed includes:

[0113] Inputting the image to be processed into the image feature extraction network, the first feature extraction layer performs feature extraction processing on the image to be processed to obtain image features of a first scale;

[0114] The i-th feature extraction layer performs downsampling and feature extraction on the image features output by the i-1-th feature extraction layer to obtain the image features of the i-th scale, where i is any positive integer in [2,M].

[0115] In one embodiment, the semantic feature processing network includes a semantic extraction module, a feature optimization module, and a regional distribution extraction module; the semantic feature processing network is used to process image features of multiple scales to obtain regional distribution features of the image to be processed, including:

[0116] The image features of multiple scales are input into the semantic feature processing network, and the semantic extraction module converts the image features of multiple scales to the same size and then fuses and processes them to obtain the global semantic features;

[0117] The feature optimization module optimizes the image features at multiple scales based on the global semantic features to obtain semantic injection features at multiple scales.

[0118] The regional distribution extraction module fuses and processes the semantic injection features of multiple scales to obtain the regional distribution features of the image to be processed.

[0119] In one embodiment, the feature optimization module optimizes the image features at multiple scales based on the global semantic features to obtain semantic injection features at multiple scales, including:

[0120] Perform dimension conversion and batch normalization on image features of multiple scales to obtain normalized image features of multiple scales;

[0121] The global semantic features are dimensionally transformed and batch normalized to obtain normalized global semantic features, and the global semantic features are dimensionally transformed, batch normalized, and activated to obtain semantic optimization weights;

[0122] The normalized image features of each scale are weighted based on the semantic optimization weights and then added to the normalized global semantic features to obtain the semantic injection features of each scale.

[0123] In one embodiment, the regional distribution extraction module fuses and processes the semantic injection features of multiple scales to obtain regional distribution features of the image to be processed, including:

[0124] The semantic injection features of multiple scales are sampled to the same size and then fused to obtain the semantic injection fusion features;

[0125] The semantic injection fusion features are convolved to obtain the regional distribution features of the image to be processed.

[0126] In one implementation, the attention mask is obtained by fusing the query information and the regional distribution features, including:

[0127] The query information and the regional distribution features are multiplied, and the multiplied value is binarized based on the attention threshold to obtain the attention mask.

[0128] In one implementation, the multiple scales include a first scale to an Mth scale of decreasing size, where M is a positive integer not less than 2; the transformer decoder is used to process the image features, the query information, and the attention mask to obtain updated query information, including:

[0129] Input the image features, query information, and attention mask of the Mth scale into the transformer decoder to obtain the query information after the first update; and obtain the attention mask after the first update by fusing the query information after the first update with the regional distribution features;

[0130] The image features of the Mj-th scale, the query information after the j-th update, and the attention mask after the j-th update are input into the transformer decoder to obtain the query information after the j+1-th update; the attention mask after the j+1-th update is obtained by fusing the query information after the j+1-th update and the regional distribution features; j is any positive integer in [1, M-2].

[0131] In one embodiment, the image segmentation result includes a category prediction result and a region prediction result of one or more objects in the image to be processed; the image segmentation result of the image to be processed obtained based on the updated query information includes:

[0132] Obtain the target category prediction result based on the updated query information;

[0133] The regional prediction result of the target is obtained based on the updated query information and regional distribution characteristics.

[0134] In one embodiment, the query information includes first query information and second query information; the transformer decoder includes a first transformer decoder and a second transformer decoder; and the image segmentation result includes a semantic segmentation result and an instance segmentation result.

[0135] The above obtains the attention mask by fusing the query information and the regional distribution features, including:

[0136] A first attention mask is obtained by fusing the first query information and the regional distribution feature; a second attention mask is obtained by fusing the second query information and the regional distribution feature;

[0137] The above transformer decoder is used to process image features, query information, and attention mask to obtain updated query information, including:

[0138] Using a first transformer decoder to process the image features, the first query information, and the first attention mask to obtain updated first query information; using a second transformer decoder to process the image features, the second query information, and the second attention mask to obtain updated second query information;

[0139] The image segmentation result of the image to be processed obtained based on the updated query information includes:

[0140] A semantic segmentation result is obtained based on the updated first query information; and an instance segmentation result is obtained based on the updated second query information.

[0141] The specific details of each part of the above-mentioned device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0142] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium, which can be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to cause the electronic device to perform the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. In an optional embodiment, the program product can be implemented as a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0143] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0144] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0145] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0146] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0147] The exemplary embodiment of the present disclosure also provides an electronic device. The electronic device may include a processor and a memory. The memory stores executable instructions of the processor, such as program codes. The processor executes the method in the exemplary embodiment by executing the executable instructions.

[0148] Reference below Fig.10 , the electronic device is exemplarily described in the form of a general-purpose computing device. It should be understood that Fig.10 The electronic device 1000 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0149] like Fig.10 As shown, the electronic device 1000 may include: a processor 1010 , a memory 1020 , a bus 1030 , an I / O (input / output) interface 1040 , and a network adapter 1050 .

[0150] The memory 1020 may include a volatile memory, such as a RAM 1021, a cache unit 1022, and may also include a non-volatile memory, such as a ROM 1023. The memory 1020 may also include one or more program modules 1024, such program modules 1024 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or a certain combination may include the implementation of a network environment. For example, the program module 1024 may include each program unit in the above-mentioned device.

[0151] The bus 1030 is used to realize the connection between different components of the electronic device 1000, and may include a data bus, an address bus, and a control bus.

[0152] The electronic device 1000 can communicate with one or more external devices 1100 (eg, a keyboard, a mouse, an external controller, etc.) through the I / O interface 1040 .

[0153] The electronic device 1000 can communicate with one or more networks through the network adapter 1050. For example, the network adapter 1050 can provide mobile communication solutions such as 3G / 4G / 5G, or provide wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. The network adapter 1050 can communicate with other modules of the electronic device 1000 through the bus 1030.

[0154] although Fig.10 Not shown, other hardware and / or software modules may also be provided in the electronic device 1000, including but not limited to: a display, a microcode, a device driver, a redundant processor, an external disk drive array, a RAID system, a tape drive, and a data backup storage system.

[0155] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0156] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as a system, method or program product. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which may be collectively referred to herein as a "circuit", "module" or "system". Those skilled in the art will readily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0157] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image segmentation method, characterized in that: include: Obtain the image to be processed and obtain query information; Extracting image features of multiple scales from the image to be processed using an image feature extraction network; Processing the image features of the multiple scales using a semantic feature processing network to obtain regional distribution features of the image to be processed; Obtaining an attention mask by fusing the query information with the regional distribution features; Processing the image features, the query information, and the attention mask using a transformer decoder to obtain updated query information; An image segmentation result of the image to be processed is obtained based on the updated query information.

2. The method according to claim 1, characterized in that The image feature extraction network includes a first feature extraction layer to an Mth feature extraction layer connected in sequence, where M is a positive integer not less than 2; and extracting image features of multiple scales from the image to be processed using the image feature extraction network includes: Inputting the image to be processed into the image feature extraction network, and performing feature extraction processing on the image to be processed by the first feature extraction layer to obtain image features of a first scale; The i-th feature extraction layer performs downsampling and feature extraction on the image features output by the i-1-th feature extraction layer to obtain the image features of the i-th scale, where i is any positive integer in [2,M].

3. The method according to claim 1, characterized in that The semantic feature processing network includes a semantic extraction module, a feature optimization module, and a regional distribution extraction module; the semantic feature processing network is used to process the image features of the multiple scales to obtain the regional distribution features of the image to be processed, including: Inputting the image features of the multiple scales into the semantic feature processing network, and the semantic extraction module fuses and processes the image features of the multiple scales to obtain a global semantic feature; The feature optimization module optimizes the image features of the multiple scales based on the global semantic features to obtain the semantic injection features of the multiple scales; The region distribution extraction module fuses and processes the semantic injection features of the multiple scales to obtain the region distribution features of the image to be processed.

4. The method according to claim 3, characterized in that The feature optimization module optimizes the image features of the multiple scales based on the global semantic features to obtain the semantic injection features of the multiple scales, including: Performing dimension conversion and batch normalization processing on the image features of the multiple scales to obtain normalized image features of the multiple scales; The global semantic features are dimensionally transformed and batch normalized to obtain normalized global semantic features, and the global semantic features are dimensionally transformed, batch normalized, and activated to obtain semantic optimization weights; The normalized image features of each scale are weighted based on the semantic optimization weights, and then added to the normalized global semantic features to obtain the semantic injection features of each scale.

5. The method according to claim 3, characterized in that: The regional distribution extraction module fuses and processes the semantic injection features of the multiple scales to obtain the regional distribution features of the image to be processed, including: Sampling the semantic injection features of the multiple scales to the same size and fusing them to obtain semantic injection fusion features; The semantic injection fusion feature is convolved to obtain the regional distribution feature of the image to be processed.

6. The method according to claim 1, characterized in that The attention mask is obtained by fusing the query information and the regional distribution features, including: The query information and the regional distribution feature are multiplied, and the multiplied value is binarized based on the attention threshold to obtain the attention mask.

7. The method according to claim 1, characterized in that The multiple scales include a first scale to an Mth scale of decreasing size, where M is a positive integer not less than 2; the using a transformer decoder to process the image features, the query information, and the attention mask to obtain updated query information includes: Inputting the image features of the Mth scale, the query information, and the attention mask into the transformer decoder to obtain the query information after the first update; and fusing the query information after the first update with the regional distribution features to obtain the attention mask after the first update; The image features of the Mj-th scale, the query information after the j-th update, and the attention mask after the j-th update are input into the transformer decoder to obtain the query information after the j+1-th update; the attention mask after the j+1-th update is obtained by fusing the query information after the j+1-th update and the regional distribution features; j is any positive integer in [1, M-2].

8. The method according to claim 1, characterized in that: The image segmentation result includes a category prediction result and a region prediction result of one or more targets in the image to be processed; the image segmentation result of the image to be processed obtained based on the updated query information includes: Obtaining a category prediction result of the target based on the updated query information; A regional prediction result of the target is obtained based on the updated query information and the regional distribution characteristics.

9. The method according to claim 1, characterized in that: The query information includes first query information and second query information; the transformer decoder includes a first transformer decoder and a second transformer decoder; the image segmentation result includes a semantic segmentation result and an instance segmentation result; The attention mask is obtained by fusing the query information and the regional distribution features, including: A first attention mask is obtained by fusing the first query information and the regional distribution feature; a second attention mask is obtained by fusing the second query information and the regional distribution feature; The using a transformer decoder to process the image features, the query information, and the attention mask to obtain updated query information includes: Using the first transformer decoder to process the image feature, the first query information, and the first attention mask to obtain updated first query information; using the second transformer decoder to process the image feature, the second query information, and the second attention mask to obtain updated second query information; The obtaining the image segmentation result of the image to be processed based on the updated query information includes: The semantic segmentation result is obtained based on the updated first query information; and the instance segmentation result is obtained based on the updated second query information.

10. An image segmentation device, characterized in that: include: A data acquisition unit is configured to acquire an image to be processed and acquire query information; An image feature extraction unit is configured to extract image features of multiple scales from the image to be processed using an image feature extraction network; A regional distribution feature extraction unit is configured to process the image features of the multiple scales using a semantic feature processing network to obtain regional distribution features of the image to be processed; an attention mask processing unit, configured to obtain an attention mask by fusing the query information and the regional distribution feature; An information updating unit is configured to process the image feature, the query information, and the attention mask using a transformer decoder to obtain updated query information; The image segmentation result output unit is configured to obtain the image segmentation result of the image to be processed based on the updated query information.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

12. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 9 by executing the executable instructions.

Images