Image semantic segmentation method and device and computer readable storage medium

Abstract

The embodiment of the invention discloses an image semantic segmentation method and device and a computer storage medium, and belongs to the technical field of computer vision. The method comprises the following steps: determining a plurality of features of an image; determining initial category embedding based on the first feature; determining learned category embedding based on the at least one second feature and the initial category embedding; and determining a semantic segmentation result of the image based on the learned category embedding and the plurality of features. In the embodiment of the invention, category embedding is introduced in the semantic segmentation process, so that the semantic segmentation effect of the image is improved through interaction between the category embedding and each feature in the image. Moreover, the initial category embedding is determined based on one feature of the image instead of random initialization, so that the initial category embedding is fused with the spatial prior information of the image, and compared with the mode that the initial category embedding is randomly initialized to be 0, the semantic segmentation effect of the image can be further improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular to an image semantic segmentation method, device, and computer-readable storage medium. Background Art

[0002] The purpose of image semantic segmentation is to predict the category to which each pixel in an image belongs. With the development of computer vision technology, image semantic segmentation technology has been widely used in scenarios such as autonomous driving and image editing.

[0003] In related technologies, the semantic features of an image can be gradually extracted by stacking convolutional layers based on a fully convolutional network, and a 1x1 convolutional layer is connected at the end to output the final semantic segmentation map. However, this semantic segmentation method is not capable of extracting features, resulting in poor semantic segmentation results. Summary of the invention

[0004] The embodiments of the present application provide a method, device and computer-readable storage medium for image semantic segmentation, which can improve the semantic segmentation effect. The technical solution is as follows:

[0005] In a first aspect, a method for semantic segmentation of an image is provided, in which multiple features of an image are determined, the multiple features including a first feature and at least one second feature; an initial category embedding is determined based on the first feature, the initial category embedding includes N vectors, each of the N vectors indicates a category, and N is a positive integer; a learned category embedding is determined based on at least one second feature and the initial category embedding; a semantic segmentation result of the image is determined based on the learned category embedding and the multiple features, the semantic segmentation result including the category to which each pixel in the image belongs.

[0006] In the embodiment of the present application, category embedding is introduced in the semantic segmentation process. The category embedding includes multiple vectors, each of which is bound to a category, so as to interact with each feature in the image through the category embedding, thereby extracting more information from the image, thereby improving the semantic segmentation effect of the image. In addition, the initial category embedding is determined based on one of the features of the image, rather than randomly initialized, so that the initial category embedding has already integrated the spatial prior information of the image. Compared with directly randomly initializing the initial category embedding to 0, the semantic segmentation method in the embodiment of the present application can further improve the semantic segmentation effect of the image.

[0007] Based on the method provided in the first aspect, in some possible implementations, the implementation process of determining the initial category embedding based on the first feature can be: determining a first image segmentation mask set based on the first feature, the first image segmentation mask set including N first image segmentation masks; performing pooling processing on each of the N first image segmentation masks to obtain N pooling results corresponding one-to-one to the N first image segmentation masks; and determining the initial category embedding based on the N pooling results.

[0008] In the embodiment of the present application, an initial category embedding can be determined by performing pooling processing on each image segmentation mask in the first image segmentation mask set corresponding to the first feature. This method is easy to implement.

[0009] Based on the method provided in the first aspect, in some possible implementations, at least one second feature is sorted according to resolution size; in this scenario, based on at least one second feature and the initial category embedding, the implementation process of determining the learned category embedding can be: for the first second feature after sorting, based on the initial category embedding and the first second feature, determine the image segmentation mask set corresponding to the first second feature, and determine the first updated category embedding based on the image segmentation mask set corresponding to the first second feature, and the image segmentation mask set corresponding to the first second feature includes N image segmentation masks; for the i-th second feature after sorting, i is greater than 1, based on the most recently updated category embedding and the i-th second feature, determine the image segmentation mask set corresponding to the i-th second feature, and determine the i-th updated category embedding based on the image segmentation mask set corresponding to the i-th second feature, and use the last updated category embedding as the learned category embedding.

[0010] Since the initial category embedding is already obtained based on the first feature, it is only necessary to interact the initial category embedding with the remaining second features in sequence to obtain the learned category embedding. In the above implementation, at least one second feature is sorted according to the resolution size, and the initial category embedding is updated in sequence according to the order of each second feature, so as to ensure that the learned category embedding incorporates the spatial prior information of all features, that is, to ensure that the learned category embedding incorporates the global contextual information in the image, thereby improving the subsequent semantic segmentation effect.

[0011] Based on the method provided in the first aspect, in some possible implementations, based on the initial category embedding and the first second feature, the implementation process of determining the image segmentation mask set corresponding to the first second feature can be: performing feature transformation on the N vectors in the initial category embedding respectively to obtain a first transformation feature set corresponding to the initial category embedding, the first transformation feature set including N transformation features corresponding one-to-one to the N vectors, and the transformation features in the first transformation feature set have the same resolution as the first second feature; performing dot product calculation between each transformation feature in the first transformation feature set and the first second feature respectively to obtain an image segmentation mask set corresponding to the first second feature.

[0012] In an embodiment of the present application, the image segmentation mask set corresponding to each second feature can be comprehensively determined based on the second feature and the most recently obtained category embedding, so that the image segmentation mask set corresponding to each second feature can represent more comprehensive information of the image.

[0013] Based on the method provided in the first aspect, in some possible implementations, the implementation process of determining the first updated category embedding based on the image segmentation mask set corresponding to the first second feature can be: based on the image segmentation mask set corresponding to the first second feature, determine the category embedding corresponding to the first second feature; fuse the category embedding corresponding to the first second feature and the initial category embedding to obtain the first updated category embedding.

[0014] In an embodiment of the present application, the category embedding obtained based on the image segmentation mask set corresponding to a second feature and the most recently obtained category embedding can be fused, and the fused category embedding can be used as the updated category embedding so that the updated category embedding can learn more information in the image.

[0015] Based on the method provided in the first aspect, in some possible implementations, the implementation process of determining the semantic segmentation result of an image based on the learned category embedding and multiple features can be: for the first second feature after sorting, based on the initial category embedding and the first second feature, determine the enhanced feature corresponding to the first second feature; for the i-th second feature after sorting, based on the category embedding after the most recent update and the i-th second feature, determine the enhanced feature corresponding to the i-th second feature; based on multiple enhanced features corresponding one-to-one to multiple second features, image segmentation mask sets corresponding to the multiple second features respectively, and the learned category embedding, determine the semantic segmentation result.

[0016] In an embodiment of the present application, while interacting with each second feature through the initial category embedding, at least one second feature can be further transformed based on the category embedding after each update to extract deeper features in each second feature, that is, to obtain an enhanced feature corresponding to each second feature.

[0017] Based on the method provided in the first aspect, in some possible implementations, based on the initial category embedding and the first second feature, the implementation process of determining the enhanced feature corresponding to the first second feature can be: performing two different feature transformations on the initial category embedding respectively, to obtain a first transformed feature set and a second transformed feature set corresponding to the initial category embedding, wherein the first transformed feature set and the second transformed feature set both include N transformed features corresponding one-to-one to the N vectors; using the first second feature as a query parameter, the first transformed feature set as a key parameter, and the second transformed feature set as a value parameter, and determining the enhanced feature corresponding to the first second feature through a multi-head self-attention MSA operation.

[0018] In an embodiment of the present application, each updated category embedding can be interacted with a second feature through the MSA operation in the attention-based feature processing model (Transformer) to obtain an enhanced feature corresponding to the second feature.

[0019] Based on the method provided in the first aspect, in some possible implementations, based on multiple enhanced features corresponding one-to-one to multiple second features, image segmentation mask sets corresponding to the multiple second features respectively, and learned category embeddings, the implementation process of determining the semantic segmentation result can be: based on the image segmentation mask sets corresponding to the multiple second features respectively and the first image segmentation mask set, determining a cascaded image segmentation mask set, the cascaded image segmentation mask set including N image segmentation masks; based on the learned category embedding and the cascaded image segmentation mask set, determining the probability that each pixel in the image belongs to each of the N categories; based on the multiple enhanced features corresponding one-to-one to the multiple second features and the first feature, determining the cascaded features, the resolution of the cascaded features being the same as the resolution of the image; based on the probability that each pixel in the image belongs to each of the N categories and the cascaded features, determining the semantic segmentation result.

[0020] In the above manner, a semantic segmentation result can be obtained based on the multi-scale enhanced features, the multi-scale image segmentation mask set corresponding to each feature, and the learned category embedding.

[0021] Based on the method provided in the first aspect, in some possible implementations, the implementation process of determining multiple features of an image can be: extracting multiple initial features of the image through a backbone network; performing feature transformation on each initial feature to obtain multiple transformed features corresponding one-to-one to the multiple initial features, and the resolution of each initial feature is twice the resolution of the corresponding transformed feature; constructing a diversity feature corresponding to each transformed feature to obtain multiple diversity features corresponding one-to-one to the multiple transformed features, and the resolution of each transformed feature is the same as the resolution of the corresponding diversity feature; splicing each transformed feature with the corresponding diversity feature, and using the spliced ​​feature as a feature corresponding to each transformed feature.

[0022] In the embodiment of the present application, after the initial features of the image are extracted through the backbone network, the transformation features corresponding to the initial features and the diversity features corresponding to the transformation features can be spliced, and the spliced ​​features are used as a feature of the image. This processing method can not only ensure the richness of the extracted features, but also reduce the number of parameters and calculations used to extract the features, thereby efficiently obtaining the features of the image.

[0023] Based on the method provided in the first aspect, in some possible implementations, the implementation process of constructing a diversity feature corresponding to each transformation feature can be: processing each transformation feature through a compression activation SE operation to obtain a diversity feature corresponding to each transformation feature.

[0024] Among them, the SE operation requires relatively few parameters and calculations, thereby improving the efficiency of acquiring image features.

[0025] Based on the method provided in the first aspect, in some possible implementations, the resolution of the first feature is smaller than the resolution of at least one second feature.

[0026] That is, the first feature is the feature with the smallest resolution among multiple features, so that the method provided in the embodiment of the present application can be well integrated with the U-shaped semantic segmentation architecture.

[0027] In a second aspect, an image semantic segmentation device is provided, wherein the image semantic segmentation device has the function of implementing the image semantic segmentation method in the first aspect. The image semantic segmentation device includes at least one module, and the at least one module is used to implement the image semantic segmentation method provided in the first aspect.

[0028] In a third aspect, an image semantic segmentation device is provided, wherein the structure of the image semantic segmentation device includes a processor and a memory, wherein the memory is used to store a program that supports the image semantic segmentation device to execute the image semantic segmentation method provided in the first aspect, and to store data involved in implementing the image semantic segmentation method provided in the first aspect. The processor is configured to execute the program stored in the memory.

[0029] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes the image semantic segmentation method described in the first aspect.

[0030] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the image semantic segmentation method described in the first aspect.

[0031] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here.

[0032] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0033] In the embodiment of the present application, category embedding is introduced in the semantic segmentation process. The category embedding includes multiple vectors, each of which is bound to a category, so as to interact with each feature in the image through the category embedding, thereby extracting more information from the image, thereby improving the semantic segmentation effect of the image. In addition, the initial category embedding is determined based on one of the features of the image, rather than randomly initialized, so that the initial category embedding has already integrated the spatial prior information of the image. Compared with directly randomly initializing the initial category embedding to 0, the semantic segmentation method in the embodiment of the present application can further improve the semantic segmentation effect of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the architecture of an image semantic segmentation system provided in an embodiment of the present application;

[0035] Figure 2 This is a flow chart of an image semantic segmentation method provided by an embodiment of the present application;

[0036] Figure 3 is a functional schematic diagram of a feature reconstruction module 01 provided in an embodiment of the present application;

[0037] Figure 4 is a functional schematic diagram of an explicit category extraction module 02 provided in an embodiment of the present application;

[0038] Figure 5 is a schematic diagram of a process for determining an enhanced feature corresponding to a second feature provided by an embodiment of the present application;

[0039] Figure 6 is a flow chart of another image semantic segmentation method provided in an embodiment of the present application;

[0040] Figure 7 This is a schematic diagram for comparing segmentation effects provided in an embodiment of the present application;

[0041] Figure 8 is another schematic diagram for comparing segmentation effects provided in an embodiment of the present application;

[0042] Fig. 9 is another schematic diagram for comparing segmentation effects provided in an embodiment of the present application;

[0043] Fig.10 It is a structural schematic diagram of an image semantic segmentation device provided in an embodiment of the present application;

[0044] Fig.11 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present application more clear, the implementation mode of the present application will be further described in detail below with reference to the accompanying drawings.

[0046] Before explaining the embodiments of the present application, the application scenarios of the embodiments of the present application are first introduced.

[0047] The image semantic segmentation method provided in the embodiments of the present application can be used in any scenario or task that requires the image to be segmented according to semantics, such as autonomous driving, medical imaging, scene understanding, etc.

[0048] For example, in application scenarios such as autonomous driving, image semantic segmentation is needed to detect the drivable area to ensure that the vehicle is driving within the drivable area. In addition, lane lines and street lights on the road are very important for composition positioning, and these two pieces of information also need to be detected through semantic segmentation. Since the semantic segmentation results will affect the route planning in the autonomous driving scenario, and thus affect the safety of the autonomous driving vehicle, the autonomous driving scenario has high requirements for the accuracy and performance of semantic segmentation.

[0049] For example, current mobile phone photography functions such as background blurring and portrait cropping all rely on semantic segmentation. Unlike semantic segmentation performed in the cloud, due to the limited computing resources of mobile phones and other terminal devices, semantic segmentation technology with low parameter and computational complexity and high accuracy is very necessary and urgent.

[0050] The image semantic segmentation system, image semantic segmentation method and related devices provided in the embodiments of the present application are explained below.

[0051] Figure 1 Schematic diagram of the architecture of an image semantic segmentation system provided in an embodiment of the present application. Figure 1 As shown, the image semantic segmentation system 00 includes a feature reconstruction module 01, an explicit category extraction module 02, and a semantic attention extraction and update module 03.

[0052] Among them, the feature reconstruction module 01 is used to obtain multiple features of the input image (hereinafter referred to as the image). The resolutions of these multiple features are different, so the multiple features can be called multi-scale features or multi-stage features. Among them, the features with large resolution can be called shallow features or large-size features. The display category extraction module 02 is used to determine the initial category embedding based on one of the features of the image, so that the initial category embedding is integrated with the spatial prior information of the image. The semantic attention extraction and update module 03 is used to interact the initial category embedding with other features to update the category embedding, and then obtain the learned category embedding. On the other hand, it is used to determine the semantic segmentation result of the image based on the learned category embedding and multiple features.

[0053] The specific implementation of the functions of each of the above modules will be explained in detail in the subsequent embodiments and will not be described in detail here.

[0054] In addition, the above modules are software modules, which can be centrally deployed on one device, which can be a mobile terminal device, a monitoring device, and a computing device in a cloud service. Optionally, these software modules can also be distributed and deployed in different computing devices of the cloud service. This embodiment of the application is not limited to this.

[0055] Figure 2 This is a flow chart of an image semantic segmentation method provided by an embodiment of the present application. Figure 2 As shown, the method includes the following steps.

[0056] Step 201: Determine a plurality of features of an image, where the plurality of features include a first feature and at least one second feature.

[0057] Step 201 may be performed, for example, by Figure 1 The feature reconstruction module 01 in is implemented.

[0058] In some embodiments, step 201 may be implemented by: extracting multiple initial features of the image through a backbone network; performing feature transformation on each initial feature to obtain multiple transformed features corresponding one-to-one to the multiple initial features, and the resolution of each initial feature is twice the resolution of the corresponding transformed feature; constructing a diversity feature corresponding to each transformed feature to obtain multiple diversity features corresponding one-to-one to the multiple transformed features, and the resolution of each transformed feature is the same as the resolution of the corresponding diversity feature; splicing each transformed feature with the corresponding diversity feature, and using the spliced ​​feature as a feature corresponding to each transformed feature.

[0059] In the embodiment of the present application, after the initial features of the image are extracted through the backbone network, the transformation features corresponding to the initial features and the diversity features corresponding to the transformation features can be spliced, and the spliced ​​features are used as a feature of the image. This processing method can not only ensure the richness of the extracted features, but also reduce the number of parameters and calculations used to extract the features, thereby efficiently obtaining the features of the image.

[0060] Among them, the backbone network (backbone) can be implemented by a convolutional neural network (CNN) for example, or by a model (Transformer) that processes features based on attention, which is not limited in the embodiments of the present application.

[0061] To facilitate subsequent understanding, CNN and the model (Transformer) that processes features based on attention are explained here.

[0062] CNN is a feature extraction network structure based on convolutional layers. The convolutional layer can extract spatial features in the image, such as edges, corners, etc. Adding a pooling layer after the convolutional layer can further reduce the size of the feature map and reduce the amount of calculation. CNN can also classify and recognize complex images by combining multiple convolutional layers and fully connected layers. Among them, the main operation in CNN is the convolution operation.

[0063] The model (Transformer) that processes features based on attention is a feature extraction network structure based on the self-attention mechanism. The self-attention mechanism can perform attention calculations on each position in the input sequence (or image) to obtain global context information. The main operation in the model (Transformer) that processes features based on attention is the self-attention operation.

[0064] In addition, feature transformation is performed on each initial feature to obtain multiple transformed features corresponding to the multiple initial features. This step can be implemented through a convolution operation or a self-attention operation.

[0065] In addition, the implementation method of constructing the diversity feature corresponding to each transformation feature may be: processing each transformation feature through a squeeze / excitation (SE) operation to obtain the diversity feature corresponding to each transformation feature.

[0066] Among them, the SE operation requires relatively few parameters and calculations, thereby improving the efficiency of acquiring image features.

[0067] For example, the compression operation can be understood as: compressing the transformation feature into a feature vector through a global average pooling operation, and the dimension of the feature vector is the same as the number of channels of the transformation feature. The activation operation can be understood as: processing the feature vector using a fully connected layer to predict the importance of each channel of the transformation feature, that is, generating a channel weight vector, and applying the channel weight vector to the transformation feature through an activation function, and the resulting feature is the diversity feature.

[0068] Furthermore, before each transformation feature is processed by the SE operation, the transformation feature can also be processed by a cheap operation. Diversity features are determined by the SE operation based on the processed transformation features. This can further improve the richness of the diversity features. Among them, the cheap operation can be implemented by a fully connected layer.

[0069] In addition, when each transformation feature and the corresponding diversity feature are spliced ​​together, the spliced ​​features can be further subjected to a convolution process, and the convolution-processed features are used as the features obtained in step 201. The purpose of the convolution process is to allow the transformation features and the diversity features in the spliced ​​features to be cross-distributed, thereby improving the richness of the features of the acquired image.

[0070] Figure 3 01 is a functional diagram of a feature reconstruction module 01 provided in an embodiment of the present application. Assume that the backbone network outputs 4 initial features, which are marked as x1, x2, x3 and x4. Among them, the resolutions of x1, x2, x3 and x4 decrease successively, that is, the sizes of these 4 initial features decrease successively. For each of the initial features x1, x2, x3 and x4, Figure 3 The feature reconstruction scheme shown is used to obtain features that correspond one to one with the four initial features, which are marked as X1, X2, X3 and X4 respectively.

[0071] Here, x1 is taken as an example to illustrate the feature reconstruction scheme. Figure 3 As shown, assume that the size of x1 is C×H×W. Among them, C is the number of channels of x1, and H×W is the resolution of x1. x1 is subjected to a layer of convolution for feature transformation to obtain the transformed feature of x1. The size of the transformed feature is exemplarily C×H / 2×W or C×H×W / 2. A copy of the transformed feature is used as an inherent feature, and at the same time, the transformed feature is subjected to a computationally cheap operation and SE operation to generate a diversity feature. Finally, the inherent feature and the diversity feature are concatenated together and convolved, and the final output feature is used as a reconstructed feature corresponding to x1, that is, X1. The reconstructed feature is a feature obtained in step 201.

[0072] like Figure 3 As shown, the inherent features and the diversity features are convolved after being concatenated, and the inherent features and the diversity features can be cross-distributed.

[0073] Optionally, in some other embodiments, step 201 can be implemented by: extracting multiple initial features of the image through a backbone network; performing a layer of convolution transformation on each initial feature to obtain a transformed feature with the same resolution as the corresponding initial feature, and using multiple transformed features corresponding one-to-one to the multiple initial features as the multiple features obtained in step 201.

[0074] For the convenience of subsequent description, one of the multiple features is referred to as the first feature, and the other features are referred to as the second features, so that the initial category embedding can be determined by the first feature, and then updated by the other second features.

[0075] In some embodiments, the resolution of the first feature is less than the resolution of at least one second feature. In other words, the first feature is the feature with the smallest resolution among the multiple features. For example, the first feature may be the aforementioned feature X4. This allows the method provided in the embodiment of the present application to be well integrated with the U-shaped semantic segmentation architecture.

[0076] Optionally, the first feature may also be a feature with the highest resolution among multiple features. For example, the first feature may be the aforementioned feature X1.

[0077] Step 202: Determine an initial category embedding based on the first feature, where the initial category embedding includes N vectors, each of the N vectors indicates a category, and N is a positive integer.

[0078] Step 201 may be performed, for example, by Figure 1 The explicit category extraction module 02 in is implemented.

[0079] In some embodiments, determining the initial category embedding based on the first feature may be implemented by: determining a first image segmentation mask set based on the first feature, the first image segmentation mask set including N first image segmentation masks; performing pooling processing on each of the N first image segmentation masks to obtain N pooling results corresponding one-to-one to the N first image segmentation masks; and determining the initial category embedding based on the N pooling results.

[0080] N is a pre-input entity category to be identified, and N can be specified by the user.

[0081] Among them, the implementation method of determining the first image segmentation mask set based on the first feature can be exemplarily: the first feature is processed through a fully connected layer to obtain a first image segmentation mask set including N first image segmentation masks. In addition, the implementation method of determining the initial category embedding based on the N pooling results can also be exemplarily: the N pooling results are processed through a fully connected layer to obtain N vectors corresponding to the N pooling results one by one, and the N vectors constitute the initial category embedding.

[0082] In addition, when performing pooling processing on each of the N first image segmentation masks, the step size and pooling method used for pooling are not limited. For example, the pooling method can be maximum pooling or average pooling.

[0083] In the embodiment of the present application, an initial category embedding can be determined by performing pooling processing on each image segmentation mask in the first image segmentation mask set corresponding to the first feature. This method is easy to implement.

[0084] Figure 4 FIG. 1 is a functional schematic diagram of an explicit category extraction module 02 provided in an embodiment of the present application. Figure 4 As shown, assuming that N is 4, 4 first image segmentation masks can be obtained based on the first feature. Figure 4 In the figure, the four first image segmentation masks are marked as 1, 2, 3 and 4 from left to right. Each of the four first segmentation image masks is divided into multiple small blocks. For any first segmentation image mask, each of the multiple small blocks is pooled to obtain a pooling result, which includes multiple pooled features corresponding to the multiple small blocks. Then, the four pooling results are respectively transformed by a fully connected layer (FC) to obtain four one-dimensional vectors corresponding to the four pooling results. These four vectors constitute the initial category embedding.

[0085] like Figure 4As shown, the four one-dimensional vectors are marked with four squares. Since these four one-dimensional vectors correspond one-to-one to the four first image segmentation masks, Figure 4 The four squares corresponding to the four one-dimensional vectors are also marked as 1, 2, 3 and 4 respectively.

[0086] In addition, you can Figure 4 The process from image segmentation mask set to class embedding is marked as explicit class embedding (ECE).

[0087] After obtaining the first image segmentation mask set, an initial category embedding can be obtained through a pooling operation and a fully connected layer. Optionally, other operations such as deep learning can be used to learn the N first image segmentation masks in the first image segmentation mask set to obtain N vectors corresponding to the N first image segmentation masks, i.e., the initial category embedding. Examples are not given one by one here.

[0088] Step 203: Determine a learned category embedding based on at least one second feature and the initial category embedding.

[0089] In some embodiments, at least one second feature is sorted by resolution. In this scenario, the implementation method of determining the learned category embedding based on at least one second feature and the initial category embedding can be: for the first second feature after sorting, based on the initial category embedding and the first second feature, determine the image segmentation mask set corresponding to the first second feature, and determine the first updated category embedding based on the image segmentation mask set corresponding to the first second feature; for the i-th second feature after sorting, i is greater than 1, based on the most recently updated category embedding and the i-th second feature, determine the image segmentation mask set corresponding to the i-th second feature, and based on the image segmentation mask set corresponding to the i-th second feature, determine the i-th updated category embedding, and use the last updated category embedding as the learned category embedding.

[0090] Since the initial category embedding is already obtained based on the first feature, we only need to interact the initial category embedding with the remaining second features in sequence to obtain the learned category embedding.

[0091] In the above implementation, at least one second feature is sorted according to the resolution size, and the initial category embedding is updated in sequence according to the order of each second feature, so as to ensure that the learned category embedding integrates the spatial prior information of all features, that is, to ensure that the learned category embedding integrates the global contextual information in the image, thereby improving the subsequent semantic segmentation effect.

[0092] For example, when the first feature is the feature with the smallest resolution among multiple features, at least one second feature is sorted in order of resolution from small to large. For example, if the first feature is X4 among the aforementioned X1, X2, X3 and X4, the initial category embedding is interacted with each feature in the order from X3 to X2 and then to X1, so as to update the initial category embedding and obtain the learned category embedding. In this way, the image semantic segmentation architecture of the embodiment of the present application can be well integrated with the U-shaped semantic segmentation architecture.

[0093] Optionally, when the first feature is the feature with the highest resolution among the multiple features, at least one second feature is sorted in descending order of resolution. For example, if the first feature is X1 among the aforementioned X1, X2, X3 and X4, the initial category embedding is interacted with each feature in order from X2 to X3 and then to X4, so as to update the initial category embedding and obtain the learned category embedding.

[0094] Among them, based on the initial category embedding and the first second feature, an implementation method for determining the image segmentation mask set corresponding to the first second feature can be exemplified as follows: feature transformation is performed on the N vectors in the initial category embedding respectively to obtain a first transformation feature set corresponding to the initial category embedding, the first transformation feature set includes N transformation features corresponding one-to-one to the N vectors, and the transformation features in the first transformation feature set have the same resolution as the first second feature; dot product calculation is performed between each transformation feature in the first transformation feature set and the first second feature respectively to obtain an image segmentation mask set corresponding to the first second feature.

[0095] Among them, the feature transformation of the N vectors in the initial category embedding can be achieved through a fully connected layer to transform the spatial dimension of each of the N vectors, thereby obtaining a transformed feature with the same resolution as the first and second features.

[0096] In addition, since the transformation features in the first transformation feature set have the same resolution as the first and second features, each transformation feature in the first transformation feature set can be directly dot-producted with the first and second features to obtain N image segmentation masks, which constitute an image segmentation mask set corresponding to the first and second features.

[0097] For example, the first transformation feature set is marked as K, and the first second feature is marked as Q. Then, the image segmentation mask set corresponding to the first second feature can be determined by the following formula:

[0098]

[0099] Among them, Mask(χ i-1,δ) represents the image segmentation mask set corresponding to the first and second features, χ i-1 represents the pixel in the first second feature, δ represents the initial category embedding, which can be called the old category embedding. S(Q,K) indicates the dot product calculation of the first second feature Q and the first transformed feature set K.

[0100] Accordingly, the implementation method of determining the image segmentation mask set corresponding to the i-th second feature based on the most recently updated category embedding and the i-th second feature can refer to the implementation method of determining the image segmentation mask set corresponding to the first second feature based on the initial category embedding and the first second feature. Specifically, feature transformation is performed on the N vectors in the most recently updated category embedding to obtain a first transformed feature set corresponding to the most recently updated category embedding, the first transformed feature set includes N transformed features corresponding one-to-one to the N vectors, and the transformed features in the first transformed feature set have the same resolution as the i-th second feature; each transformed feature in the first transformed feature set is dot-producted with the i-th second feature to obtain an image segmentation mask set corresponding to the i-th second feature. The detailed implementation method will not be repeated here.

[0101] In an embodiment of the present application, the image segmentation mask set corresponding to each second feature can be comprehensively determined based on the second feature and the most recently obtained category embedding, so that the image segmentation mask set corresponding to each second feature can represent more comprehensive information of the image.

[0102] In addition, an exemplary implementation method of determining the first updated category embedding based on the image segmentation mask set corresponding to the first second feature can be: based on the image segmentation mask set corresponding to the first second feature, determine the category embedding corresponding to the first second feature; fuse the category embedding corresponding to the first second feature and the initial category embedding to obtain the first updated category embedding.

[0103] Among them, based on the image segmentation mask set corresponding to the first and second features, the category embedding corresponding to the first and second features is determined, which can also be achieved through pooling operations and fully connected layers. For detailed implementation methods, please refer to Figure 4 The process shown will not be repeated here.

[0104] For example, Figure 4 The process from the image segmentation mask set to the category embedding is marked as explicit class embedding (ECE). Based on the image segmentation mask set corresponding to the first second feature, the category embedding corresponding to the first second feature can be determined by the following formula:

[0105]

[0106] in, Represents the category embedding corresponding to the first and second features, which can be called the new category embedding.

[0107] In the embodiment of the present application, when fusing the category embedding corresponding to the first second feature and the initial category embedding, the fusion can be performed in a skip connection form, which is also called a residual form. Optionally, the fusion can also be performed in other forms. The following is an example of fusion in the form of skip connection.

[0108] In some embodiments, the process of fusing the category embedding corresponding to the first second feature and the initial category embedding in the form of skip connections can be: the category embedding corresponding to the first second feature and the initial category embedding, that is, the new category embedding and the old category embedding, are respectively transformed through a fully connected layer so that the transformed new category embedding and the old category embedding have the same size, and the transformed new category embedding and the old category embedding are dot-producted, and then the result of the dot product calculation is skip-connected, and the result obtained is the updated category embedding.

[0109] For example, the above process can be implemented by the following two formulas respectively:

[0110]

[0111] in, It means that the new category embedding is transformed through the fully connected layer, φ4(δ) means that the old category embedding is transformed through the fully connected layer, and ⊙ means the dot product calculation.

[0112] U=σ(ψ1(F u ))

[0113]

[0114] Among them, ψ1(F u ) represents the F u Transform and normalize the transformed features. σ represents the transformation of ψ1(F u ) for processing. Represents a fully connected layer The transformed features are normalized. represents the fused category embedding, that is, the updated category embedding.

[0115] Accordingly, the implementation method of determining the i-th updated category embedding based on the image segmentation mask set corresponding to the i-th second feature can refer to the aforementioned implementation method of determining the first updated category embedding based on the image segmentation mask set corresponding to the first second feature. Specifically, based on the image segmentation mask set corresponding to the i-th second feature, determine the category embedding corresponding to the i-th second feature; fuse the category embedding corresponding to the i-th second feature and the category embedding after the most recent update to obtain the i-th updated category embedding. The detailed implementation method will not be repeated here.

[0116] In an embodiment of the present application, the category embedding obtained based on the image segmentation mask set corresponding to a second feature and the most recently obtained category embedding can be fused, and the fused category embedding can be used as the updated category embedding so that the updated category embedding can learn more information in the image.

[0117] Optionally, in some other embodiments, after obtaining the category embedding corresponding to the ith feature in the above manner, the category embedding corresponding to the ith feature may be directly used as the ith updated category embedding.

[0118] Optionally, in some other embodiments, the image segmentation mask set corresponding to each second feature can be determined based only on each second feature, and then the category embedding corresponding to each second feature can be determined. In this scenario, it is necessary to fuse the category embedding corresponding to each second feature with the most recently updated category embedding, and the fused category embedding is used as the updated category embedding.

[0119] Step 204: Determine a semantic segmentation result of the image based on the learned category embedding and multiple features. The semantic segmentation result includes the category to which each pixel in the image belongs.

[0120] In some embodiments, in a scenario where at least one second feature is sorted according to resolution size, step 204 may be implemented as follows: for the first second feature after sorting, based on the initial category embedding and the first second feature, determine the enhanced feature corresponding to the first second feature; for the i-th second feature after sorting, based on the category embedding after the most recent update and the i-th second feature, determine the enhanced feature corresponding to the i-th second feature; based on multiple enhanced features corresponding one-to-one to multiple second features, image segmentation mask sets corresponding to the multiple second features respectively, and the learned category embedding, determine the semantic segmentation result.

[0121] In an embodiment of the present application, while interacting with each second feature through the initial category embedding, at least one second feature can be further transformed based on the category embedding after each update to extract deeper features in each second feature, that is, to obtain an enhanced feature corresponding to each second feature.

[0122] Among them, based on the initial category embedding and the first second feature, the implementation method of determining the enhanced feature corresponding to the first second feature can be: performing two different feature transformations on the initial category embedding respectively, to obtain a first transformation feature set and a second transformation feature set corresponding to the initial category embedding, the first transformation feature set and the second transformation feature set both include N transformation features corresponding one-to-one to N vectors; using the first second feature as a query parameter, the first transformation feature set as a key parameter, and the second transformation feature set as a value parameter, and determining the enhanced feature corresponding to the first second feature through a multi-head selfattention (MSA) operation.

[0123] In an embodiment of the present application, each updated category embedding can be interacted with a second feature through the MSA operation in the attention-based feature processing model (Transformer) to obtain an enhanced feature corresponding to the second feature.

[0124] The sizes of the transformation features in the first transformation feature set and the second transformation feature set are both the same as the sizes of the corresponding second features.

[0125] Among them, MSA is a multi-head attention operation that extends the self-attention mechanism to multiple heads, thereby enhancing the attention to different features. The input of MSA includes three parameters: query parameter (query), key parameter (key) and value parameter (value). For a given query parameter, MSA will dot product the key parameter and the query parameter, and then multiply the dot product result by the value parameter for output. The multi-head mechanism of MSA can effectively improve the expressiveness of the extracted features, and can also learn more diverse and complex features.

[0126] Figure 5 is a flow chart of determining an enhanced feature corresponding to a second feature provided by an embodiment of the present application. Figure 5As shown in the figure, a second feature is used as Q (i.e., query parameter), and the category embedding after the most recent update is transformed by different fully connected layers, and the transformed features are respectively used as K (i.e., key parameter) and V (i.e., value parameter). Q, K, and V are subjected to MSA operation to obtain the features output by the MSA operation, and the features output by the MSA operation are normalized by the normalization layer (LayerNorm), and the normalized features are transformed by a multilayer perceptron (MLP) composed of fully connected layers to output the final enhanced features.

[0127] Among them, through the MSA operation, determining the enhanced feature corresponding to the first second feature can be implemented by the following formula:

[0128]

[0129] MSA(δ,χ i-1 )=Softmax(S(Q,K))V

[0130] Where S(Q,K) represents the result of dot product of key parameter and query parameter. i-1 ) represents the features output by the MSA operation.

[0131] In addition, if Figure 5 As shown, by performing dot product calculation on Q and K, the image segmentation mask set corresponding to the second feature can be obtained.

[0132] Optionally, determining the enhanced feature corresponding to the first second feature based on the initial category embedding and the first second feature may also be implemented by other methods such as convolution operation, which will not be described in detail here.

[0133] Accordingly, the enhanced feature corresponding to the ith second feature can be determined based on the most recently updated category embedding and the ith second feature by referring to the aforementioned implementation method of determining the enhanced feature corresponding to the first second feature, which will not be repeated here.

[0134] In addition, based on multiple enhanced features corresponding one-to-one to multiple second features, image segmentation mask sets corresponding to the multiple second features respectively, and the learned category embedding, the implementation method for determining the semantic segmentation result can be: based on the image segmentation mask sets corresponding to the multiple second features respectively and the first image segmentation mask set, determine a cascaded image segmentation mask set, the cascaded image segmentation mask set includes N image segmentation masks; based on the learned category embedding and the cascaded image segmentation mask set, determine the probability that each pixel in the image belongs to each of the N categories; based on the multiple enhanced features corresponding one-to-one to the multiple second features and the first feature, determine the cascade feature; based on the probability that each pixel in the image belongs to each of the N categories and the cascade feature, determine the semantic segmentation result.

[0135] In the above manner, a semantic segmentation result can be obtained based on the multi-scale enhanced features, the multi-scale image segmentation mask set corresponding to each feature, and the learned category embedding.

[0136] Among them, based on the image segmentation mask sets corresponding to multiple second features and the first image segmentation mask set, the implementation method of determining the cascaded image segmentation mask can be: superimposing the image segmentation masks of the same category in N categories corresponding to different image segmentation mask sets respectively, to obtain N cascaded image segmentation masks, and the N cascaded image segmentation masks constitute a cascaded image segmentation mask set.

[0137] When superimposing image segmentation masks of the same category, it is necessary to convert image segmentation masks in different image segmentation mask sets to the same size. Specifically, this can be achieved through a fully connected layer, which will not be described in detail here.

[0138] In addition, assuming that the size of the first feature is the smallest, based on multiple enhanced features corresponding one-to-one to multiple second features and the first feature, the implementation method of determining the cascade feature can be: up-sampling the first feature and the non-largest enhanced feature among the multiple enhanced features respectively to obtain the first feature and other enhanced features of the same size as the largest enhanced feature, and then cascading all the up-sampled features with the largest enhanced feature to obtain the cascade feature.

[0139] In addition, based on the learned category embedding and the cascaded image segmentation mask set, the probability of each pixel in the image belonging to each of the N categories can be determined by transforming the learned category embedding through a fully connected layer to obtain a feature of the same size as each image segmentation mask in the cascaded image segmentation mask set, and performing dot product calculations on the feature and each image segmentation mask in the cascaded image segmentation mask set. The dot product result can represent the probability that each pixel in the image belongs to each of the N categories.

[0140] In addition, the cascade features are subjected to a layer of convolution processing, and the features after the convolution processing are added to the dot product calculation results, and then the semantic segmentation results are output. Among them, the cascade features can basically restore the original input image by performing a layer of convolution processing. The features after the convolution processing are added to the dot product calculation results, and the output semantic segmentation results can include the probability that each pixel in the image belongs to each category in the N categories. When displaying the semantic segmentation results, for any pixel in the image, the category with the highest probability can be used as the category to which the pixel belongs.

[0141] Optionally, in other embodiments, deeper features in each second feature may not be extracted, but the image segmentation result may be determined directly based on the multiple features obtained in step 201, the image segmentation mask sets corresponding to the multiple features, and the learned category embedding. For the detailed implementation, reference may be made to the aforementioned implementation of determining the semantic segmentation result based on the multiple enhanced features corresponding to the multiple second features, the image segmentation mask sets corresponding to the multiple second features, and the learned category embedding, which will not be described in detail here.

[0142] Below Figure 6 Take the image semantic segmentation method provided in the embodiment of the present application as an example to illustrate.

[0143] like Figure 6 As shown, given an input image, the backbone network first extracts features at different stages, and the features at different stages are extracted through Figure 3 The feature reconstruction module shown in the figure further abstracts the function and obtains the final features of different stages (i.e. Figure 6 Based on the feature X4 (as the first feature) in the last stage, a fully connected layer is used to output the first image segmentation mask set corresponding to the feature X4, and then Figure 4 The ECE process of the display category extraction module shown in the figure outputs the initial category embedding. The initial category embedding is then passed through Figure 6 The SA module and update (update, U) module in the image processing unit interact with features X3, X2, and X1 (as three second features) in turn, so as to output the enhanced features corresponding to each second feature based on the SA module output, and output the updated category embedding through the U module. The final output category embedding (i.e., the learned category embedding) is then transformed through a fully connected layer (FC), and the transformed features are multiplied with the cascaded image segmentation mask set corresponding to the four features. The multiplication result is added to the features transformed by the cascaded features through the convolutional layer (conv) to obtain the final semantic segmentation map.

[0144] For a given image, Figure 6The semantic segmentation method shown in the figure updates the category embeddings based on the image segmentation mask set from deep features to shallow features (i.e., from X4 to X3 to X2 to X1) in sequence to enhance the expressiveness of the intermediate layer features. At the same time, the multi-scale enhanced features, the multi-scale image segmentation mask set and the learned category embedding are combined to jointly generate the semantic segmentation results.

[0145] In summary, in the embodiment of the present application, category embedding is introduced in the semantic segmentation process. The category embedding includes multiple vectors, each of which is bound to a category, so as to interact with each feature in the image through the category embedding, thereby extracting more information from the image, thereby improving the semantic segmentation effect of the image. In addition, the initial category embedding is determined based on one of the features of the image, rather than randomly initialized, so that the initial category embedding has already integrated the spatial prior information of the image. Compared with directly randomly initializing the initial category embedding to 0, the semantic segmentation method in the embodiment of the present application can further improve the semantic segmentation effect of the image.

[0146] like Figure 1 As shown, the embodiment of the present application also proposes a new segmentation network that uses a segmentation binary map of image features (i.e., image segmentation mask) to initialize category embedding. The segmentation network consists of a feature reconstruction module, an explicit category extraction module, and a semantic attention extraction and update module.

[0147] Among them, the feature reconstruction module is used to ensure the diversity of features extracted by the backbone network. The explicit category extraction module is exemplarily composed of a spatial pooling layer and a linear projection layer (i.e., a fully connected layer) to extract the initial category embedding from the image segmentation mask. The semantic attention extraction and update module is exemplarily implemented by a model (Transformer) that processes features based on attention, so that the initial class embedding extracted by the explicit category extraction module is input into the model (Transformer) that processes features based on attention. The model (Transformer) that processes features based on attention sequentially uses shallow features (such as the aforementioned X3, X2, and X1) as query parameters, and transforms the most recently obtained category embedding as key parameters and value parameters to interact with shallow features through MSA operations, that is, encourages shallow features to interact with category embeddings with spatial information, thereby realizing the extraction of enhanced features of shallow features and continuing to update the most recently obtained category embeddings.

[0148] To further illustrate the technical effect of the embodiments of the present application. Experiments were conducted on three classic datasets using different image semantic segmentation methods. The three datasets are the ADE20K dataset (a scene dataset with 20,000 images and 150 types of targets), the PASCAL-Context dataset (a total of 459 annotated categories, containing 10,103 images, of which 4,998 are used for training sets and 5,105 are used for validation sets), and the Cityscapes dataset (also known as an urban landscape dataset, which contains various stereoscopic video sequences recorded from street scenes in 50 different cities). The experimental results are as follows: Figure 7 , Figure 8 and Fig. 9 Based on Figure 7 , Figure 8 and Fig. 9 As shown in the comparison, the image semantic segmentation method provided in the embodiment of the present application achieves a higher segmentation effect (mIoU) with less computational effort (GFLOPs) and fewer parameters (param).

[0149] It should be noted that Figure 7 , Figure 8 and Fig. 9 The other semantic segmentation methods except the embodiment of the present application are several classic language segmentation methods at present, and the embodiment of the present application will not be explained one by one here. In addition, the backbone network used in each semantic segmentation method is also a classic backbone network, which will not be explained here.

[0150] Fig.10 Schematic diagram of the structure of an image semantic segmentation device provided in an embodiment of the present application. Fig.10 As shown, the device 1000 includes:

[0151] The first determination module 1001 is used to determine multiple features of the image, where the multiple features include a first feature and at least one second feature. For specific implementation methods, please refer to Figure 2 Step 201 in the embodiment.

[0152] The second determination module 1002 is used to determine the initial category embedding based on the first feature, where the initial category embedding includes N vectors, each of which indicates a category, and N is a positive integer; the specific implementation method can be referred to Figure 2 Step 202 in the embodiment.

[0153] The third determination module 1003 is used to determine the learned category embedding based on at least one second feature and the initial category embedding; the specific implementation method can refer to Figure 2 Step 203 in the embodiment.

[0154] The fourth determination module 1004 is used to determine the semantic segmentation result of the image based on the learned category embedding and multiple features. The semantic segmentation result includes the category to which each pixel in the image belongs. For specific implementation methods, please refer to Figure 2 Step 204 in an embodiment.

[0155] Optionally, the second determining module is used to:

[0156] Determine a first image segmentation mask set based on the first feature, where the first image segmentation mask set includes N first image segmentation masks;

[0157] Performing pooling processing on each of the N first image segmentation masks to obtain N pooling results corresponding to the N first image segmentation masks one by one;

[0158] Based on the N pooling results, the initial category embedding is determined.

[0159] Optionally, at least one second feature is sorted according to resolution size;

[0160] The third determination module is used for:

[0161] For the sorted first second feature, based on the initial category embedding and the first second feature, determine an image segmentation mask set corresponding to the first second feature, and determine a first updated category embedding based on the image segmentation mask set corresponding to the first second feature, the image segmentation mask set corresponding to the first second feature includes N image segmentation masks;

[0162] For the sorted i-th second feature, i is greater than 1, based on the most recently updated category embedding and the i-th second feature, determine the image segmentation mask set corresponding to the i-th second feature, and based on the image segmentation mask set corresponding to the i-th second feature, determine the i-th updated category embedding, and use the last updated category embedding as the learned category embedding.

[0163] Optionally, the third determining module is used to:

[0164] Perform feature transformation on the N vectors in the initial category embedding respectively to obtain a first transformed feature set corresponding to the initial category embedding, the first transformed feature set including N transformed features corresponding one-to-one to the N vectors, and the transformed features in the first transformed feature set have the same resolution as the first second feature;

[0165] A dot product calculation is performed between each transformed feature in the first transformed feature set and the first second feature, to obtain an image segmentation mask set corresponding to the first second feature.

[0166] Optionally, the third determining module is used to:

[0167] Determine a category embedding corresponding to the first second feature based on the image segmentation mask set corresponding to the first second feature;

[0168] The category embedding corresponding to the first second feature and the initial category embedding are fused to obtain the first updated category embedding.

[0169] Optionally, the fourth determining module is used to:

[0170] For the sorted first second feature, based on the initial category embedding and the first second feature, determining an enhanced feature corresponding to the first second feature;

[0171] For the sorted i-th second feature, based on the most recently updated category embedding and the i-th second feature, determine an enhanced feature corresponding to the i-th second feature;

[0172] A semantic segmentation result is determined based on at least one enhanced feature corresponding one-to-one to at least one second feature, an image segmentation mask set corresponding to each of the at least one second feature, and the learned category embedding.

[0173] Optionally, the fourth determining module is used to:

[0174] Perform two different feature transformations on the initial category embedding respectively to obtain a first transformation feature set and a second transformation feature set corresponding to the initial category embedding, wherein the first transformation feature set and the second transformation feature set both include N transformation features corresponding one-to-one to the N vectors;

[0175] The first second feature is used as the query parameter, the first transformed feature set is used as the key parameter, and the second transformed feature set is used as the value parameter. Through the multi-head self-attention MSA operation, the enhanced feature corresponding to the first second feature is determined.

[0176] Optionally, the fourth determining module is used to:

[0177] Determine a cascaded image segmentation mask set based on the image segmentation mask sets corresponding to the at least one second feature and the first image segmentation mask set, wherein the cascaded image segmentation mask set includes N image segmentation masks;

[0178] Based on the learned category embedding and the cascaded image segmentation mask set, determine the probability that each pixel in the image belongs to each of the N categories;

[0179] Determine a cascade feature based on at least one enhanced feature corresponding one-to-one to at least one second feature and the first feature, wherein a resolution of the cascade feature is the same as a resolution of the image;

[0180] The semantic segmentation result is determined based on the probability that each pixel in the image belongs to each of the N categories and the cascade features.

[0181] Optionally, the first determining module is used to:

[0182] Extract multiple initial features of the image through the backbone network;

[0183] Perform feature transformation on each initial feature to obtain multiple transformed features corresponding to the multiple initial features, and the resolution of each initial feature is twice the resolution of the corresponding transformed feature;

[0184] Constructing a diversity feature corresponding to each transformation feature to obtain a plurality of diversity features corresponding to the plurality of transformation features one by one, wherein the resolution of each transformation feature is the same as the resolution of the corresponding diversity feature;

[0185] Each transformation feature is concatenated with the corresponding diversity feature, and the concatenated feature is used as a feature corresponding to each transformation feature.

[0186] Optionally, the first determining module is used to:

[0187] Each transformed feature is processed by the compression activation SE operation to obtain the diversity feature corresponding to each transformed feature.

[0188] Optionally, a resolution of the first feature is smaller than a resolution of the at least one second feature.

[0189] In summary, in the embodiment of the present application, category embedding is introduced in the semantic segmentation process. The category embedding includes multiple vectors, each of which is bound to a category, so as to interact with each feature in the image through the category embedding, thereby extracting more information from the image, thereby improving the semantic segmentation effect of the image. In addition, the initial category embedding is determined based on one of the features of the image, rather than randomly initialized, so that the initial category embedding has already integrated the spatial prior information of the image. Compared with directly randomly initializing the initial category embedding to 0, the semantic segmentation method in the embodiment of the present application can further improve the semantic segmentation effect of the image.

[0190] It should be noted that: the image semantic segmentation device provided in the above embodiment only uses the division of the above functional modules as an example when performing image semantic segmentation. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the image semantic segmentation device provided in the above embodiment and the image semantic segmentation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0191] Fig.11 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The image semantic segmentation device in the above embodiment can be Fig.11 The computer device shown is implemented. Fig.11 The computer device includes at least one processor 1101 , a communication bus 1102 , a memory 1103 and at least one communication interface 1104 .

[0192] The processor 1101 may be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

[0193] The communication bus 1102 may include a pathway for transmitting information between the above-mentioned components.

[0194] The memory 1103 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1103 may exist independently and be connected to the processor 1101 via the communication bus 1102. The memory 1103 may also be integrated with the processor 1101.

[0195] The memory 1103 is used to store the program code for executing the solution of the present application, and the execution is controlled by the processor 1101. The processor 1101 is used to execute the program code stored in the memory 1103. The program code may include one or more software modules. Any module in the image semantic segmentation architecture in the aforementioned embodiment can determine the data used to develop the application through the processor 1101 and one or more software modules in the program code in the memory 1103.

[0196] The communication interface 1104 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0197] In a specific implementation, as an embodiment, a computer device may include multiple processors, such as Fig.11 1 and 1105. Each of these processors may be a single-CPU processor or a multi-CPU processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0198] In a specific implementation, as an embodiment, the computer device may further include an output device 1106 and an input device 1107. The output device 1106 communicates with the processor 1101 and may display information in a variety of ways. For example, the output device 1106 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 1107 communicates with the processor 1101 and may receive user input in a variety of ways. For example, the input device 1107 may be a mouse, a keyboard, a touch screen device, or a sensor device.

[0199] The above-mentioned computer device can be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device can be a desktop computer, a portable computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device. The embodiment of the present application does not limit the type of computer device.

[0200] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0201] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0202] The above content is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A method for image semantic segmentation, characterized in that: The method comprises: determining a plurality of features of the image, the plurality of features comprising a first feature and at least one second feature; Determine an initial category embedding based on the first feature, the initial category embedding comprising N vectors, each of the N vectors indicating a category, and N is a positive integer; determining a learned class embedding based on the at least one second feature and the initial class embedding; A semantic segmentation result of the image is determined based on the learned category embedding and the multiple features, where the semantic segmentation result includes the category to which each pixel in the image belongs.

2. The method according to claim 1, characterized in that The determining an initial category embedding based on the first feature comprises: Determine a first image segmentation mask set based on the first feature, where the first image segmentation mask set includes N first image segmentation masks; Performing pooling processing on each of the N first image segmentation masks to obtain N pooling results corresponding one-to-one to the N first image segmentation masks; Based on the N pooling results, the initial category embedding is determined.

3. The method according to claim 1 or 2, characterized in that The at least one second feature is sorted according to resolution size; The determining, based on the at least one second feature and the initial category embedding, of the learned category embedding comprises: For the sorted first second feature, based on the initial category embedding and the first second feature, determine an image segmentation mask set corresponding to the first second feature, and determine a first updated category embedding based on the image segmentation mask set corresponding to the first second feature, the image segmentation mask set corresponding to the first second feature including N image segmentation masks; For the i-th second feature after sorting, where i is greater than 1, an image segmentation mask set corresponding to the i-th second feature is determined based on the most recently updated category embedding and the i-th second feature, and an i-th updated category embedding is determined based on the image segmentation mask set corresponding to the i-th second feature, and the category embedding after the last update is used as the learned category embedding.

4. The method according to claim 3, characterized in that The step of determining an image segmentation mask set corresponding to the first second feature based on the initial category embedding and the first second feature comprises: Performing feature transformation on the N vectors in the initial category embedding respectively to obtain a first transformation feature set corresponding to the initial category embedding, wherein the first transformation feature set includes N transformation features corresponding one-to-one to the N vectors, and the transformation features in the first transformation feature set have the same resolution as the first second feature; A dot product calculation is performed between each transformation feature in the first transformation feature set and the first second feature, to obtain an image segmentation mask set corresponding to the first second feature.

5. The method according to claim 3 or 4, characterized in that The step of determining a first updated category embedding based on the image segmentation mask set corresponding to the first second feature comprises: Determining a category embedding corresponding to the first second feature based on an image segmentation mask set corresponding to the first second feature; The category embedding corresponding to the first second feature and the initial category embedding are fused to obtain the first updated category embedding.

6. The method according to any one of claims 3 to 5, characterized in that: The determining the semantic segmentation result of the image based on the learned category embedding and the multiple features includes: For the sorted first second feature, based on the initial category embedding and the first second feature, determining an enhanced feature corresponding to the first second feature; For the sorted i-th second feature, based on the most recently updated category embedding and the i-th second feature, determine an enhanced feature corresponding to the i-th second feature; The semantic segmentation result is determined based on at least one enhanced feature corresponding one-to-one to the at least one second feature, an image segmentation mask set corresponding to each of the at least one second feature, and the learned category embedding.

7. The method according to claim 6, characterized in that The step of determining, based on the initial category embedding and the first second feature, an enhanced feature corresponding to the first second feature comprises: Performing two different feature transformations on the initial category embedding respectively to obtain a first transformation feature set and a second transformation feature set corresponding to the initial category embedding, wherein the first transformation feature set and the second transformation feature set both include N transformation features corresponding one-to-one to the N vectors; The first second feature is used as a query parameter, the first transformed feature set is used as a key parameter, and the second transformed feature set is used as a value parameter. Through a multi-head self-attention MSA operation, an enhanced feature corresponding to the first second feature is determined.

8. The method according to claim 6 or 7, characterized in that The determining the semantic segmentation result based on at least one enhanced feature corresponding one-to-one to the at least one second feature, the image segmentation mask set corresponding to the at least one second feature respectively, and the learned category embedding includes: Determine a cascaded image segmentation mask set based on the image segmentation mask sets respectively corresponding to the at least one second feature and the first image segmentation mask set, wherein the cascaded image segmentation mask set includes N image segmentation masks; Determining a probability that each pixel in the image belongs to each of the N categories based on the learned category embedding and the cascaded image segmentation mask set; Determine a cascade feature based on a plurality of enhanced features corresponding one-to-one to the at least one second feature and the first feature, wherein a resolution of the cascade feature is the same as a resolution of the image; The semantic segmentation result is determined based on the probability that each pixel in the image belongs to each of the N categories and the cascade feature.

9. The method according to any one of claims 1 to 8, characterized in that: The determining of multiple features of the image comprises: Extracting a plurality of initial features of the image through a backbone network; Performing feature transformation on each initial feature to obtain a plurality of transformed features corresponding to the plurality of initial features, wherein the resolution of each initial feature is twice the resolution of the corresponding transformed feature; Constructing a diversity feature corresponding to each transformation feature to obtain a plurality of diversity features corresponding one-to-one to the plurality of transformation features, wherein the resolution of each transformation feature is the same as the resolution of the corresponding diversity feature; Each transformation feature is concatenated with the corresponding diversity feature, and the concatenated feature is used as a feature corresponding to each transformation feature.

10. The method according to claim 9, characterized in that The constructing of the diversity feature corresponding to each transformation feature includes: Each transformed feature is processed by the compression activation SE operation to obtain the diversity feature corresponding to each transformed feature.

11. The method according to any one of claims 1 to 10, characterized in that: A resolution of the first feature is less than a resolution of the at least one second feature.

12. An image semantic segmentation device, characterized in that: The device comprises: A first determination module, configured to determine a plurality of features of an image, wherein the plurality of features include a first feature and at least one second feature; A second determination module, configured to determine an initial category embedding based on the first feature, wherein the initial category embedding includes N vectors, each of the N vectors indicates a category, and N is a positive integer; A third determination module, configured to determine a learned category embedding based on the at least one second feature and the initial category embedding; A fourth determination module is used to determine a semantic segmentation result of the image based on the learned category embedding and the multiple features, wherein the semantic segmentation result includes a category to which each pixel in the image belongs.

13. The device according to claim 12, characterized in that The second determining module is used for: Determine a first image segmentation mask set based on the first feature, where the first image segmentation mask set includes N first image segmentation masks; Performing pooling processing on each of the N first image segmentation masks to obtain N pooling results corresponding one-to-one to the N first image segmentation masks; Based on the N pooling results, the initial category embedding is determined.

14. The device according to claim 12 or 13, characterized in that The at least one second feature is sorted according to resolution size; The third determination module is used for: For the sorted first second feature, based on the initial category embedding and the first second feature, determine an image segmentation mask set corresponding to the first second feature, and determine a first updated category embedding based on the image segmentation mask set corresponding to the first second feature, the image segmentation mask set corresponding to the first second feature including N image segmentation masks; For the i-th second feature after sorting, where i is greater than 1, an image segmentation mask set corresponding to the i-th second feature is determined based on the most recently updated category embedding and the i-th second feature, and an i-th updated category embedding is determined based on the image segmentation mask set corresponding to the i-th second feature, and the category embedding after the last update is used as the learned category embedding.

15. The device according to claim 14, characterized in that The third determination module is used for: Performing feature transformation on the N vectors in the initial category embedding respectively to obtain a first transformation feature set corresponding to the initial category embedding, wherein the first transformation feature set includes N transformation features corresponding one-to-one to the N vectors, and the transformation features in the first transformation feature set have the same resolution as the first second feature; A dot product calculation is performed between each transformation feature in the first transformation feature set and the first second feature, to obtain an image segmentation mask set corresponding to the first second feature.

16. The device according to claim 14 or 15, characterized in that The third determination module is used for: Determining a category embedding corresponding to the first second feature based on an image segmentation mask set corresponding to the first second feature; The category embedding corresponding to the first second feature and the initial category embedding are fused to obtain the first updated category embedding.

17. The device according to any one of claims 14 to 16, characterized in that: The fourth determination module is used for: For the sorted first second feature, based on the initial category embedding and the first second feature, determining an enhanced feature corresponding to the first second feature; For the sorted i-th second feature, based on the most recently updated category embedding and the i-th second feature, determine an enhanced feature corresponding to the i-th second feature; The semantic segmentation result is determined based on at least one enhanced feature corresponding one-to-one to the at least one second feature, an image segmentation mask set corresponding to each of the at least one second feature, and the learned category embedding.

18. The device according to claim 17, characterized in that The fourth determination module is used for: Performing two different feature transformations on the initial category embedding respectively to obtain a first transformation feature set and a second transformation feature set corresponding to the initial category embedding, wherein the first transformation feature set and the second transformation feature set both include N transformation features corresponding one-to-one to the N vectors; The first second feature is used as a query parameter, the first transformed feature set is used as a key parameter, and the second transformed feature set is used as a value parameter. Through a multi-head self-attention MSA operation, an enhanced feature corresponding to the first second feature is determined.

19. The device according to claim 17 or 18, characterized in that The fourth determination module is used for: Determine a cascaded image segmentation mask set based on the image segmentation mask sets respectively corresponding to the at least one second feature and the first image segmentation mask set, wherein the cascaded image segmentation mask set includes N image segmentation masks; Determining a probability that each pixel in the image belongs to each of the N categories based on the learned category embedding and the cascaded image segmentation mask set; Determine a cascade feature based on a plurality of enhanced features corresponding one-to-one to the at least one second feature and the first feature, wherein a resolution of the cascade feature is the same as a resolution of the image; The semantic segmentation result is determined based on the probability that each pixel in the image belongs to each of the N categories and the cascade feature.

20. The device according to any one of claims 12 to 19, characterized in that: The first determining module is used for: Extracting a plurality of initial features of the image through a backbone network; Performing feature transformation on each initial feature to obtain a plurality of transformed features corresponding to the plurality of initial features, wherein the resolution of each initial feature is twice the resolution of the corresponding transformed feature; Constructing a diversity feature corresponding to each transformation feature to obtain a plurality of diversity features corresponding one-to-one to the plurality of transformation features, wherein the resolution of each transformation feature is the same as the resolution of the corresponding diversity feature; Each transformation feature is concatenated with the corresponding diversity feature, and the concatenated feature is used as a feature corresponding to each transformation feature.

21. The device according to claim 20, characterized in that The first determining module is used for: Each transformed feature is processed by the compression activation SE operation to obtain the diversity feature corresponding to each transformed feature.

22. The device according to any one of claims 12 to 21, characterized in that: A resolution of the first feature is less than a resolution of the at least one second feature.

23. An image semantic segmentation device, characterized in that: The device includes a memory and a processor; The memory is used to store a program that supports the device to execute the method described in any one of claims 1 to 11, and to store data involved in implementing the method described in any one of claims 1 to 11; The processor is configured to execute the program stored in the memory.

24. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 11.

Images