Image segmentation methods, apparatus, electronic devices and storage media

Abstract

This invention provides an image segmentation method, apparatus, electronic device, and storage medium. The method includes: acquiring an image to be segmented; inputting the image to be segmented into an image feature extraction model to obtain image features output by the image feature extraction model; and performing image segmentation on the image to be segmented based on the image features. The image feature extraction model includes a parallel three-branch attention mechanism, a visual converter, and a convolutional model. The three-branch attention mechanism is used to extract token feature information corresponding to tokens in the image to be segmented. The visual converter is used to extract global feature information of the image to be segmented, and the convolutional model is used to extract local feature information of the image to be segmented. The method, apparatus, electronic device, and storage medium provided by this invention improve the comprehensiveness of feature information covered by image features, further improve the quality of image feature extraction, and improve the accuracy and reliability of image segmentation.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an image segmentation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Convolutional neural networks, as a deep learning model widely used in image processing, have achieved many successful applications.

[0003] In existing technologies, traditional convolutional neural networks can only extract local information while ignoring global information, resulting in certain limitations when processing large and multi-scale images. Self-attention mechanisms are a powerful image feature extraction method, but they also have some limitations, one of which is the lack of local information.

[0004] Therefore, the feature information extracted based on traditional convolutional neural networks and self-attention mechanisms is not comprehensive enough, resulting in low accuracy and reliability of image segmentation based on the extracted features. Summary of the Invention

[0005] This invention provides an image segmentation method, apparatus, electronic device, and storage medium to address the shortcomings of existing technologies that extract feature information based on traditional convolutional neural networks and self-attention mechanisms is not comprehensive enough, resulting in low accuracy and reliability of image segmentation based on the extracted features.

[0006] This invention provides an image segmentation method, comprising:

[0007] Obtain the image to be segmented;

[0008] The image to be segmented is input into the image feature extraction model to obtain the image features output by the image feature extraction model;

[0009] Based on the image features, image segmentation is performed on the image to be segmented;

[0010] The image feature extraction model includes a parallel three-branch attention mechanism, a visual converter, and a convolutional model. The three-branch attention mechanism is used to extract token feature information corresponding to tokens in the image to be segmented. The visual converter is used to extract global feature information of the image to be segmented. The convolutional model is used to extract local feature information of the image to be segmented.

[0011] According to an image segmentation method provided by the present invention, the image feature extraction model includes a first extraction branch, a second extraction branch, and a third extraction branch. Each of the first extraction branch, the second extraction branch, and the third extraction branch includes a feature extraction module. The feature extraction module includes the parallel three-branch attention mechanism, the visual converter, and the convolution model.

[0012] The first extraction branch is used to extract the first image features of the image to be segmented based on the feature extraction module;

[0013] The second extraction branch is used to downsample the image to be segmented to obtain the first downsampled image features, and to extract features from the first downsampled image features based on the feature extraction module, and then upsample the features obtained by feature extraction to obtain the second image features;

[0014] The third extraction branch is used to further downsample the first downsampled image features to obtain the second downsampled image features, and to extract features from the second downsampled image features based on the feature extraction module, and then upsample the features obtained by feature extraction to obtain the third image features;

[0015] The first extraction branch is also used to fuse the first image features, the second image features and the third image features to obtain fused image features, and to perform image segmentation on the fused image features based on the feature extraction module.

[0016] According to an image segmentation method provided by the present invention, the three-branch attention mechanism includes a first attention branch, a second attention branch, a third attention branch, and a fusion branch;

[0017] The first attention branch is used to extract column attention from the image to be segmented to obtain column attention features;

[0018] The second attention branch is used to extract line attention from the image to be segmented, thereby obtaining line attention features;

[0019] The third attention branch is used to extract channel attention from the image to be segmented, thereby obtaining channel attention features;

[0020] The first attention branch is further configured to determine fusion features based on the column attention features, the row attention features, and the channel attention features;

[0021] The fusion branch is used to fuse the fusion features and the feature map of the image to be segmented to obtain attention features.

[0022] According to an image segmentation method provided by the present invention, the first attention branch, the second attention branch and the third attention branch each include a first fully connected layer, a first normalized layer, an activation layer, a second fully connected layer and a second normalized layer connected in sequence.

[0023] According to an image segmentation method provided by the present invention, the visual converter includes a first visual conversion layer and a second visual conversion layer;

[0024] The first visual transformation layer is used to embed the feature map of the image to be segmented into a position to obtain a position embedding feature, and to perform layer regularization on the position embedding feature to obtain a regularized feature. Then, multi-head self-attention extraction is performed on the regularized feature to obtain a multi-head self-attention feature, and the first visual transformation feature is obtained based on the multi-head self-attention feature and the position embedding feature.

[0025] The second visual transformation layer is used to perform layer regularization on the first visual transformation feature to obtain layer regularized features, and then perform perceptual classification on the layer regularized features to obtain the second visual transformation feature, and obtain the visual transformation feature based on the first visual transformation feature and the second visual transformation feature.

[0026] According to an image segmentation method provided by the present invention, the convolution model includes multiple convolution modules. The convolution modules are used to perform convolution operations on the feature maps of the image to be segmented to obtain convolution features, perform layer normalization on the convolution features to obtain normalized features, and activate the normalized features to obtain image convolution features.

[0027] According to an image segmentation method provided by the present invention, the feature extraction module is further configured to obtain target extraction features based on the attention features, the visual transformation features, and the image convolution features.

[0028] The present invention also provides an image segmentation apparatus, comprising:

[0029] The acquisition unit is used to acquire the image to be segmented;

[0030] The feature extraction unit is used to input the image to be segmented into the image feature extraction model to obtain the image features output by the image feature extraction model;

[0031] An image segmentation unit is used to perform image segmentation on the image to be segmented based on the image features.

[0032] The image feature extraction model includes a parallel three-branch attention mechanism, a visual converter, and a convolutional model. The three-branch attention mechanism is used to extract token feature information corresponding to tokens in the image to be segmented. The visual converter is used to extract global feature information of the image to be segmented. The convolutional model is used to extract local feature information of the image to be segmented.

[0033] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image segmentation method as described above.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image segmentation method as described above.

[0035] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image segmentation method as described above.

[0036] The image segmentation method, apparatus, electronic device, and storage medium provided by this invention include an image feature extraction model comprising a parallel three-branch attention mechanism, a visual converter, and a convolution model. This allows for the extraction of not only token feature information corresponding to tokens in the image to be segmented, but also global and local feature information of the image to be segmented. This significantly enhances the extraction effect of each feature, improves the comprehensiveness of the feature information covered by the image features, and further improves the quality of image feature extraction. Therefore, it ensures the effectiveness of image segmentation and improves the accuracy and reliability of image segmentation. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating the image segmentation method provided by the present invention;

[0039] Figure 2 This is a schematic diagram of the image feature extraction model provided by the present invention;

[0040] Figure 3 This is a schematic diagram of the feature extraction module provided by the present invention;

[0041] Figure 4 This is a schematic diagram of the three-branch attention mechanism provided by the present invention;

[0042] Figure 5 This is a schematic diagram of the structure of the visual converter provided by the present invention;

[0043] Figure 6 This is a schematic diagram of the image segmentation device provided by the present invention;

[0044] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0046] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and that the objects distinguished by "first," "second," etc., are generally of the same class.

[0047] In related technologies, traditional convolutional neural networks can only extract local information while ignoring global information, which leads to certain limitations when processing large and multi-scale images. Therefore, in order to better handle global and multi-scale information, many new convolutional neural network structures have emerged in recent years, such as global average pooling, spatial pyramid pooling, and multi-scale convolution.

[0048] Despite the achievements of these new architectures, some problems remain. For example, global average pooling cannot handle images with rich detail; spatial pyramid pooling requires explicit specification of different scales, leading to increased computational cost; and multi-scale convolution requires the design of complex network structures, increasing model complexity and training difficulty.

[0049] Self-attention is a powerful image feature extraction method, but it also has some limitations, one of which is the lack of local information. Because self-attention captures similarity by calculating global feature vectors, it may ignore certain local features. For example, when dealing with object edges or details, self-attention may not accurately capture these important local details because it only focuses on global features.

[0050] Another drawback of the self-attention mechanism is that it ignores the overall information within each patch. When calculating the weight of each pixel, the self-attention mechanism only uses the similarity between that pixel and other pixels, resulting in incomplete or inaccurate extracted features.

[0051] To address the above problems, this invention provides an image segmentation method. Figure 1 This is a flowchart illustrating the image segmentation method provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0052] Step 110: Obtain the image to be segmented.

[0053] Specifically, an image to be segmented can be acquired. The image to be segmented here refers to the image that needs to be segmented later. The image to be segmented can be pre-acquired by an image acquisition device, or it can be captured in real time, or it can be downloaded or scanned by the Internet. For example, the image to be segmented can be an image of a natural disaster, etc. This embodiment of the invention does not make specific limitations in this regard.

[0054] Step 120: Input the image to be segmented into the image feature extraction model to obtain the image features output by the image feature extraction model;

[0055] Step 130: Based on the image features, perform image segmentation on the image to be segmented;

[0056] The image feature extraction model includes a parallel three-branch attention mechanism, a visual converter, and a convolutional model. The three-branch attention mechanism is used to extract token feature information corresponding to tokens in the image to be segmented. The visual converter is used to extract global feature information of the image to be segmented. The convolutional model is used to extract local feature information of the image to be segmented.

[0057] Specifically, after obtaining the image to be segmented, the image to be segmented can be input into the image feature extraction model to obtain the image features output by the image feature extraction model.

[0058] The image feature extraction model here can include a parallel three-branch attention mechanism, a Vision Transformer (ViT), and a convolutional model. The three-branch attention mechanism here can include a first attention branch, a second attention branch, a third attention branch, and a fusion branch.

[0059] The three-branch attention mechanism here can be used to extract token feature information corresponding to the token in the image to be segmented, the visual converter here is used to extract global feature information of the image to be segmented, and the convolutional model here can be used to extract local feature information of the image to be segmented.

[0060] The token here refers to the image to be segmented, which is divided into different image blocks by a sliding window and each image block is encoded. Here, the encoding of each image block can be done using a cascaded multilayer convolutional neural network (CNN), a deep neural network (DNN), or a combination of CNN and DNN, etc. The embodiments of the present invention do not specifically limit this.

[0061] The token feature information here refers to the internal feature information of each token in the image to be segmented.

[0062] It is understandable that the image feature extraction model, which includes a parallel three-branch attention mechanism, a visual converter, and a convolutional model, can not only extract the token feature information corresponding to the token in the image to be segmented, but also extract the global feature information of the image to be segmented, as well as the local feature information of the image to be segmented. This greatly enhances the extraction effect of each feature information and improves the comprehensiveness of the feature information covered by the image features.

[0063] Then, based on image features, image segmentation is performed on the image to be segmented, thereby improving the accuracy and reliability of image segmentation.

[0064] Here, other image processing can also be performed on the image to be segmented based on image features, such as object detection, semantic segmentation, image reconstruction, etc., and the embodiments of the present invention do not specifically limit this.

[0065] The method provided in this invention includes a parallel three-branch attention mechanism, a visual converter, and a convolutional model for image feature extraction. This allows for the extraction of not only token feature information corresponding to tokens in the image to be segmented, but also global and local feature information. This significantly enhances the extraction effect of each feature, improves the comprehensiveness of the feature information covered by the image features, and further improves the quality of image feature extraction. Therefore, it ensures the effectiveness of image segmentation and improves the accuracy and reliability of image segmentation.

[0066] Traditional convolutional neural networks typically employ multiple convolutional operations to extract image features. However, this method often only captures relatively large features and fails to fully utilize the detailed information in the image, especially edge information. Therefore, introducing a hierarchical structure into image feature extraction can help the model fully utilize information at different scales in the image and improve its ability to extract feature edge information.

[0067] Based on the above embodiments, Figure 2 This is a schematic diagram of the image feature extraction model provided by the present invention, as shown below. Figure 2 As shown, the image feature extraction model includes a first extraction branch, a second extraction branch, and a third extraction branch, and each of the first extraction branch, the second extraction branch, and the third extraction branch includes a feature extraction module;

[0068] Figure 3 This is a schematic diagram of the feature extraction module provided by the present invention, as shown below. Figure 3 As shown, the feature extraction module includes the parallel three-branch attention mechanism, the visual converter, and the convolutional model;

[0069] The first extraction branch is used to extract the first image features of the image to be segmented based on the feature extraction module;

[0070] The second extraction branch is used to downsample the image to be segmented to obtain the first downsampled image features, and to extract features from the first downsampled image features based on the feature extraction module, and then upsample the features obtained by feature extraction to obtain the second image features;

[0071] The third extraction branch is used to further downsample the first downsampled image features to obtain the second downsampled image features, and to extract features from the second downsampled image features based on the feature extraction module, and then upsample the features obtained by feature extraction to obtain the third image features;

[0072] The first extraction branch is also used to fuse the first image features, the second image features and the third image features to obtain fused image features, and to perform image segmentation on the fused image features based on the feature extraction module.

[0073] Specifically, the image feature extraction model can include a hierarchical structure of a first extraction branch, a second extraction branch, and a third extraction branch. This hierarchical structure can help the model make full use of information at different scales in the image and improve its ability to extract feature edge information.

[0074] The first, second, and third extraction branches here all include feature extraction modules.

[0075] The first extraction branch here is used to extract the first image features of the image to be segmented based on the feature extraction module. The first image features have the same size as the feature map of the image to be segmented. The feature extraction module may include a parallel three-branch attention mechanism, a visual converter, and a convolutional model. It can also fuse the features output by the three-branch attention mechanism, the visual converter, and the convolutional model, and then input the fused features into the decoder to obtain the output result.

[0076] The second extraction branch here is used to downsample the image to be segmented to obtain first downsampled image features. Based on the feature extraction module, features are extracted from the first downsampled image features, and then the extracted features are upsampled to obtain second image features. That is, the size of the second image features is the same as the size of the feature map of the image to be segmented. Here, downsampling the image to be segmented can be done by a factor of 2, etc., and this embodiment of the invention does not specifically limit this.

[0077] The third extraction branch here is used to further downsample the features of the first downsampled image to obtain the features of the second downsampled image. Features are then extracted from the second downsampled image features based on the feature extraction module, and the extracted features are upsampled to obtain the features of the third image. That is, the size of the third image features is the same as the size of the feature map of the image to be segmented. Here, further downsampling of the features of the first downsampled image can involve downsampling the features of the first downsampled image by a factor of 2, etc., and this embodiment of the invention does not specifically limit this.

[0078] The first extraction branch here can also be used to fuse the first image features, the second image features, and the third image features to obtain fused image features, and then perform image segmentation based on the fused image features using the feature extraction module. The fused image features here are features that combine the first image features, the second image features, and the third image features.

[0079] Understandably, the first, second, and third extraction branches all include parallel three-branch attention mechanisms, visual converters, and convolutional models. This allows for the extraction of not only token features corresponding to the tokens in the image to be segmented, but also global and local features, significantly enhancing the extraction effectiveness of each feature and improving the comprehensiveness of the image features, thus further improving the quality of image feature extraction. Furthermore, introducing a hierarchical structure in image feature extraction helps the model fully utilize information at different scales in the image, improving its ability to extract feature edge information. This ensures the effectiveness of image segmentation and further enhances its accuracy and reliability.

[0080] Based on the above embodiments, Figure 4 This is a schematic diagram of the three-branch attention mechanism provided by the present invention, as shown below. Figure 4 As shown, the three-branch attention mechanism includes a first attention branch, a second attention branch, a third attention branch, and a fusion branch;

[0081] The first attention branch is used to extract column attention from the image to be segmented to obtain column attention features;

[0082] The second attention branch is used to extract line attention from the image to be segmented, thereby obtaining line attention features;

[0083] The third attention branch is used to extract channel attention from the image to be segmented, thereby obtaining channel attention features;

[0084] The first attention branch is further configured to determine fusion features based on the column attention features, the row attention features, and the channel attention features;

[0085] The fusion branch is used to fuse the fusion features and the feature map of the image to be segmented to obtain attention features.

[0086] Specifically, the three-branch attention mechanism can include a first attention branch, a second attention branch, a third attention branch, and a fusion branch. The first attention branch is used to extract column attention from the image to be segmented, obtaining column attention features. These column attention features reflect attention feature information at the column direction level. For example, the row dimension of the first attention branch can be multiplied by W, and the channel dimension by C, to obtain the expanded first attention branch T. HF ∈R C ×H×W .

[0087] For example, global average pooling can be used to process the feature map of the image to be segmented into three branches, as shown in the formula:

[0088] T F =σ(T) WF +T HF +T CF )

[0089] Where σ represents the nonlinear activation sigmoid function, T WF T HF and T CF These represent the first attention branch, the second attention branch, and the third attention branch, respectively.

[0090] For example, when weighting columns, the first attention branch performs global pooling on the rows and channels of the feature map, eliminating the influence of rows and channels on column information. Then, a fully connected layer is used to weight the column vectors. Specifically, to reduce parameter overhead, the hidden activation size is set to... Where r is the reduction ratio, the processing formula for the first attention branch is as follows:

[0091] T WF =BN(FC(ReLU(BN1(FC(s)) H )))))

[0092] =BN(W1(ReLU(BN1(W0s) H +b0)))+b1)

[0093] in, b1∈R H FC represents a fully connected layer, T WF The column attention feature is represented by BN1, which represents the normalization layer, ReLU represents the activation layer, and BN represents the normalization layer. It is worth noting that the activation function ReLU is applied after the BN1 layer.

[0094] In the process of weighting attention on rows, columns, and channels, these column-related operations are also used on rows and channels.

[0095] The second attention branch here is used to extract row attention from the image to be segmented, obtaining row attention features. These row attention features reflect attention feature information at the row direction level. For example, the column dimension of the second attention branch can be multiplied by H, and the channel dimension can be multiplied by C to obtain the expanded second attention branch T. WF ∈R C×H×W .

[0096] The third attention branch here is used to extract channel attention from the image to be segmented, obtaining channel attention features. These channel attention features reflect attention feature information at the channel orientation level. For example, the row dimension of the third attention branch can be multiplied by W, and the column dimension by H, to obtain the expanded third attention branch T. CF ∈R C×H×W .

[0097] Here, the second attention branch extracts row attention features from the image to be segmented, and the third attention branch extracts channel attention features from the image to be segmented. The formula for obtaining channel attention features is similar to that of the first attention branch, and will not be repeated here.

[0098] The first attention branch can also be used to determine the fusion feature T based on column attention features, row attention features, and channel attention features. F The formula is as follows:

[0099] T F =σ(T) WF +T CF +T HF )

[0100] Where σ represents the nonlinear activation sigmoid, T WF T represents column attention features. CF T represents row attention features. HF This represents the channel attention feature.

[0101] The fusion branch here is used to fuse the fused features and the feature map of the image to be segmented to obtain attention features.

[0102] Here, feature fusion is performed on the fused features and the feature map of the image to be segmented. This can be done by concatenating the fused features and the feature map of the image to be segmented, or by using an attention mechanism to weight the fused features and the feature map of the image to be segmented before concatenation. This embodiment of the invention does not specifically limit the specific method.

[0103] That is, the weighted result of each branch is expanded to the size of its original feature map. Since average pooling was used initially, the expanded result is on the same order of magnitude as the feature map of the image to be segmented. Finally, these expanded feature maps are summed and then multiplied by the original feature map to complete the weighting operation on the feature maps.

[0104] The specific details are as follows:

[0105] F′=F+F×T F

[0106] Where × represents element-wise multiplication, and F represents the feature map of the image to be segmented, T F F represents the fusion feature, and F′ represents the final attention feature.

[0107] Based on the above embodiments, the first attention branch, the second attention branch, and the third attention branch all include a first fully connected layer, a first normalized layer, an activation layer, a second fully connected layer, and a second normalized layer connected in sequence.

[0108] Specifically, the first attention branch, the second attention branch, and the third attention branch can each include a first fully connected layer (FC), a first normalized layer, an activation layer, a second fully connected layer, and a second normalized layer connected in sequence.

[0109] The first fully connected layer here may be the same as or different from the second fully connected layer. This embodiment of the invention does not specifically limit this.

[0110] The first normalization layer here may be the same as or different from the second normalization layer. This embodiment of the invention does not specifically limit this.

[0111] The first normalization layer here can be LN, BN (Batch Normalization), or IN. The activation layer here can use the GELU activation function, the Sigmoid activation function, or the ReLU activation function. This embodiment of the invention does not specifically limit this.

[0112] Based on the above embodiments, Figure 5 This is a schematic diagram of the structure of the visual converter provided by the present invention, as shown below. Figure 5 As shown, the visual converter includes a first visual conversion layer and a second visual conversion layer;

[0113] The first visual transformation layer is used to embed the feature map of the image to be segmented into a position to obtain a position embedding feature, and to perform layer regularization on the position embedding feature to obtain a regularized feature. Then, multi-head self-attention extraction is performed on the regularized feature to obtain a multi-head self-attention feature, and the first visual transformation feature is obtained based on the multi-head self-attention feature and the position embedding feature.

[0114] The second visual transformation layer is used to perform layer regularization on the first visual transformation feature to obtain layer regularized features, and then perform perceptual classification on the layer regularized features to obtain the second visual transformation feature, and obtain the visual transformation feature based on the first visual transformation feature and the second visual transformation feature.

[0115] Specifically, the visual transformer can include a first visual transformer layer and a second visual transformer layer. The first visual transformer layer is used to embed the feature map of the image to be segmented into positions, obtaining positional embedding features. That is, positional embeddings are added to the feature map to mark the position of each token in the segmentation. Then, these tokens are fully connected in the dimensional direction and decomposed into three tokens of the same size, named key(k), query(q), and value(v). From the perspective of the attention mechanism, this is actually a channel-wise attention weighting of the tokens.

[0116] The visual transformer transposes and multiplies k and q, then normalizes the result, thus obtaining the relationships between tokens. Therefore, the essence of the self-attention mechanism is to replace the query with a weighted value.

[0117] The specific formula is as follows:

[0118]

[0119] Where, d k The number of columns in matrices q and k is the vector dimension, k T Let k be the transpose matrix.

[0120] The transpose of k and q is actually a matching of the relationships between each token and all other tokens, including itself, ultimately yielding the relationships between each token and all other tokens. This relationship is then weighted over v to obtain the final output. From the perspective of attention mechanisms, the above operation is undoubtedly a spatial attention weighting.

[0121] Next, layer regularization is applied to the location embedding features, that is, the token set is regularized (standardized) to obtain regularized features. Then, multi-head self-attention is applied to the regularized features to obtain multi-head self-attention features. Finally, the first visual transformation features are obtained by fusing the multi-head self-attention features and the location embedding features.

[0122] The second visual transformation layer here is used to perform layer regularization on the first visual transformation features to obtain layer regularized features. Then, the layer regularized features are subjected to perceptual classification to obtain the second visual transformation features. Finally, the visual transformation features are obtained by fusing the first and second visual transformation features.

[0123] Here, a multilayer perceptron (MLP) can be used to perform perceptual classification on layer-regularized features.

[0124] Based on the above embodiments, the convolution model includes multiple convolution modules. The convolution modules are used to perform convolution operations on the feature maps of the image to be segmented to obtain convolution features, perform layer normalization on the convolution features to obtain normalized features, and activate the normalized features to obtain image convolution features.

[0125] Specifically, a convolutional model can include multiple convolutional modules. The convolutional modules are used to perform convolution operations on the feature maps of the image to be segmented to obtain convolutional features, perform layer normalization on the convolutional features to obtain normalized features, and activate the normalized features to obtain image convolutional features.

[0126] First, a 1×1 convolutional block can be used to reduce the size of the feature map to half its original size. Then, two 3×3 convolutional blocks are used, and finally, a 1×1 convolutional block is used to restore it to its original size. This allows for more efficient extraction of local feature information while significantly reducing the number of parameters.

[0127] Here, the convolutional features are normalized to obtain normalized features. LN, BN (Batch Normalization), or IN can be used. Here, the normalized features are activated to obtain image convolutional features. GELU activation function, Sigmoid activation function, or ReLU activation function can be used. This embodiment of the invention does not make specific limitations on this.

[0128] Based on the above embodiments, the feature extraction module is further configured to obtain target extraction features based on the attention features, the visual transformation features, and the image convolution features.

[0129] Specifically, the feature extraction module can also be used to obtain target extraction features based on attention features, visual transformation features, and image convolution features. Here, attention features, visual transformation features, and image convolution features can be fused to obtain target extraction features. The target extraction features here are the features finally output by the feature extraction module.

[0130] Here, feature fusion of attention features, visual transformation features, and image convolution features can be achieved by concatenating the attention features, visual transformation features, and image convolution features, or by using an attention mechanism to weight the attention features, visual transformation features, and image convolution features before concatenation, etc. The embodiments of the present invention do not specifically limit this.

[0131] The image segmentation apparatus provided by the present invention will be described below. The image segmentation apparatus described below can be referred to in correspondence with the image segmentation method described above.

[0132] Based on the above embodiments, the present invention provides an image segmentation apparatus. Figure 6 This is a schematic diagram of the image segmentation device provided by the present invention, as shown below. Figure 6 As shown, the device includes:

[0133] Acquisition unit 610 is used to acquire the image to be segmented;

[0134] Feature extraction unit 620 is used to input the image to be segmented into the image feature extraction model to obtain the image features output by the image feature extraction model;

[0135] Image segmentation unit 630 is used to perform image segmentation on the image to be segmented based on the image features;

[0136] The image feature extraction model includes a parallel three-branch attention mechanism, a visual converter, and a convolutional model. The three-branch attention mechanism is used to extract token feature information corresponding to tokens in the image to be segmented. The visual converter is used to extract global feature information of the image to be segmented. The convolutional model is used to extract local feature information of the image to be segmented.

[0137] The apparatus provided in this invention includes a parallel three-branch attention mechanism, a visual converter, and a convolutional model for image feature extraction. This allows for the extraction of not only token feature information corresponding to tokens in the image to be segmented, but also global and local feature information of the image to be segmented. This significantly enhances the extraction effect of each feature, improves the comprehensiveness of the feature information covered by the image features, and further improves the quality of image feature extraction. Therefore, it ensures the effectiveness of image segmentation and improves the accuracy and reliability of image segmentation.

[0138] Based on any of the above embodiments, the image feature extraction model includes a first extraction branch, a second extraction branch, and a third extraction branch. Each of the first extraction branch, the second extraction branch, and the third extraction branch includes a feature extraction module. The feature extraction module includes the parallel three-branch attention mechanism, the visual converter, and the convolution model.

[0139] The first extraction branch is used to extract the first image features of the image to be segmented based on the feature extraction module;

[0140] The second extraction branch is used to downsample the image to be segmented to obtain the first downsampled image features, and to extract features from the first downsampled image features based on the feature extraction module, and then upsample the features obtained by feature extraction to obtain the second image features;

[0141] The third extraction branch is used to further downsample the first downsampled image features to obtain the second downsampled image features, and to extract features from the second downsampled image features based on the feature extraction module, and then upsample the features obtained by feature extraction to obtain the third image features;

[0142] The first extraction branch is also used to fuse the first image features, the second image features and the third image features to obtain fused image features, and to perform image segmentation on the fused image features based on the feature extraction module.

[0143] Based on any of the above embodiments, the three-branch attention mechanism includes a first attention branch, a second attention branch, a third attention branch, and a fusion branch;

[0144] The first attention branch is used to extract column attention from the image to be segmented to obtain column attention features;

[0145] The second attention branch is used to extract line attention from the image to be segmented, thereby obtaining line attention features;

[0146] The third attention branch is used to extract channel attention from the image to be segmented, thereby obtaining channel attention features;

[0147] The first attention branch is further configured to determine fusion features based on the column attention features, the row attention features, and the channel attention features;

[0148] The fusion branch is used to fuse the fusion features and the feature map of the image to be segmented to obtain attention features.

[0149] Based on any of the above embodiments, the first attention branch, the second attention branch, and the third attention branch all include a first fully connected layer, a first normalized layer, an activation layer, a second fully connected layer, and a second normalized layer connected in sequence.

[0150] Based on any of the above embodiments, the visual converter includes a first visual conversion layer and a second visual conversion layer;

[0151] The first visual transformation layer is used to embed the feature map of the image to be segmented into a position to obtain a position embedding feature, and to perform layer regularization on the position embedding feature to obtain a regularized feature. Then, multi-head self-attention extraction is performed on the regularized feature to obtain a multi-head self-attention feature, and the first visual transformation feature is obtained based on the multi-head self-attention feature and the position embedding feature.

[0152] The second visual transformation layer is used to perform layer regularization on the first visual transformation feature to obtain layer regularized features, and then perform perceptual classification on the layer regularized features to obtain the second visual transformation feature, and obtain the visual transformation feature based on the first visual transformation feature and the second visual transformation feature.

[0153] Based on any of the above embodiments, the convolution model includes multiple convolution modules. The convolution modules are used to perform convolution operations on the feature maps of the image to be segmented to obtain convolution features, perform layer normalization on the convolution features to obtain normalized features, and activate the normalized features to obtain image convolution features.

[0154] Based on any of the above embodiments, the feature extraction module is further configured to obtain target extraction features based on the attention features, the visual transformation features, and the image convolution features.

[0155] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute an image segmentation method, which includes: acquiring an image to be segmented; inputting the image to be segmented into an image feature extraction model to obtain image features output by the image feature extraction model; and performing image segmentation on the image to be segmented based on the image features. The image feature extraction model includes a parallel three-branch attention mechanism, a visual converter, and a convolutional model. The three-branch attention mechanism is used to extract token feature information corresponding to tokens in the image to be segmented; the visual converter is used to extract global feature information of the image to be segmented; and the convolutional model is used to extract local feature information of the image to be segmented.

[0156] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image segmentation method provided by the above methods. The method includes: acquiring an image to be segmented; inputting the image to be segmented into an image feature extraction model to obtain image features output by the image feature extraction model; and performing image segmentation on the image to be segmented based on the image features. The image feature extraction model includes a parallel three-branch attention mechanism, a visual converter, and a convolutional model. The three-branch attention mechanism is used to extract token feature information corresponding to tokens in the image to be segmented. The visual converter is used to extract global feature information of the image to be segmented. The convolutional model is used to extract local feature information of the image to be segmented.

[0158] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image segmentation method provided by the above methods. The method includes: acquiring an image to be segmented; inputting the image to be segmented into an image feature extraction model to obtain image features output by the image feature extraction model; and performing image segmentation on the image to be segmented based on the image features. The image feature extraction model includes a parallel three-branch attention mechanism, a visual converter, and a convolutional model. The three-branch attention mechanism is used to extract token feature information corresponding to tokens in the image to be segmented. The visual converter is used to extract global feature information of the image to be segmented. The convolutional model is used to extract local feature information of the image to be segmented.

[0159] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0160] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image segmentation method characterized by, The method comprises: acquiring an image to be segmented; inputting the image to be segmented into an image feature extraction model to obtain image features output by the image feature extraction model; performing image segmentation on the image to be segmented based on the image features. The image feature extraction model comprises a parallel three-branch attention mechanism, a visual transformer and a convolution model, the three-branch attention mechanism is used to extract token feature information corresponding to tokens of the image to be segmented, the visual transformer is used to extract global feature information of the image to be segmented, and the convolution model is used to extract local feature information of the image to be segmented; the token refers to an image block obtained by dividing the image to be segmented into different image blocks in a sliding window manner and encoding each image block; and the token feature information refers to token internal feature information reflected by each token in the image to be segmented; The image feature extraction model comprises a first extraction branch, a second extraction branch and a third extraction branch, and each of the first extraction branch, the second extraction branch and the third extraction branch comprises a feature extraction module, and the feature extraction module comprises the parallel three-branch attention mechanism, the visual transformer and the convolution model; The three-branch attention mechanism comprises a first attention branch, a second attention branch, a third attention branch and a fusion branch.

2. The image segmentation method of claim 1, wherein, The first extraction branch is configured to extract first image features of the image to be segmented based on the feature extraction module; The second extraction branch is configured to down-sample the image to be segmented to obtain first down-sampled image features, extract features of the first down-sampled image features based on the feature extraction module, up-sample the extracted features, and obtain second image features; The third extraction branch is configured to down-sample the first down-sampled image features again to obtain second down-sampled image features, extract features of the second down-sampled image features based on the feature extraction module, up-sample the extracted features, and obtain third image features; The first extraction branch is further configured to fuse the first image features, the second image features and the third image features to obtain fused image features, and perform image segmentation on the fused image features based on the feature extraction module.

3. The image segmentation method of claim 1, wherein, The first attention branch is configured to perform column attention extraction on the image to be segmented to obtain column attention features; The second attention branch is configured to perform row attention extraction on the image to be segmented to obtain row attention features; The third attention branch is configured to perform channel attention extraction on the image to be segmented to obtain channel attention features; The first attention branch is further configured to determine fused features based on the column attention features, the row attention features and the channel attention features; The fusion branch is configured to perform feature fusion on the fused features and feature maps of the image to be segmented to obtain attention features.

4. The image segmentation method of claim 3, wherein, The first attention branch, the second attention branch and the third attention branch each comprise a first fully connected layer, a first normalization layer, an activation layer, a second fully connected layer and a second normalization layer connected in sequence.

5. The image segmentation method of claim 1, wherein, The visual converter comprises a first visual conversion layer and a second visual conversion layer. The first visual conversion layer is configured to perform position embedding on a feature map of the image to be segmented to obtain a position embedding feature, perform layer regularization on the position embedding feature to obtain a regularized feature, perform multi-head self-attention extraction on the regularized feature to obtain a multi-head self-attention feature, and obtain a first visual conversion feature based on the multi-head self-attention feature and the position embedding feature. The second visual conversion layer is configured to perform layer regularization on the first visual conversion feature to obtain a layer regularization feature, perform perception classification on the layer regularization feature to obtain a second visual conversion feature, and obtain a visual conversion feature based on the first visual conversion feature and the second visual conversion feature.

6. The image segmentation method of claim 1, wherein, The convolution model comprises a plurality of convolution modules, and each convolution module is configured to perform convolution operation on a feature map of the image to be segmented to obtain a convolution feature, perform layer normalization on the convolution feature to obtain a normalized feature, and perform activation on the normalized feature to obtain an image convolution feature.

7. The image segmentation method of any one of claims 1 to 6, characterized in that, The feature extraction module is further configured to obtain a target extraction feature based on the attention feature, the visual conversion feature and the image convolution feature.

8. An image segmentation apparatus characterized by comprising: It comprises: An acquisition unit configured to acquire an image to be segmented; A feature extraction unit configured to input the image to be segmented into an image feature extraction model to obtain an image feature output by the image feature extraction model; An image segmentation unit configured to perform image segmentation on the image to be segmented based on the image feature; The image feature extraction model comprises a three-branch attention mechanism, a visual converter and a convolution model in parallel, the three-branch attention mechanism is configured to extract token feature information corresponding to tokens of the image to be segmented, the visual converter is configured to extract global feature information of the image to be segmented, and the convolution model is configured to extract local feature information of the image to be segmented; the token refers to an image block obtained by dividing the image to be segmented into different image blocks in a sliding window manner; and the token feature information refers to token internal feature information reflected by each token in the image to be segmented. The image feature extraction model comprises a first extraction branch, a second extraction branch and a third extraction branch, and each of the first extraction branch, the second extraction branch and the third extraction branch comprises a feature extraction module, and the feature extraction module comprises the three-branch attention mechanism, the visual converter and the convolution model in parallel. The three-branch attention mechanism comprises a first attention branch, a second attention branch, a third attention branch and a fusion branch.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the image segmentation method of any one of claims 1 to 7 when executing the program. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the image segmentation method of any one of claims 1 to 7 when executed by the processor.

Images