Image segmentation method and device, electronic equipment and storage medium

CN118941793APending Publication Date: 2024-11-12WUHAN UNIV OF TECH
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202411006896.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

[0005]有鉴于此,有必要提供一种图像分割方法、装置、电子设备及存储介质,用以解决现有技术中存在对植物表型图像进行分割时,花费成本较高、分割算法的鲁棒性较差导致分割结果不稳定,分割计算较复杂的技术问题

Benefits of technology

[0013] The beneficial effects of the present invention are: the method provided by the present invention improves and optimizes Attention Unet to obtain a fully trained image segmentation model GAttention Unet, and performs semantic segmentation of two-dimensional plant phenotype images based on the fully trained image segmentation model. Using the improved The RFB (Receptive Field Block, receptive field module) further improves the encoder part, and gMLP (MLPs with gating, gated multi-layer perceptron) is used to improve the decoder part. Since the present invention performs semantic segmentation based on two-dimensional plant phenotype images, the data collection cost is low. The fully trained image segmentation model combines the ideas of Unet and attention mechanism, which can better extract features, reduce redundant calculations, and improve the efficiency and speed of the model. The improved RFB module (i.e., NRFB module) can effectively capture multi-scale features in images and improve the ability to express important features, thereby improving model performance and improving model robustness. gMLP combines a multi-layer perceptron (MLP) with a gating mechanism. The model structure is relatively simple, the computational complexity is low, the number of parameters is small, and the computational overhead during training and inference is small.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941793A_ABST
    Figure CN118941793A_ABST
Patent Text Reader

Abstract

The invention provides an image segmentation method and device, electronic equipment and a storage medium, and belongs to the technical field of image semantic segmentation, and the method comprises the steps: obtaining a target two-dimensional image, carrying out the multi-scale feature extraction of the target two-dimensional image based on a sensing field module in an encoder, and obtaining an optimized feature map; and carrying out attention weight calculation on the optimized feature map based on a gated multilayer perceptron in a decoder to obtain a segmentation result of the target image. The plant phenotype image segmentation method and device solve the technical problems that in the prior art, when a plant phenotype image is segmented, the cost is high, the robustness of a segmentation algorithm is poor, consequently, the segmentation result is unstable, and segmentation calculation is complex.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image semantic segmentation, and in particular to an image segmentation method, device, electronic equipment and storage medium. Background Art

[0002] Semantic segmentation is an image segmentation technology in the field of computer vision, which aims to divide the input image pixels into different semantic categories, so as to achieve semantic understanding and classification of each pixel in the image. Semantic segmentation of plant phenotypic images can accurately identify the various parts of the plant, including stems, leaves and flowers, etc., which is helpful for studying the growth and development process of plants, trait characteristics, and plant selection and breeding; it can also help identify abnormal parts in plant images, such as lesions and pests, etc., which helps to realize pest and disease monitoring and prevention and improve agricultural production efficiency. In addition, semantic segmentation of plant phenotypes can also provide basic data support for agricultural intelligence and environmental monitoring, which is helpful for the precise management of agricultural production and the protection of the ecological environment. At present, most researchers mainly solve the problem of plant segmentation based on two-dimensional images or three-dimensional point clouds.

[0003] In the segmentation of plant phenotypes based on 3D point clouds, laser scanning and other technologies are mainly used to obtain 3D point cloud data of plants, and various segmentation algorithms are used to segment plant phenotypes based on the extracted features and the spatial relationship of point cloud data. In the segmentation of plant phenotypes based on 2D images, traditional computer vision technology or deep learning methods are used to extract and analyze the color, texture, shape and other features of plant images to achieve pixel-level classification of plant parts in the image.

[0004] In the current methods of segmenting plant phenotypes based on three-dimensional point clouds, the acquisition cost is high because high-cost equipment such as laser scanning is usually required to obtain three-dimensional point cloud data of plants; in addition, the robustness of the segmentation algorithm is poor for point cloud data acquired under different lighting and environmental conditions, resulting in unstable segmentation results. Segmentation methods based on deep learning may have high computational complexity and require a lot of computing resources and time, especially when processing large-scale image data. Therefore, there are technical problems in the prior art when segmenting plant phenotypic images, such as high cost, poor robustness of the segmentation algorithm resulting in unstable segmentation results, and complex segmentation calculations. Summary of the invention

[0005] In view of this, it is necessary to provide an image segmentation method, device, electronic device and storage medium to solve the technical problems in the prior art that when segmenting plant phenotype images, the cost is high, the robustness of the segmentation algorithm is poor, resulting in unstable segmentation results, and the segmentation calculation is complex.

[0006] In order to solve the above technical problems, on the one hand, the present invention provides an image segmentation method, comprising: Acquire a target two-dimensional image; Performing multi-scale feature extraction on the target two-dimensional image based on the receptive field module in the encoder to obtain an optimized feature map; Performing attention weight calculation on the optimized feature map based on the gated multilayer perceptron in the decoder to obtain a segmentation result of the target image; The receptive field module includes: four feature extraction submodules and three maximum pooling layers; one maximum pooling layer connects two feature extraction submodules; Wherein, the feature extraction submodule includes: an adder and an extraction unit; Based on the adder of the first feature extraction submodule of the receptive field module in the encoder, the target two-dimensional image is residually connected to obtain a residual map, and based on the extraction unit of the first feature extraction submodule of the receptive field module in the encoder, multi-scale feature extraction is performed on the residual map to obtain a first feature map; Performing residual connection on the first feature map based on the adder of the second feature extraction submodule of the receptive field module in the encoder to obtain a residual map, and performing multi-scale feature extraction on the residual map based on the extraction unit of the second feature extraction submodule of the receptive field module in the encoder to obtain a second feature map; Performing residual connection on the second feature map based on the adder of the third feature extraction submodule of the receptive field module in the encoder to obtain a residual map, and performing multi-scale feature extraction on the residual map based on the extraction unit of the third feature extraction submodule of the receptive field module in the encoder to obtain a third feature map; Based on the adder of the fourth feature extraction submodule of the receptive field module in the encoder, the third feature map is subjected to residual connection to obtain a residual map, based on the extraction unit of the fourth feature extraction submodule of the receptive field module in the encoder, the residual map is subjected to multi-scale feature extraction to obtain a fourth feature map, and based on the fourth feature map, an optimized feature map is obtained; Amplifying the optimized feature map based on the upsampling layer in the decoder to obtain an amplified image, and calculating the attention weight of the amplified image based on the gated multilayer perceptron in the decoder to obtain a segmentation result of the target image; The gated multilayer perceptron includes: four attention mechanism units and three upsampling layers; one upsampling layer connects two attention mechanism units; The attention mechanism unit includes: an attention mechanism layer, a channel splicing layer and a convolution layer; Based on the attention mechanism layer of the fourth attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the enlarged image and the acquired fourth feature map to obtain a fused feature image, and based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a fourth fused spliced ​​image; Based on the attention mechanism layer of the third attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the fourth fused spliced ​​image and the acquired third feature map to obtain a fused feature image, and based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a third fused spliced ​​image; Based on the attention mechanism layer of the second attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the third fused spliced ​​image and the acquired second feature map to obtain a fused feature image, and based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a second fused spliced ​​image; Based on the attention mechanism layer of the first attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the second fused spliced ​​image and the acquired first feature map to obtain a fused feature image, based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a first fused spliced ​​image, and based on the convolution layer, the first fused spliced ​​image is processed to obtain a segmentation result of the target image.

[0007] In a possible implementation, the encoder includes: a convolutional layer, a maximum pooling layer, and a receptive field module connected in series in sequence.

[0008] In a possible implementation, the decoder includes: an upsampling layer, a convolution layer, an activation function, and a gated multilayer perceptron; the convolution layer and the gated multilayer perceptron are connected in series.

[0009] In a possible implementation, the performing attention weight calculation on the first fused spliced ​​image based on the convolution layer to obtain a segmentation result of the target image includes: A convolution operation is performed on the first fused stitched image based on the convolution layer to obtain an image after the convolution operation, and the image after the convolution operation is processed based on an activation function to obtain a segmentation result of the target image.

[0010] On the other hand, the present invention further provides an image segmentation device, characterized in that it comprises: An image acquisition module is used to acquire a target two-dimensional image; An encoder module, used for performing multi-scale feature extraction on the target two-dimensional image based on the receptive field module in the encoder to obtain an optimized feature map; a decoder module, used for performing attention weight calculation on the optimized feature map based on the gated multi-layer perceptron in the decoder to obtain a segmentation result of the target image; The receptive field module includes: four feature extraction submodules and three maximum pooling layers; one maximum pooling layer connects two feature extraction submodules; Wherein, the feature extraction submodule includes: an adder and an extraction unit; Based on the adder of the first feature extraction submodule of the receptive field module in the encoder, the target two-dimensional image is residually connected to obtain a residual map, and based on the extraction unit of the first feature extraction submodule of the receptive field module in the encoder, multi-scale feature extraction is performed on the residual map to obtain a first feature map; Performing residual connection on the first feature map based on the adder of the second feature extraction submodule of the receptive field module in the encoder to obtain a residual map, and performing multi-scale feature extraction on the residual map based on the extraction unit of the second feature extraction submodule of the receptive field module in the encoder to obtain a second feature map; Performing residual connection on the second feature map based on the adder of the third feature extraction submodule of the receptive field module in the encoder to obtain a residual map, and performing multi-scale feature extraction on the residual map based on the extraction unit of the third feature extraction submodule of the receptive field module in the encoder to obtain a third feature map; Based on the adder of the fourth feature extraction submodule of the receptive field module in the encoder, the third feature map is subjected to residual connection to obtain a residual map, based on the extraction unit of the fourth feature extraction submodule of the receptive field module in the encoder, the residual map is subjected to multi-scale feature extraction to obtain a fourth feature map, and based on the fourth feature map, an optimized feature map is obtained; Amplifying the optimized feature map based on the upsampling layer in the decoder to obtain an amplified image, and calculating the attention weight of the amplified image based on the gated multilayer perceptron in the decoder to obtain a segmentation result of the target image; The gated multilayer perceptron includes: four attention mechanism units and three upsampling layers; one upsampling layer connects two attention mechanism units; The attention mechanism unit includes: an attention mechanism layer, a channel splicing layer and a convolution layer; Based on the attention mechanism layer of the fourth attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the enlarged image and the acquired fourth feature map to obtain a fused feature image, and based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a fourth fused spliced ​​image; Based on the attention mechanism layer of the third attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the fourth fused spliced ​​image and the acquired third feature map to obtain a fused feature image, and based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a third fused spliced ​​image; Based on the attention mechanism layer of the second attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the third fused spliced ​​image and the acquired second feature map to obtain a fused feature image, and based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a second fused spliced ​​image; Based on the attention mechanism layer of the first attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the second fused spliced ​​image and the acquired first feature map to obtain a fused feature image, based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a first fused spliced ​​image, and based on the convolution layer, the first fused spliced ​​image is processed to obtain a segmentation result of the target image.

[0011] On the other hand, the present invention also provides an electronic device, including a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the image segmentation method in any of the above implementations.

[0012] On the other hand, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instruction, which, when executed by a processor, can implement the steps of the image segmentation method in any of the above-mentioned implementations.

[0013] The beneficial effects of the present invention are as follows: the method provided by the present invention improves and optimizes the Attention Unet to obtain a well-trained image segmentation model GAttention Unet, performs semantic segmentation on two-dimensional plant phenotype images based on the well-trained image segmentation model, uses the improved RFB (Receptive Field Block, receptive field module) to further improve the encoder part, and uses gMLP (MLPs with gating, gated multilayer perceptron) to improve the decoder part. Since the present invention performs semantic segmentation based on two-dimensional plant phenotype images, the data collection cost is low. The well-trained image segmentation model combines the ideas of Unet and attention mechanism, can better perform feature extraction, and reduces redundant calculations, thereby improving the efficiency and speed of the model. The improved RFB module (i.e., NRFB module) can effectively capture multi-scale features in the image, improve the ability to express important features, thereby improving the performance of the model and improving the robustness of the model. gMLP combines a multi-layer perceptron (MLP) with a gating mechanism, and the model structure is relatively simple, the computational complexity is low, and the number of parameters is small, and the computational overhead during training and reasoning is small. BRIEF DESCRIPTION OF THE DRAWINGS In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0014] Figure 1 A schematic diagram of an embodiment of the image segmentation method provided by the present invention; Figure 2 A network structure diagram of an embodiment of the image segmentation method provided by the present invention; Figure 3a and Figure 3b A performance comparison diagram of an embodiment of the image segmentation method provided by the present invention: Figure 4 A schematic diagram of the structure of an embodiment of the image segmentation device provided by the present invention; Figure 5 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0015] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0016] In the description of the embodiments of the present application, unless otherwise specified, “plurality” means two or more than two.

[0017] The terms "including" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product or equipment comprising a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules not clearly listed or inherent to these processes, methods, products or equipment.

[0018] The naming or numbering of the steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0019] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0020] The present invention provides an image segmentation method, device, electronic device and storage medium, which are respectively described below.

[0021] Figure 1 A schematic diagram of an embodiment of the image segmentation method provided by the present invention and Figure 2 A network structure diagram of an embodiment of the image segmentation method provided by the present invention includes: S101, acquiring a target two-dimensional image; S102, performing multi-scale feature extraction on the target two-dimensional image based on the receptive field module in the encoder to obtain an optimized feature map; S103, calculating the attention weights of the optimized feature map based on the gated multilayer perceptron in the decoder to obtain a segmentation result of the target image.

[0022] It should be noted that the receptive field module of the encoder in this embodiment is an improved receptive field module RFB (i.e., NRFB module) to improve the encoder part. Similarly, the encoder part is improved by a gated multilayer perceptron in the decoder. Since the present invention performs semantic segmentation based on two-dimensional plant phenotype images, the data collection cost is low. The well-trained image segmentation model combines the ideas of Unet and attention mechanism, can better perform feature extraction, and reduces redundant calculations, thereby improving the efficiency and speed of the model. The improved RFB module (i.e., NRFB module) can effectively capture multi-scale features in the image, improve the ability to express important features, thereby improving the performance of the model and improving the robustness of the model.

[0023] It can be understood that the present invention is divided into two parts: an encoder and a decoder.

[0024] (1) Encoder part: Input Image After the first convolution operation Get feature map , as shown below:

[0025]

[0026] Represents the feature map obtained after the first convolution operation, express After the maximum pooling operation The feature map obtained by the operation.

[0027] when hour, Respectively indicate the 1st, 2nd, and 3rd After the maximum pooling operation The obtained feature map.

[0028] when When the input image After convolution operation And the maximum pooling operation is obtained Feature Map , as shown below:

[0029] Preferably, in this embodiment, replace , and Instead of using concatenation to achieve feature fusion, the addition operation is introduced to implement residual connection for feature fusion, which enhances the network's ability to transfer information during the feature fusion process and helps to better utilize feature information from different branches.

[0030] (2) Decoder part: Feature Map After upsampling Get feature map , applying the gated attention mechanism Make feature map and Perform feature fusion to obtain feature map , the feature map and Perform channel splicing operations And convolution operation Get feature map , as shown below:

[0031]

[0032]

[0033] when hour, Indicates that the previous layer has been spliced ​​through the channel And convolution operation The feature map obtained later.

[0034] when When the feature map After convolution operation Get feature map , applying the gated attention mechanism Make feature map and Perform feature fusion to obtain feature map , the feature map and Perform channel splicing operations And convolution operation Get feature map , as shown below:

[0035]

[0036]

[0037]

[0038] Finally, the feature map Perform convolution operation get , and then through the activation function , and get the final prediction result, as shown in the following formula:

[0039]

[0040] In the decoder part, the present invention applies the gMLP structure to In the proposed method, gMLP is used to further process the attention weights and residual connections are introduced to enhance the attention model's perception of global and positional information, enabling the model to better understand the spatial structure and positional relationship in the input data.

[0041] In some embodiments of the present invention, The optimized feature map is enlarged based on the upsampling layer in the decoder to obtain an enlarged image, and the attention weight is calculated on the enlarged image based on the gated multi-layer perceptron in the decoder to obtain the segmentation result of the target image.

[0042] It is understandable that the upsampling module For the input feature map , convolution operation It can be expressed as:

[0043] in, It is a transposed convolution operation, which is used for upsampling to expand the spatial dimension of the input tensor to a larger size, which can be expressed as:

[0044] in, is the input feature map, is the convolution kernel, is the size of the convolution kernel, is the step length, are the row and column indices of the output feature map, is the convolution kernel of the transposed convolution, It is a two-dimensional convolution operation The convolution kernel.

[0045] In some embodiments of the present invention, the encoder includes: a convolutional layer, a maximum pooling layer and a receptive field module connected in series in sequence.

[0046] It is understandable that the encoder part also includes the following operations: 1. Convolution operation : Input Image , convolution operation Contains two consecutive convolutional layers , each convolutional layer is followed by a normalization function And the activation function , as shown below:

[0047] in, and Represent the convolution kernels of two convolutions respectively.

[0048] 1.1 Convolutional Layer : represents a two-dimensional convolution operation, is the input feature map, is the convolution kernel. Assume that the input feature map The size is ,in Indicates height, Indicates width, Indicates the number of channels (input channel number). The specific operation is shown in the following formula:

[0049] in, represents the convolution operation, Represents the bias term.

[0050] 1.2 Normalization function : Indicates that the input is normalized. is the input feature map, is the normalized function The output feature map after The specific operation is shown as follows:

[0051] in, Representation feature map The pixel plane coordinates of Indicates the number of channels, It is in the channel The mean value of pixels on It is in the channel The pixel variance on is a small constant to prevent division by 0, and are learnable scaling and offset parameters.

[0052] 1.3 Activation Function : is the input feature map, After the activation function The output feature map after The specific operation is shown as follows:

[0053] in, Representation feature map The pixel plane coordinates of Indicates the number of channels, and max means taking the larger value of the two, that is, if If it is greater than zero, the output , otherwise output zero.

[0054] 2. Maximum pooling operation : It is a pooling operation that traverses each channel and takes the maximum value in each pooling window. is the input feature map, After the maximum pooling operation The output feature map after The specific operation is shown as follows:

[0055] in, kernel is the size of the pooling window, stride is the step size of the pooling operation, m,n Represents the coordinate position of the pixel in the pooling window.

[0056] 3. Convolution operation :

[0057]

[0058] in, and The operation is similar, but different parameters are set in the convolution operation, that is, different convolution kernels and .Depend on, and get and , respectively as follows:

[0059]

[0060] in, is the input feature map, and the above formulas represent x and operate, , , and Respectively represent The convolution kernel of each convolution in , , , and Respectively represent The convolution kernel of each convolution.

[0061] For the input feature map , convolution operation It can be expressed as:

[0062]

[0063]

[0064]

[0065] in, To perform convolution operation The convolution kernel.

[0066] In this section, the present invention will Using splicing to achieve feature fusion, instead of introducing addition operations to achieve residual connection, and the improved (called ), which enhances the information transmission ability of the network in the feature fusion process and helps to better utilize the feature information from different branches.

[0067] In some embodiments of the present invention, the receptive field module includes: four feature extraction submodules and three maximum pooling layers; one maximum pooling layer connects two feature extraction submodules; Among them, the feature extraction submodule includes: an adder and an extraction unit; Based on the adder of the first feature extraction submodule of the receptive field module in the encoder, the target two-dimensional image is residually connected to obtain a residual map, and based on the extraction unit of the first feature extraction submodule of the receptive field module in the encoder, multi-scale feature extraction is performed on the residual map to obtain a first feature map; Performing residual connection on the first feature map based on the adder of the second feature extraction submodule of the receptive field module in the encoder to obtain a residual map, and performing multi-scale feature extraction on the residual map based on the extraction unit of the second feature extraction submodule of the receptive field module in the encoder to obtain a second feature map; Performing residual connection on the second feature map based on the adder of the third feature extraction submodule of the receptive field module in the encoder to obtain a residual map, and performing multi-scale feature extraction on the residual map based on the extraction unit of the third feature extraction submodule of the receptive field module in the encoder to obtain a third feature map; The adder of the fourth feature extraction submodule of the receptive field module in the encoder performs residual connection on the third feature map to obtain a residual map, and the extraction unit of the fourth feature extraction submodule of the receptive field module in the encoder performs multi-scale feature extraction on the residual map to obtain a fourth feature map, and based on the fourth feature map, an optimized feature map is obtained.

[0068] In order to further illustrate the embodiment of the present invention, the gated attention mechanism unit is now described. For further introduction: Specifically include: 1) Layer normalization operation : is the layer normalization operation, g is the input feature map, After layer normalization operation The output feature map after is as follows:

[0069] in, is the layer normalization operation, for g The mean pixel value in for g The variance of the pixel values ​​in .

[0070] 2) One-dimensional convolution operation : represents a one-dimensional convolution operation, g is the input feature map, is the convolution kernel, After a one-dimensional convolution operation The output feature map after is shown in the following formula:

[0071] in, represents the convolution operation, represents the bias term, is the activation function.

[0072] 3) Spatial gating unit :

[0073] in, g is the input feature map, For passing through the space gating unit The output feature map after .

[0074] The specific operation can be expressed as: 1. Input tensor : ,in, Represents a tensor, len is the length of the tensor, and ffn represents the dimension of the feature.

[0075] 2. First, Split into two parts along the last dimension u and v :

[0076] 3. Then v To control the space gate:

[0077] 4. Let u and Multiply element by element to get :

[0078] in, Represents element-wise multiplication.

[0079] 4) Gated Perceptron :

[0080] in, g is the input feature map, For gated perceptron The output feature map after .

[0081] exist The specific operations are as follows: , in, is the input feature map go through The output obtained is, yes The average pixel value, yes The pixel variance value of Perform a linear transformation to obtain : ,in, It is a linear transformation layer, and the specific operation is: ,in, is the weight matrix, is the bias vector.

[0082] Is an activation function, the specific operation is: is the cumulative distribution function of the standard normal distribution , then Passed to the spatial gating unit ,get , and then again through a : , and finally: .

[0083] 5) Activation Function : The present invention uses The formula expression is: ,in, g is the input value, and e is a natural constant.

[0084] 6) Gated Attention Mechanism :

[0085] Attention Mechanism Used in The module is improved. The structure is applied to the attention mechanism, using Further processing of the attention weights and the introduction of residual connections can enhance the attention model's ability to perceive global and positional information, allowing the model to better understand the spatial structure and positional relationship in the input data. Accepts two input tensors g and p ,in g Used to generate attention weights, p is the input feature tensor, as shown below: (1) Input tensor:

[0086]

[0087] in, represents a tensor, and Respectively g and p The number of feature channels, Indicates the height and width of the feature map.

[0088] (2) Feature transformation: g and p After feature transformation, we get and

[0089]

[0090]

[0091] in, The number of feature channels after transformation, and They are linear transformation layers:

[0092]

[0093] in, and Perform two-dimensional convolution operations on g and p respectively The convolution kernel.

[0094] (3) Use gMLP to To process:

[0095]

[0096]

[0097] (4) Final p and By element-wise multiplication we get .

[0098] Channel splicing operation : .

[0099] As shown in the above formula, assuming that the two input tensors are x and d, after the channel splicing operation Cat(), we get .

[0100] The present invention uses Concatenate two tensors along dimension 1, assuming that the shapes of x and d are and , then after the connection The shape is .

[0101] In some embodiments of the present invention, a gated multilayer perceptron includes: four attention mechanism units and three upsampling layers; one upsampling layer connects two attention mechanism units; Among them, the attention mechanism unit includes: attention mechanism layer, channel splicing layer and convolution layer; Based on the attention mechanism layer of the fourth attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the enlarged image and the acquired fourth feature map to obtain a fused feature image, and based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a fourth fused spliced ​​image; Based on the attention mechanism layer of the third attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the fourth fused spliced ​​image and the acquired third feature map to obtain a fused feature image, and based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a third fused spliced ​​image; Based on the attention mechanism layer of the second attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the third fused spliced ​​image and the acquired second feature map to obtain a fused feature image, and based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a second fused spliced ​​image; Based on the attention mechanism layer of the first attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the second fused spliced ​​image and the acquired first feature map to obtain a fused feature image, based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a first fused spliced ​​image, and based on the convolution layer, the first fused spliced ​​image is processed to obtain a segmentation result of the target image.

[0102] It can be understood that the gated multilayer perceptron includes: four attention mechanism units and three upsampling layers; one upsampling layer connects two attention mechanism units; Among them, the first attention mechanism unit includes: attention mechanism layer, channel splicing layer and convolution layer; The attention mechanism layer of the first attention mechanism unit of the gated multilayer perceptron in the decoder performs feature fusion on the enlarged image to obtain a fused feature image, the channel splicing layer performs a splicing operation on the fused feature image to obtain a fused spliced ​​image, and the convolution layer processes the fused spliced ​​image to obtain a segmentation result of the target image.

[0103] In some embodiments of the present invention, performing attention weight calculation on the first fused spliced ​​image based on the convolution layer to obtain a segmentation result of the target image includes: A convolution operation is performed on the first fused stitched image based on the convolution layer to obtain an image after the convolution operation, and the image after the convolution operation is processed based on an activation function to obtain a segmentation result of the target image.

[0104] It can be understood that the present invention improves and optimizes Attention Unet to obtain GAttention Unet, performs semantic segmentation based on two-dimensional plant phenotype images, uses the improved RFB (ie, NRFB module) to further improve the encoder part, and uses gMLP to improve the decoder part.

[0105] Since the present invention performs semantic segmentation based on two-dimensional plant phenotype images, the acquisition cost of two-dimensional images is low, which reduces the training and use costs. Attention Unet combines the ideas of Unet and attention mechanism, can better perform feature extraction, reduce redundant calculations, and improve the efficiency and speed of the model.

[0106] The NRFB module can effectively capture multi-scale features in the image and improve the ability to express important features, thereby improving the performance of the model and enhancing the robustness of the model. The present invention introduces addition operations in the feature fusion of NRFB to achieve residual connections, which enhances the network's ability to transfer information during the feature fusion process and helps to better utilize feature information from different branches. Compared with the original use of splicing operations for feature fusion, the computational cost of addition is lower because it does not require additional calculations and memory consumption to process splicing operations, which improves the computational efficiency of the network, especially when processing large-scale data, and can reduce the training and inference costs of the model. gMLP combines a multi-layer perceptron (MLP) with a gating mechanism. The model structure is relatively simple, the computational complexity is low, and the number of parameters is small, so the computational overhead during training and reasoning is small. The present invention applies the gMLP structure to the attention mechanism, which can enhance the attention model's ability to perceive global and positional information, so that the model can better understand the spatial structure and positional relationship in the input data.

[0107] Figure 3a and Figure 3b A performance comparison diagram of an embodiment of the image segmentation method provided by the present invention includes: The present invention uses a small plant semantic segmentation dataset as a training set, which contains 110 two-dimensional plant phenotype images and corresponding mask images, all saved in png format and set to the same 512×512 resolution.

[0108] Unet, Attention Unet and GAttention Unet proposed in this invention are used to train on the data set respectively. The loss comparison diagram obtained is shown in Figure 3a. It can be seen that the loss of the improved GAttention Unet decreases the fastest and is the most stable. The obtained Iou comparison diagram is shown in Figure 3a. Figure 3bAs shown, it can be seen that the improved GAttention Unet has the fastest Iou improvement speed, and has the best and most stable effect in subsequent training. In summary, the GAttention Unet proposed in the present invention has excellent performance, which not only improves training efficiency, saves training time and computing resources, but also has stronger reliability and stability, and can obtain more accurate segmentation results. gMLP combines a multi-layer perceptron (MLP) with a gating mechanism. The model structure is relatively simple, the computational complexity is low, and the number of parameters is small, and the computational overhead during training and reasoning is small. The present invention applies the gMLP structure to the attention mechanism, which can enhance the attention model's perception of global and positional information, so that the model can better understand the spatial structure and positional relationship in the input data. As shown in Table 1 and Table 2: Table 1: Loss and Iou values ​​after 100 epochs

[0109] Table 2: Loss and Iou values ​​achieved after 200 epochs

[0110] It can be understood that Unet, Attention Unet and the improved GAttention Unet proposed in the present invention are trained on the same data set. The above two tables show the loss and Iou values ​​of each model after 100 and 200 epochs respectively. It can be seen that the loss function value of the improved GAttention Unet designed in this patent is smaller than that of Unet and Attention Unet, and the segmentation accuracy is higher than that of Unet and Attention Unet.

[0111] It can be further understood that, in particular, Unet, Attention Unet and GAttention Unet proposed in the present invention are trained on the same data set and the loss is calculated using the Dice coefficient. The loss comparison chart is as follows: Figure 3a As shown, the following points can be seen: First, the training loss of the improved GAttention Unet decreases fastest in the initial stage, and it only takes 8 epochs to drop below 0.1. The required epochs are only 1 / 12 of Unet and 1 / 2 of Attention Unet, so it can improve training efficiency and save training time and computing resources.

[0112] Second, the training loss of the improved GAttention Unet is smaller than that of the other two models during the entire training process, that is, the fitting effect is the best under the same training epoch.

[0113] Third, as can be seen from the figure, from beginning to end, the loss fluctuation of the improved GAttention Unet is the smallest, showing a more stable trend, which can make it more conducive to application in real life.

[0114] Similarly, the Iou comparison charts obtained by training Unet, Attention Unet and GAttention Unet proposed in this invention on the same dataset are shown in Figure 2. Figure 3b , we can see the following points: First, the Iou of the improved GAttention Unet increases fastest in the initial stage. It only takes 7 epochs to increase Iou to above 0.8, while Attention Unet needs 15 epochs and Unet needs 45 epochs. Moreover, the improved GAttention Unet only needs 70 epochs to increase Iou to above 0.9, while Attention Unet needs 142 epochs, and the original Unet still does not reach 0.9 after 200 epochs. Therefore, the training efficiency can be improved, saving training time and computing resources.

[0115] Second, the Iou of the improved GAttention Unet is higher than that of the other two models during the entire training process, that is, under the same training epoch, the training effect is the best and the obtained model has the highest accuracy.

[0116] Third, as can be seen from the figure, from beginning to end, the Iou fluctuation of the improved GAttention Unet is the smallest, showing a more stable trend, which makes the model more reliable and can make it more conducive to application in real life.

[0117] Figure 4 A schematic diagram of the structure of an embodiment of the image segmentation device provided by the present invention comprises: An image acquisition module 401 is used to acquire a target two-dimensional image; The encoder module 402 is used to extract multi-scale features of the target two-dimensional image based on the receptive field module in the encoder to obtain an optimized feature map; The decoder module 403 is used to calculate the attention weights of the optimized feature map based on the gated multi-layer perceptron in the decoder to obtain the segmentation result of the target image.

[0118] The image segmentation device 400 provided in the above embodiment can implement the technical solution described in the above image segmentation method embodiment. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above image segmentation method embodiment, which will not be repeated here.

[0119] like Figure 5 As shown, the present invention also provides an electronic device 500. The electronic device 500 includes a processor 501, a memory 502 and a display 503. Figure 5 Only some components of the electronic device 500 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0120] In some embodiments, the processor 501 may be a central processing unit (CPU), a microprocessor or other data processing chip, and is used to run program codes or process data stored in the memory 502, such as the image segmentation method of the present invention.

[0121] In some embodiments, the processor 501 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processor 501 may be local or remote. In some embodiments, the processor 501 may be implemented in a cloud platform. In one embodiment, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc., or any combination thereof.

[0122] In some embodiments, the memory 502 may be an internal storage unit of the electronic device 500, such as a hard disk or memory of the electronic device 500. In other embodiments, the memory 502 may also be an external storage device of the electronic device 500, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 500.

[0123] Furthermore, the memory 502 may include both an internal storage unit of the electronic device 500 and an external storage device. The memory 502 is used to store application software installed in the electronic device 500 and various data.

[0124] In some embodiments, the display 503 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 503 is used to display information on the electronic device 500 and to display a visual user interface. The components 501-503 of the electronic device 500 communicate with each other via a system bus.

[0125] In one embodiment, when the processor 501 executes the image segmentation program in the memory 502, the following steps may be implemented: Acquire a target two-dimensional image; Based on the receptive field module in the encoder, multi-scale feature extraction is performed on the target two-dimensional image to obtain an optimized feature map; Based on the gated multi-layer perceptron in the decoder, the attention weights of the optimized feature map are calculated to obtain the segmentation result of the target image.

[0126] It should be understood that: when the processor 501 executes the image segmentation program in the memory 502, in addition to the above functions, other functions may also be implemented, and details may refer to the description of the above corresponding method embodiments.

[0127] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 500 mentioned, and the electronic device 500 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 500 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0128] Accordingly, an embodiment of the present application also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, the steps or functions in the image segmentation method provided in the above-mentioned method embodiments can be implemented.

[0129] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0130] The image segmentation method, device, electronic device and storage medium provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image segmentation method, characterized in that: include: Acquire a target two-dimensional image; Performing multi-scale feature extraction on the target two-dimensional image based on the receptive field module in the encoder to obtain an optimized feature map; Performing attention weight calculation on the optimized feature map based on the gated multilayer perceptron in the decoder to obtain a segmentation result of the target image; The receptive field module includes: four feature extraction submodules and three maximum pooling layers; one maximum pooling layer connects two feature extraction submodules; Wherein, the feature extraction submodule includes: an adder and an extraction unit; Based on the adder of the first feature extraction submodule of the receptive field module in the encoder, the target two-dimensional image is residually connected to obtain a residual map, and based on the extraction unit of the first feature extraction submodule of the receptive field module in the encoder, multi-scale feature extraction is performed on the residual map to obtain a first feature map; Performing residual connection on the first feature map based on the adder of the second feature extraction submodule of the receptive field module in the encoder to obtain a residual map, and performing multi-scale feature extraction on the residual map based on the extraction unit of the second feature extraction submodule of the receptive field module in the encoder to obtain a second feature map; Performing residual connection on the second feature map based on the adder of the third feature extraction submodule of the receptive field module in the encoder to obtain a residual map, and performing multi-scale feature extraction on the residual map based on the extraction unit of the third feature extraction submodule of the receptive field module in the encoder to obtain a third feature map; Based on the adder of the fourth feature extraction submodule of the receptive field module in the encoder, the third feature map is subjected to residual connection to obtain a residual map, based on the extraction unit of the fourth feature extraction submodule of the receptive field module in the encoder, the residual map is subjected to multi-scale feature extraction to obtain a fourth feature map, and based on the fourth feature map, an optimized feature map is obtained; Amplifying the optimized feature map based on the upsampling layer in the decoder to obtain an amplified image, and calculating the attention weight of the amplified image based on the gated multilayer perceptron in the decoder to obtain a segmentation result of the target image; The gated multilayer perceptron includes: four attention mechanism units and three upsampling layers; one upsampling layer connects two attention mechanism units; The attention mechanism unit includes: an attention mechanism layer, a channel splicing layer and a convolution layer; Based on the attention mechanism layer of the fourth attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the enlarged image and the acquired fourth feature map to obtain a fused feature image, and based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a fourth fused spliced ​​image; Based on the attention mechanism layer of the third attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the fourth fused spliced ​​image and the acquired third feature map to obtain a fused feature image, and based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a third fused spliced ​​image; Based on the attention mechanism layer of the second attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the third fused spliced ​​image and the acquired second feature map to obtain a fused feature image, and based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a second fused spliced ​​image; Based on the attention mechanism layer of the first attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the second fused spliced ​​image and the acquired first feature map to obtain a fused feature image, based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a first fused spliced ​​image, and based on the convolution layer, the first fused spliced ​​image is processed to obtain a segmentation result of the target image.

2. The image segmentation method according to claim 1, characterized in that: The encoder includes: a convolutional layer, a maximum pooling layer and a receptive field module connected in series in sequence.

3. The image segmentation method according to claim 1, characterized in that: The decoder comprises: an upsampling layer, a convolution layer, an activation function and a gated multilayer perceptron; the convolution layer and the gated multilayer perceptron are connected in series.

4. The image segmentation method according to claim 1, characterized in that: The step of performing attention weight calculation on the first fused spliced ​​image based on the convolution layer to obtain a segmentation result of the target image includes: A convolution operation is performed on the first fused stitched image based on the convolution layer to obtain an image after the convolution operation, and the image after the convolution operation is processed based on an activation function to obtain a segmentation result of the target image.

5. An image segmentation device, characterized in that: include: An image acquisition module is used to acquire a target two-dimensional image; An encoder module, used for performing multi-scale feature extraction on the target two-dimensional image based on a receptive field module in the encoder to obtain an optimized feature map; A decoder module, used for calculating the attention weights of the optimized feature map based on a gated multilayer perceptron in the decoder to obtain a segmentation result of the target image; The receptive field module includes: four feature extraction submodules and three maximum pooling layers; one maximum pooling layer connects two feature extraction submodules; Wherein, the feature extraction submodule includes: an adder and an extraction unit; Based on the adder of the first feature extraction submodule of the receptive field module in the encoder, the target two-dimensional image is residually connected to obtain a residual map, and based on the extraction unit of the first feature extraction submodule of the receptive field module in the encoder, multi-scale feature extraction is performed on the residual map to obtain a first feature map; Performing residual connection on the first feature map based on the adder of the second feature extraction submodule of the receptive field module in the encoder to obtain a residual map, and performing multi-scale feature extraction on the residual map based on the extraction unit of the second feature extraction submodule of the receptive field module in the encoder to obtain a second feature map; Performing residual connection on the second feature map based on the adder of the third feature extraction submodule of the receptive field module in the encoder to obtain a residual map, and performing multi-scale feature extraction on the residual map based on the extraction unit of the third feature extraction submodule of the receptive field module in the encoder to obtain a third feature map; Based on the adder of the fourth feature extraction submodule of the receptive field module in the encoder, the third feature map is subjected to residual connection to obtain a residual map, based on the extraction unit of the fourth feature extraction submodule of the receptive field module in the encoder, the residual map is subjected to multi-scale feature extraction to obtain a fourth feature map, and based on the fourth feature map, an optimized feature map is obtained; Amplifying the optimized feature map based on the upsampling layer in the decoder to obtain an amplified image, and calculating the attention weight of the amplified image based on the gated multilayer perceptron in the decoder to obtain a segmentation result of the target image; The gated multilayer perceptron includes: four attention mechanism units and three upsampling layers; one upsampling layer connects two attention mechanism units; The attention mechanism unit includes: an attention mechanism layer, a channel splicing layer and a convolution layer; Based on the attention mechanism layer of the fourth attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the enlarged image and the acquired fourth feature map to obtain a fused feature image, and based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a fourth fused spliced ​​image; Based on the attention mechanism layer of the third attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the fourth fused spliced ​​image and the acquired third feature map to obtain a fused feature image, and based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a third fused spliced ​​image; Based on the attention mechanism layer of the second attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the third fused spliced ​​image and the acquired second feature map to obtain a fused feature image, and based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a second fused spliced ​​image; Based on the attention mechanism layer of the first attention mechanism unit of the gated multilayer perceptron in the decoder, feature fusion is performed on the second fused spliced ​​image and the acquired first feature map to obtain a fused feature image, based on the channel splicing layer, a splicing operation is performed on the fused feature image to obtain a first fused spliced ​​image, and based on the convolution layer, the first fused spliced ​​image is processed to obtain a segmentation result of the target image.

6. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the image segmentation method as described in any one of claims 1 to 4 above.

7. A computer-readable storage medium, characterized in that: Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps of the image segmentation method described in any one of claims 1 to 4.

Citation Information

Patent Citations

  • Two-dimensional medical image segmentation method and system

    CN116433914A