Intelligent image segmentation method and system
By introducing a combination of multiple downsampling, position attention, and channel attention modules into the image segmentation network, the problems of low image segmentation accuracy and efficiency in existing technologies are solved, and more efficient image segmentation results are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUPERACCURACY SCIENCE & TECHNOLOGY CO LTD
- Filing Date
- 2023-05-06
- Publication Date
- 2026-04-21
AI Technical Summary
Existing image segmentation methods fail to effectively enhance the information between features at different scales through attention mechanisms between downsampling and upsampling, resulting in low segmentation accuracy and efficiency.
By combining multiple downsampling modules, positional attention modules, upsampling modules, and channel attention modules, and through convolution operations and attention calculations of multi-scale feature matrices, the accuracy and efficiency of image segmentation are improved.
By highlighting the relative positional information of features at different scales during the encoding stage and extracting the channel dimension information of multi-scale features during the decoding stage, the accuracy and efficiency of image segmentation are improved.
Smart Images
Figure CN116681888B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and specifically to an intelligent image segmentation method and system. Background Technology
[0002] Currently, the main methods for segmenting CT images include manual annotation by doctors, traditional algorithm segmentation, and deep learning segmentation.
[0003] Among them, the manual annotation method by doctors requires professional physicists to spend a lot of time and energy on manual annotation, and it is gradually being abandoned due to the high time and economic costs.
[0004] Traditional algorithms perform edge detection and region delineation based on information such as grayscale values in CT images. However, factors such as unclear color and texture of some small organs, deformation of the organs themselves, and unclear images in a few cases can all lead to a decrease in the accuracy of traditional algorithms. Therefore, this method is gradually being abandoned.
[0005] Deep learning is a statistical method that learns from large amounts of data, resulting in better generalization and robustness, and thus achieving better segmentation accuracy. It is a segmentation method that is gradually gaining popularity.
[0006] Existing deep learning methods primarily improve upon the UNET network architecture by introducing a dual attention mechanism between UNET downsampling and upsampling. Other methods utilize the RepVGG network architecture, leveraging its branch structure and parameter reconfiguration to reduce the number of parameters and improve generalization ability.
[0007] However, the above method introduces an attention mechanism within a single scale feature between downsampling and upsampling without considering information enhancement between different scale features, resulting in low segmentation accuracy. Introducing an attention mechanism between each downsampling and upsampling will lead to low segmentation efficiency.
[0008] Accordingly, a new technical solution is needed in this field to solve the above problems. Summary of the Invention
[0009] To overcome the above-mentioned shortcomings, this invention is proposed to provide an intelligent image segmentation method and system that solves, or at least partially solves, the technical problem of how to simultaneously improve image segmentation accuracy and segmentation efficiency.
[0010] In a first aspect, an intelligent image segmentation method is provided, the method comprising:
[0011] Obtain the image to be segmented;
[0012] The image to be segmented is input into a trained image segmentation network, and the image segmentation result is obtained based on the following steps, wherein the image segmentation network includes multiple downsampling modules, multiple location attention modules, multiple upsampling modules, and one channel attention module:
[0013] The image is downsampled using the multiple downsampling modules to obtain multiple first feature matrices at different scales.
[0014] Based on the multiple positional attention modules, convolution operations and positional attention calculations are performed on the multiple first feature matrices respectively to obtain multiple second feature matrices of different scales;
[0015] Based on the multiple upsampling modules, the multiple second feature matrices are upsampled to obtain multiple third feature matrices of different scales;
[0016] The plurality of third feature matrices are processed to obtain a fourth feature matrix of a preset output size;
[0017] The image segmentation result is obtained by performing channel attention calculation on the fourth feature matrix of the preset output size based on the channel attention module.
[0018] In one technical solution of the above-mentioned intelligent image segmentation method, the step of performing convolution operations on the plurality of first feature matrices based on the plurality of position attention modules includes:
[0019] Perform convolution operations with different kernel sizes on all of the first feature matrices.
[0020] Alternatively, perform multiple convolution operations with different kernel sizes on a portion of the first feature matrices, and perform a single convolution operation on the other first feature matrices.
[0021] In one technical solution of the above-mentioned intelligent image segmentation method, the step of performing convolution operations and positional attention calculations on the multiple first feature matrices based on the multiple positional attention modules to obtain multiple second feature matrices of different scales includes:
[0022] Based on different positional attention modules, the convolution operation and positional attention calculation are performed on the first feature matrices of different scales to obtain the corresponding positional attention matrices;
[0023] Each of the corresponding position attention matrices is added to the corresponding first feature matrix to obtain the corresponding plurality of second feature matrices.
[0024] In one technical solution of the above-mentioned intelligent image segmentation method, the step of performing positional attention calculations on the plurality of first feature matrices based on the plurality of positional attention modules includes:
[0025] Obtain the scale feature matrix obtained from the convolution operation;
[0026] The corresponding positional attention matrix is obtained based on the scale feature matrix obtained from the convolution operation.
[0027] In one technical solution of the above-mentioned intelligent image segmentation method, obtaining the scale feature matrix obtained by the convolution operation includes:
[0028] When the convolution operation is to perform multiple convolution operations on all the first feature matrices with convolution kernels of different sizes, multiple feature matrices of different scales corresponding to each first feature matrix are obtained.
[0029] When the convolution operation is to perform multiple convolution operations with different-sized convolution kernels on a portion of the first feature matrix and perform a single convolution operation on the other first feature matrices, multiple feature matrices of different scales corresponding to the portion of the first feature matrix are obtained, and a single-scale feature matrix corresponding to the other first feature matrix is obtained.
[0030] In one technical solution of the above-mentioned intelligent image segmentation method, the step of processing the plurality of third feature matrices to obtain a fourth feature matrix of a preset output size includes:
[0031] The third feature matrix obtained from each upsampling operation is saved to obtain multiple third feature matrices at different scales;
[0032] The plurality of third feature matrices are adjusted to a preset output size to obtain a plurality of third feature matrices with a preset output size;
[0033] The third feature matrices of the multiple preset output sizes are fused to obtain the fourth feature matrix of the preset output size.
[0034] In one technical solution of the above-mentioned intelligent image segmentation method, the channel attention calculation based on the channel attention module on the fourth feature matrix of the preset output size includes:
[0035] The fourth feature matrix is compressed in dimension based on global average pooling.
[0036] The weights of each channel are predicted based on the fourth feature matrix after dimensional compression, and the channel attention is calculated based on the weights to obtain the fifth feature matrix.
[0037] In one technical solution of the above-mentioned intelligent image segmentation method, the method further includes:
[0038] The fifth feature matrix is convolved to obtain the image segmentation result containing the predicted label matrix.
[0039] In a second aspect, an intelligent image segmentation system is provided, comprising a processor and a storage device, the storage device being adapted to store multiple lines of program code, the program code being adapted to be loaded and run by the processor to perform the intelligent image segmentation method described in any of the above-described technical solutions.
[0040] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and run by a processor to perform the intelligent image segmentation method described in any of the above-described intelligent image segmentation methods.
[0041] The present invention comprises one or more of the following technical solutions:
[0042] Beneficial effects:
[0043] In implementing the technical solution of the present invention, an image to be segmented is acquired, the image to be segmented is input into a trained image segmentation network, and an image segmentation result is obtained based on the following steps: wherein the image segmentation network includes multiple downsampling modules, multiple location attention modules, multiple upsampling modules, and one channel attention module.
[0044] Multiple downsampling modules are used to perform multiple downsampling operations on the image to obtain multiple first feature matrices at different scales. Multiple positional attention modules are used to perform convolution operations and positional attention calculations on the multiple first feature matrices to obtain multiple second feature matrices at different scales. Multiple upsampling modules are used to perform upsampling operations on the multiple second feature matrices to obtain multiple third feature matrices at different scales. The multiple third feature matrices are processed to obtain a fourth feature matrix of a preset output size. The fourth feature matrix of the preset output size is then subjected to channel attention calculations using a channel attention module to obtain the image segmentation result.
[0045] Through the above implementation method, image segmentation results can be obtained based on the trained image segmentation network. In the encoding stage, the positional information of specific features is extracted through the positional attention module, which can highlight the relative position of features at different scales. In the decoding stage, multi-scale features are processed and the channel attention module is used to extract the channel dimension information of multi-scale features. Compared with the network structure of the existing UNET network parallel attention mechanism, it can effectively improve the accuracy of image segmentation while improving the efficiency of segmentation. Attached Figure Description
[0046] The disclosure of this invention will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Wherein:
[0047] Figure 1 This is a schematic flowchart of the main steps of an intelligent image segmentation method according to an embodiment of the present invention;
[0048] Figure 2 This is a schematic diagram of the structure of an image segmentation network according to an embodiment of the present invention;
[0049] Figure 3 This is a schematic diagram of image segmentation network training according to an embodiment of the present invention;
[0050] Figure 4 This is a schematic diagram of the main steps of obtaining multiple second feature matrices of different scales by performing convolution operations and positional attention calculations on multiple first feature matrices based on multiple positional attention modules according to an embodiment of the present invention.
[0051] Figure 5 This is a schematic diagram of the main steps of calculating positional attention for multiple first feature matrices based on multiple positional attention modules according to an embodiment of the present invention.
[0052] Figure 6 This is a schematic diagram of the main steps of processing a third feature matrix of multiple scales to obtain a fourth feature matrix of a preset output size according to an embodiment of the present invention.
[0053] Figure 7 This is a schematic diagram of the main steps of calculating channel attention for a fourth feature matrix of a preset output size based on a channel attention module according to an embodiment of the present invention.
[0054] Figure 8 This is a schematic diagram of the main structure of an embodiment of an intelligent image segmentation system according to the present invention.
[0055] List of reference numerals in the attached diagram:
[0056] 801: Processor; 802: Storage device. Detailed Implementation
[0057] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.
[0058] In the description of this invention, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Non-transitory computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.
[0059] Here we will first explain some of the terms involved in this invention.
[0060] CNN (Convolutional Neural Networks) is a class of deep feedforward neural networks that incorporate convolutional computations. It is one of the representative algorithms of deep learning. Convolutional neural networks possess representation learning capabilities, enabling them to perform shift-invariant classification of input information according to their hierarchical structure; therefore, they are also known as shift-invariant artificial neural networks (SIANN).
[0061] UNet is a CNN network consisting of 4 downsampling layers, 4 upsampling layers, and a skip connection-like structure. Its key feature is that the convolutional layers are completely symmetrical in their downsampling and upsampling portions, and the feature maps at the downsampling end can skip deep sampling and be concatenated to the corresponding upsampling end. Initially, UNet was primarily used for semantic segmentation of medical images, and its application has since been extended to semantic segmentation of 3D video data and super-resolution image generation. UNet is a versatile fully convolutional network, and several improved versions have been developed for specific problems, such as HDense-UNet, which incorporates residual blocks at the downsampling end, and UNet++, which includes deep supervision and model pruning.
[0062] Encoder-decoder architecture: This refers to generating an output sequence y from an input sequence x. Encoding involves transforming the input sequence into a fixed-length vector; decoding involves transforming the previously generated fixed vector back into the output sequence.
[0063] Downsampling: A simple feature extraction technique used in convolutional layers, often called extraction, involves extracting a smaller portion from a larger set. In image processing, downsampling essentially reduces the size of an image to fit the display area, generating a thumbnail. For example, pooling or convolutional layers in CNN networks are examples of downsampling. However, while convolution reduces the image size for feature extraction, pooling downsampling reduces the dimensionality of those features.
[0064] Upsampling: After downsampling for feature extraction, the input image needs to be restored to its original size for further pixel classification, a process known as semantic segmentation. Common upsampling methods include bilinear interpolation, deconvolution, and unpooling.
[0065] Location attention: used to capture the spatial dependency between any two locations in a feature map. For a given feature, it is updated by a weighted sum of features at all locations. The weights are the feature similarity between the corresponding two locations. Therefore, any two locations with existing similar features can contribute to each other's improvement, regardless of the distance between them.
[0066] Channel attention: used to learn the importance of each channel. It is achieved by compressing the space of the feature map and then learning along the channel dimension to obtain the importance of each channel.
[0067] Convolution: In image processing, convolution refers to performing a series of operations on each pixel of an image using a convolution kernel. The convolution kernel is a matrix used in image processing, also known as a mask, and is a parameter used in operations with the original image. A convolution kernel is typically a square grid structure (e.g., a 3x3 matrix or pixel region), where each square has a weight value. When performing convolution calculations, the center of the kernel is placed on the pixel to be calculated. The kernel's product with the pixel value it covers is then calculated sequentially, and the sum of these products represents the new pixel value at that location.
[0068] Pooling: Pooling is a method for compressing images. Its significance lies in feature dimensionality reduction. Pooling technology significantly reduces the consumption of computational resources and also has the advantage of reducing model overfitting. The idea of pooling comes from image feature aggregation statistics. Simply put, although pooling blurs the image, it does not affect image recognition or location determination. Another advantage of pooling is translation invariance; if an object undergoes a small translation in the image (not exceeding the receptive field), such displacement will not affect the pooling effect, thus not affecting the model's feature map extraction.
[0069] Linear activation function: A special type of activation function that does not use any other activation function. Instead, it directly multiplies the input by a weight coefficient and uses the output as its output. It is the simplest and most commonly used activation function. Linear activation functions can simplify the construction of neural networks in computers and can achieve non-linear processing, but its performance is not as good as that of non-linear activation functions.
[0070] Softmax layer: A fully connected layer used to map the outputs of multiple neurons in a convolutional neural network to the (0, 1) interval, giving the probability of each classification.
[0071] Sigmoid function: A common sigmoid function in biology, also known as an sigmoid growth curve. In information science, due to its monotonically increasing and inverse monotonically increasing properties, the Sigmoid function is often used as an activation function for neural networks, mapping variables to the (0, 1) interval.
[0072] As described in the background section, in existing deep learning methods, introducing an attention mechanism within a single scale feature between downsampling and upsampling does not take into account the information enhancement between different scale features, resulting in low segmentation accuracy. Introducing an attention mechanism between each downsampling and upsampling will lead to low segmentation efficiency.
[0073] To address the aforementioned problems, this invention provides an intelligent image segmentation method and system.
[0074] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of an intelligent image segmentation method according to an embodiment of the present invention. Figure 1 As shown, the intelligent image segmentation method in this embodiment of the invention mainly includes the following steps S101 to S107.
[0075] Step S101: Obtain the image to be segmented.
[0076] Step S102: Input the image to be segmented into the trained image segmentation network and obtain the image segmentation result based on the following steps.
[0077] The image segmentation network includes multiple downsampling modules, multiple location attention modules, multiple upsampling modules, and one channel attention module.
[0078] Step S103: Perform downsampling operations on the image based on multiple downsampling modules to obtain multiple first feature matrices at different scales.
[0079] Step S104: Perform convolution operations and positional attention calculations on multiple first feature matrices based on multiple positional attention modules to obtain multiple second feature matrices of different scales.
[0080] Step S105: Perform upsampling operations on multiple second feature matrices based on multiple upsampling modules to obtain multiple third feature matrices at different scales.
[0081] Step S106: Process the multiple third feature matrices to obtain a fourth feature matrix with a preset output size.
[0082] Step S107: Perform channel attention calculation on the fourth feature matrix of the preset output size based on the channel attention module to obtain the image segmentation result.
[0083] Based on the method described in steps S101 to S107 above, image segmentation results can be obtained based on the trained image segmentation network. In the encoding stage, the positional information of specific features is extracted through the positional attention module, which can highlight the relative position of features at different scales. In the decoding stage, multi-scale features are processed, and the channel dimension information of multi-scale features is extracted using the channel attention module. Compared with the network structure of the existing UNET network parallel attention mechanism, this method effectively improves the accuracy of image segmentation while also improving the efficiency of segmentation.
[0084] The following provides further explanation of steps S101 to S107.
[0085] In some embodiments of step S101 above, the image to be segmented is a single-modal grayscale image, such as computed tomography (CT), single-photon emission computed tomography (SPECT), ultrasound (US), magnetic resonance imaging (MRI), etc. in medical images.
[0086] Specifically, taking CT images as an example, step S101 includes the following steps S1011 to S1012.
[0087] Step S1011: Perform CT irradiation on the case using medical equipment to obtain the initial DCM imaging file.
[0088] DCM files are a type of file that conforms to the Digital Imaging and Communications in Medicine (DICOM) standard. It is a digital imaging technology that can be used to store various image information. The DICOM standard supports a variety of medical devices, including electrocardiograms, magnetic resonance imaging, angiography, and echocardiography. Therefore, DCM files are widely used in the medical industry.
[0089] Step S1012: Preprocess the DCM imaging file.
[0090] Specifically, including:
[0091] (1) Parse the DCM imaging file into a TIF image;
[0092] Since DCM files cannot be directly used for neural network training, they are parsed into TIF images.
[0093] TIF images (also known as TIFF images), or TagImageFileFormat, are tag image file formats. They are one of the most commonly used image file formats in graphic image processing. Although the image format is very complex, it has been widely used because it is flexible in storing image information, can support many color systems, and is independent of the operating system.
[0094] (2) Perform grayscale truncation on the TIF image;
[0095] Gray-scale truncation of TIF images can better highlight the color, texture, and boundaries of organs.
[0096] (3) Normalize the image.
[0097] Normalization refers to the process of performing a series of standard processing transformations on an image to transform it into a fixed standard form.
[0098] Generally speaking, the image normalization process includes four steps: coordinate centering, x-shearing normalization, scaling normalization, and rotation normalization. Normalization makes images resistant to geometric transformations by identifying invariants in the image, thus revealing that the images were originally identical or part of a series.
[0099] Through steps S1011 to S1012 described above, the image to be segmented can be obtained. It should be noted that the examples of the image to be segmented and the methods for obtaining the image to be segmented above are merely illustrative. The image to be segmented can be any single-modal grayscale image. In practical applications, those skilled in the art can choose the appropriate method for obtaining the image to be segmented according to the specific scenario, and no limitation is made here.
[0100] The above is a further explanation of step S101. The following is a further explanation of step S102.
[0101] In some implementations of step S102 above, the image segmentation network can be improved based on the existing CNN network, such as adding a position attention module between the downsampling module and the upsampling module in the UNET network encoder-decoder structure, and adding a channel attention module after the upsampling module.
[0102] It should be noted that the above only uses the UNET network as an example. The intelligent image segmentation method provided by this invention can also be applied to any encoder-decoder structure CNN network, and is not limited here.
[0103] In some implementations, see Appendix Figure 2 , Figure 2 This is a schematic diagram of the structure of an image segmentation network according to an embodiment of the present invention. Figure 2 As shown, the image segmentation network includes four downsampling modules, four location attention modules, four upsampling modules, and one channel attention module.
[0104] Further, see appendix. Figure 3 , Figure 3 This is a schematic diagram of image segmentation network training according to an embodiment of the present invention.
[0105] In some implementations, the method for training an image segmentation network mainly includes the following steps:
[0106] (1) Obtain the original images and perform preprocessing to obtain the training set;
[0107] Taking CT images as an example, the details of acquiring the original image and performing preprocessing are described in steps S1011 to S1012 above, and will not be repeated here.
[0108] (2) Convert the training set into a tensor input image segmentation network;
[0109] In deep learning, a Tensor is a multilinear mapping defined on the Cartesian product of some vector spaces and some dual spaces. Its coordinates are quantities with |n| components in |n|-dimensional space, where each component is a function of the coordinates. During coordinate transformations, these components also undergo linear transformations according to certain rules. In deep learning, a Tensor is essentially a multidimensional array, designed to create higher-dimensional matrices and vectors.
[0110] (3) The downsampling module, position attention module, upsampling module and channel attention module in the image segmentation network are trained based on the training set;
[0111] (4) The error function is calculated based on the original classification label matrix, and the image segmentation network is updated and backpropagated based on the calculation results;
[0112] The original classification label matrix is formed by labeling the original image with classification labels (e.g., labeling different categories of organs in a CT image), and then converting it into a matrix with the same format as the image segmentation result output by the image segmentation network through one-hot encoding. This facilitates the calculation of error functions between the predicted classification label matrices of different categories output by the image segmentation network and the original classification labels.
[0113] Furthermore, based on the error function calculation results, including weight parameters, gradients, etc., the image segmentation network is updated and backpropagated.
[0114] (5) When the error function of the image segmentation network converges to the preset error value, the training of the image segmentation network is completed.
[0115] The preset error value is a stable low value. Those skilled in the art can set the preset error value according to the actual application scenario, and there is no limitation here.
[0116] Furthermore, after the image segmentation network is trained, the image to be segmented can be input into the trained image segmentation network to obtain the image segmentation result.
[0117] The above is a further explanation of step S102. The following is a further explanation of step S103.
[0118] In some embodiments of step S103 above, the downsampling module is a multi-scale encoder structure consisting of convolutional layers, pooling layers, and linear activation functions.
[0119] In some implementations, based on Figure 2 The four downsampling modules shown perform downsampling operations on the image to extract features, resulting in four first feature matrices at different scales.
[0120] Feature matrices at different scales represent feature matrices with different viewpoints. The scale includes depth, width, and number of channels.
[0121] The above is a further explanation of step S103. The following is a further explanation of step S104.
[0122] In some embodiments of step S104 above, the multiple position attention modules include multi-scale position attention modules and / or single-scale position attention modules.
[0123] Specifically, see the appendix. Figure 4 , Figure 4 This is a schematic diagram of the main steps of obtaining multiple second feature matrices of different scales by performing convolution operations and positional attention calculations on multiple first feature matrices based on multiple positional attention modules according to an embodiment of the present invention.
[0124] like Figure 4 As shown, step S104 mainly includes the following steps S1041 to S1042:
[0125] Step S1041: Perform convolution operations and positional attention calculations on the first feature matrices of different scales based on different positional attention modules to obtain the corresponding positional attention matrices.
[0126] In some implementations, performing convolution operations on multiple first feature matrices based on multiple positional attention modules mainly includes:
[0127] Perform convolution operations with kernels of different sizes on all the first feature matrices.
[0128] At this point, all the position attention modules are multi-scale position attention modules.
[0129] Alternatively, perform multiple convolution operations with kernels of different sizes on some of the first feature matrices, and perform one convolution operation on the other first feature matrices.
[0130] At this point, the multiple position attention modules include multi-scale position attention modules and single-scale position attention modules.
[0131] Specifically, multiple convolution operations with different kernel sizes can be performed on all the four first feature matrices based on four multi-scale positional attention modules; or multiple convolution operations with different kernel sizes can be performed on the first first feature matrix based on only one multi-scale positional attention module, and a single convolution operation can be performed on the last three first feature matrices based on three single-scale positional attention modules; or multiple convolution operations with different kernel sizes can be performed on the first two first feature matrices based on two multi-scale positional attention modules, and a single convolution operation can be performed on the last two first feature matrices based on two single-scale positional attention modules, etc.
[0132] It should be noted that, based on the comparative data obtained from the ablation experiment, it is more effective to perform multiple convolution operations with different kernel sizes on the first two first feature matrices using two multi-scale positional attention modules, and to perform a single convolution operation on the latter two first feature matrices using two single-scale positional attention modules.
[0133] Furthermore, in some embodiments, see Appendix Figure 5 , Figure 5 This is a schematic flowchart illustrating the main steps of calculating positional attention for multiple first feature matrices based on multiple positional attention modules according to an embodiment of the present invention. Figure 5 As shown, it mainly includes the following steps S501 to S502:
[0134] Step S501: Obtain the scale feature matrix obtained from the convolution operation.
[0135] In some implementations, when the above convolution operation is to perform multiple convolution operations on all first feature matrices with convolution kernels of different sizes, multiple feature matrices of different scales corresponding to each first feature matrix are obtained.
[0136] Specifically, the four first feature matrices obtained in step S103 can be denoted as A1, A2, A3, and A4. Based on four multi-scale positional attention modules, A1, A2, A3, and A4 are subjected to multiple convolution operations with convolution kernels of different sizes.
[0137] For example, based on four multi-scale positional attention modules, three convolution operations with kernels of 3x3, 5x5, and 7x7 are performed on A1, A2, A3, and A4 respectively. Then, for A1, A2, A3, and A4, three scale feature matrices will be obtained. Taking A1 as an example, the three scale feature matrices are denoted as B3, B5, and B7. Then, three 1x1 convolution operations are performed on B3, B5, and B7 respectively to extract features Ci, Di, and Ei, where i is the size of the convolution kernel for the scale features. For example, three 1x1 convolution operations are performed on B3 to extract feature matrices C3, D3, and E3; three 1x1 convolution operations are performed on B5 to extract feature matrices C5, D5, and E5; and three 1x1 convolution operations are performed on B7 to extract feature matrices C7, D7, and E7.
[0138] When the above convolution operation involves performing multiple convolution operations with different kernel sizes on a portion of the first feature matrix and performing a single convolution operation on other first feature matrices, multiple feature matrices of different scales corresponding to the portion of the first feature matrix are obtained, as well as a single-scale feature matrix corresponding to other first feature matrices are obtained.
[0139] Specifically, multiple convolution operations with different kernel sizes can be performed on A1 and A2 based on two multi-scale positional attention modules, and a single convolution operation can be performed on A3 and A4 based on two single-scale positional attention modules.
[0140] For example, by performing three convolution operations on A1 and A2 with kernels of 3x3, 5x5, and 7x7 respectively, based on two multi-scale positional attention modules, three scale feature matrices will be obtained for both A1 and A2. Taking A1 as an example, the three scale feature matrices are denoted as B3, B5, and B7. Then, three 1x1 convolution operations are performed on B3, B5, and B7 respectively to extract features Ci, Di, and Ei, where i is the size of the convolution kernel for the scale features. For example, three 1x1 convolution operations are performed on B3 to extract feature matrices C3, D3, and E3; three 1x1 convolution operations are performed on B5 to extract feature matrices C5, D5, and E5; and three 1x1 convolution operations are performed on B7 to extract feature matrices C7, D7, and E7.
[0141] Based on two single-scale positional attention modules, A3 and A4 are convolved once with a 3x3 kernel to obtain a single-scale feature matrix B3'. Then, B3' is convolved three times with a 1x1 kernel to extract feature matrices C3', D3', and E3'.
[0142] The above is an explanation of step S401.
[0143] Step S502: Obtain the corresponding positional attention matrix based on the scale feature matrix obtained from the convolution operation.
[0144] Taking the example of performing multiple convolution operations on A1 and A2 with convolution kernels of different sizes based on two multi-scale position attention modules, and performing one convolution operation on A3 and A4 based on two single-scale position attention modules, the feature matrices Ci and Di extracted from the feature matrices B3, B5, and B7 of A1 at the three scales are multiplied by a dot product to obtain the position correlation; then, after passing through a softmax layer, they are multiplied by the feature matrix Ei to obtain the position attention matrix.
[0145] The above is an explanation of step S1041.
[0146] Step S1042: Add each corresponding position attention matrix to the corresponding first feature matrix to obtain multiple corresponding second feature matrices.
[0147] Specifically, each first feature matrix is added to its corresponding position attention matrix to obtain the position-enhanced feature matrix, denoted as G.
[0148] Taking the example of performing multiple convolution operations with different-sized kernels on A1 and A2 based on two multi-scale positional attention modules, and performing one convolution operation on A3 and A4 based on two single-scale positional attention modules, for A1 and A2, three position-enhanced features G3, G5, and G7 can be obtained. At this time, multi-scale fusion of G3, G5, and G7 using a 1x1 convolution operation can obtain the second feature matrix of the corresponding scale of A1 and A2; for A3 and A4, one position-enhanced second feature matrix can be obtained. Based on A1, A2, A3, and A4, four second feature matrices of different scales can be obtained.
[0149] The above is a further explanation of step S104. The following is a further explanation of step S105.
[0150] In some embodiments of step S105 above, the upsampling module can be a multi-scale decoder structure composed of a deconvolution layer, a bilinear interpolation layer, or an anti-pooling layer.
[0151] Because deconvolution operations have better feature representation capabilities, in some implementations, the upsampling module can use a multi-scale decoder structure composed of deconvolution layers.
[0152] Specifically, multiple second feature matrices are put into their respective upsampling modules, and upsampling is performed using a deconvolution operation with a 3x3 kernel to obtain multiple third feature matrices of different scales.
[0153] It should be noted that the above description of the upsampling module is only illustrative. In practical applications, those skilled in the art can set the structure of the upsampling module according to the specific scenario, and no limitation is made here.
[0154] The above is a further explanation of step S105. The following is a further explanation of step S106.
[0155] In some embodiments of step S106 above, see Appendix Figure 6 , Figure 6 This is a schematic flowchart illustrating the main steps of processing a third feature matrix of multiple scales to obtain a fourth feature matrix of a preset output size according to an embodiment of the present invention. Figure 6 As shown, it mainly includes the following steps S601 to S603:
[0156] Step S601: Save the third feature matrix obtained from each upsampling operation to obtain the third feature matrix at multiple scales.
[0157] Step S602: Adjust the multiple third feature matrices to a preset output size to obtain multiple third feature matrices with preset output sizes.
[0158] In some implementations, the preset output size is the same as the size of the image to be segmented in the input image segmentation network, that is, multiple third feature matrices of different scales are all adjusted to the size of the image to be segmented.
[0159] In other implementations, the preset output size may be different from the size of the image to be segmented in the input image segmentation network, which is not limited here.
[0160] Step S603: Perform feature fusion on multiple third feature matrices with preset output sizes to obtain a fourth feature matrix with preset output sizes.
[0161] Specifically, multiple third feature matrices of the same size obtained in step S602 can be spliced together to obtain a fourth feature matrix of a preset output size.
[0162] The above is a further explanation of step S106. The following is a further explanation of step S107.
[0163] In some embodiments of step S107 above, see Appendix Figure 7 , Figure 7 This is a schematic flowchart illustrating the main steps of calculating channel attention for a fourth feature matrix of a preset output size based on a channel attention module according to an embodiment of the present invention. Figure 7 As shown, it mainly includes the following steps S701 to S702:
[0164] Step S701: Perform dimensionality compression on the fourth feature matrix based on global average pooling operation.
[0165] Specifically, the scale of the fourth feature matrix is H×W×C (H, W, and C represent height, width, and number of channels, respectively). Dimension compression compresses the scale of the fourth feature matrix from H×W×C to 1×1×C, that is, compresses H×W to 1×1. This step is achieved by global average pooling.
[0166] The global average pooling operation calculates the average value of the entire feature matrix, compressing the scale from H×W×C to 1×1×C, resulting in four feature matrices of the same size but different channel dimensions.
[0167] Step S702: Predict the weight of each channel based on the fourth feature matrix after dimensional compression, and calculate the channel attention based on the weight to obtain the fifth feature matrix.
[0168] In some implementations, the fourth feature matrix with a scale of 1×1×C obtained from the above compression is incorporated into the fully connected layer to predict the importance of each channel, that is, weighting different dimensions separately. Furthermore, the sigmoid activation function is used in the channel attention module to perform channel attention calculation on the fourth feature matrix to obtain the weighted fifth feature matrix.
[0169] The above is a further explanation of step S107.
[0170] In some embodiments, after completing step S107, the method further includes:
[0171] The fifth feature matrix is convolved to obtain the image segmentation result containing the predicted label matrix.
[0172] For example, if the image to be segmented in the input image segmentation network is a CT image, a prediction label matrix containing the number of organ categories can be obtained after one convolution.
[0173] The method provided by this invention utilizes a multi-scale dual-attention image segmentation network to enhance the information of shallow CT image features, thereby improving the feature recognition accuracy of small organs and the efficiency of image segmentation. Therefore, the aforementioned image segmentation network can also be applied to other general-purpose encoding / decoding segmentation networks.
[0174] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effects of the present invention, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders, and these variations are all within the scope of protection of the present invention.
[0175] Those skilled in the art will understand that all or part of the processes in the method of the above embodiment of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0176] Furthermore, this invention also provides an intelligent image segmentation system. (See appendix.) Figure 8 , Figure 8 This is a schematic diagram of the main structure of an embodiment of an intelligent image segmentation system according to the present invention. Figure 8 As shown, the intelligent image segmentation system in this embodiment of the invention mainly includes a processor 801 and a storage device 802. The storage device 802 can be configured to store a program for executing the intelligent image segmentation method of the above-described method embodiment. The processor 801 can be configured to execute the program in the storage device 802, which includes, but is not limited to, a program for executing the intelligent image segmentation method of the above-described method embodiment. For ease of explanation, only the parts related to the embodiments of the present invention are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of the present invention.
[0177] In some possible embodiments of the present invention, the intelligent image segmentation system may include multiple processors 801 and multiple storage devices 802. The program executing the intelligent image segmentation method of the above method embodiments can be divided into multiple subroutines, each of which can be loaded and run by a processor 801 to perform different steps of the intelligent image segmentation method of the above method embodiments. Specifically, each subroutine can be stored in different storage devices 802, and each processor 801 can be configured to execute programs in one or more storage devices 802 to jointly implement the intelligent image segmentation method of the above method embodiments; that is, each processor 801 executes different steps of the intelligent image segmentation method of the above method embodiments to jointly implement the intelligent image segmentation method of the above method embodiments.
[0178] Furthermore, the present invention also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program for executing the intelligent image segmentation method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described intelligent image segmentation method. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.
[0179] The technical solution of the present invention has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions resulting from such changes or substitutions will all fall within the scope of protection of the present invention.
Claims
1. An intelligent image segmentation method, characterized in that, The method includes: Obtain the image to be segmented; The image to be segmented is input into a trained image segmentation network, and the image segmentation result is obtained based on the following steps, wherein the image segmentation network includes multiple downsampling modules, multiple location attention modules, multiple upsampling modules, and one channel attention module: The image is downsampled using the multiple downsampling modules to obtain multiple first feature matrices at different scales. Based on the multiple positional attention modules, convolution operations and positional attention calculations are performed on the multiple first feature matrices respectively to obtain multiple second feature matrices of different scales; Based on the multiple upsampling modules, the multiple second feature matrices are upsampled to obtain multiple third feature matrices of different scales; The plurality of third feature matrices are processed to obtain a fourth feature matrix of a preset output size; Based on the channel attention module, channel attention is calculated on the fourth feature matrix of the preset output size to obtain the image segmentation result; The convolution operation on the multiple first feature matrices based on the multiple position attention modules includes: performing multiple convolution operations on all the first feature matrices with convolution kernels of different sizes; or, performing multiple convolution operations on some of the first feature matrices with convolution kernels of different sizes, and performing a single convolution operation on the other first feature matrices. The step of performing convolution operations and positional attention calculations on the multiple first feature matrices based on the multiple positional attention modules to obtain multiple second feature matrices of different scales includes: performing the convolution operation and positional attention calculations on the first feature matrices of different scales based on different positional attention modules to obtain corresponding positional attention matrices; and adding each corresponding positional attention matrix to the corresponding first feature matrix to obtain the corresponding multiple second feature matrices.
2. The intelligent image segmentation method according to claim 1, characterized in that, The step of performing position attention calculations on the multiple first feature matrices based on the multiple position attention modules includes: Obtain the scale feature matrix obtained from the convolution operation; The corresponding positional attention matrix is obtained based on the scale feature matrix obtained from the convolution operation.
3. The intelligent image segmentation method according to claim 2, characterized in that, The process of obtaining the scale feature matrix obtained by the convolution operation includes: When the convolution operation is to perform multiple convolution operations on all the first feature matrices with convolution kernels of different sizes, multiple feature matrices of different scales corresponding to each first feature matrix are obtained. When the convolution operation is to perform multiple convolution operations with different-sized convolution kernels on a portion of the first feature matrix and perform a single convolution operation on the other first feature matrices, multiple feature matrices of different scales corresponding to the portion of the first feature matrix are obtained, and a single-scale feature matrix corresponding to the other first feature matrix is obtained.
4. The intelligent image segmentation method according to claim 1, characterized in that, The process of processing the plurality of third feature matrices to obtain a fourth feature matrix of a preset output size includes: The third feature matrix obtained from each upsampling operation is saved to obtain multiple third feature matrices at different scales; The plurality of third feature matrices are adjusted to a preset output size to obtain a plurality of third feature matrices with a preset output size; The third feature matrices of the multiple preset output sizes are fused to obtain the fourth feature matrix of the preset output size.
5. The intelligent image segmentation method according to claim 1, characterized in that, The channel attention calculation based on the fourth feature matrix of the preset output size by the channel attention module includes: The fourth feature matrix is compressed in dimension based on global average pooling. The weights of each channel are predicted based on the fourth feature matrix after dimensional compression, and the channel attention is calculated based on the weights to obtain the fifth feature matrix.
6. The intelligent image segmentation method according to claim 5, characterized in that, The method further includes: The fifth feature matrix is convolved to obtain the image segmentation result containing the predicted label matrix.
7. An intelligent image segmentation system, comprising a processor and a storage device, wherein the storage device is adapted to store multiple lines of program code, characterized in that, The program code is adapted to be loaded and run by the processor to perform the intelligent image segmentation method according to any one of claims 1 to 6.
8. A computer-readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the intelligent image segmentation method according to any one of claims 1 to 6.