Image segmentation method based on deformable convolution
By using a deformable convolution kernel to dynamically adjust the size and shape of the convolution kernel in the image segmentation algorithm, combined with feature extraction and upsampling network, the problem of poor effect of traditional image segmentation algorithm in complex segmentation scenarios is solved, and more efficient segmentation performance.
Patent Information
- Application Number
- CN202410512249.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-26
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2044-04-26
AI Technical Summary
Traditional image segmentation algorithms cannot obtain ideal results when processing complex and fine segmentation scenarios, especially high noise and low resolution medical images, and the convolution kernel size and receptive field require artificial design, which is difficult to adapt to the needs of different tasks. .
Using the image segmentation method based on deformable convolution, the size and shape of the convolution kernel are dynamically adjusted by using the deformable convolution kernel in the feature extraction network, and combining feature extraction and upsampling network for layer-by-layer feature extraction and recovery, realizing Adaptation to different segmentation scenarios.
It effectively improves the performance of the image segmentation algorithm, can better adapt to different segmentation scenarios, reduce noise interference, and improve segmentation effect.
Smart Images
Figure CN118397274B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning image segmentation, and in particular to an image segmentation method based on deformable convolution. Background Art
[0002] Traditional image segmentation algorithms all have major drawbacks. For example, traditional threshold segmentation, region segmentation, edge segmentation, or traditional methods based on clustering often require manual design and are more suitable for simpler segmentation scenarios. However, for more sophisticated and complex segmentation scenarios, such as medical images with high noise and low resolution problems, ideal results are often not achieved.
[0003] Image segmentation algorithms based on deep learning can better solve the problems of traditional image segmentation algorithms. Through the encoder-decoder structure, the discriminative feature semantics contained in the image can be gradually projected to the pixel space consistent with the original spatial resolution for dense prediction and classification, thereby meeting the requirements for more complex and fine image segmentation scenarios. However, the convolution kernels used in traditional encoders are often of fixed size, and their parameters also need to be manually designed. For different tasks, the size of the convolution kernel often needs to be manually designed. If the receptive field is too small, it is not enough to model long-distance dependency and capture large-sized objects; if the receptive field is too large, it is easy to introduce too much noise and invalid information, which will ultimately affect the overall segmentation performance. Summary of the invention
[0004] The purpose of the invention is to provide an image segmentation method based on deformable convolution, through the deformable convolution kernel in the feature extraction network, the size and shape of the convolution kernel can be adjusted according to different segmentation scenarios, so as to provide solutions to adapt to different segmentation scenarios.
[0005] Technical solution: The image segmentation method based on deformable convolution described in the present invention comprises the following steps:
[0006] Step 1, using an image acquisition device to obtain an original image, and preprocessing the original image to obtain an input image;
[0007] Step 2: The input image obtained after image preprocessing is used as the input of the feature extraction network. The feature extraction network extracts features of the input image layer by layer, and then outputs the image features and semantic information extracted from each layer to the corresponding layer of the upsampling network through the cross connection;
[0008] Step 3: The upsampling network restores the image size layer by layer by combining the image features and semantic information of each layer to obtain the network prediction segmentation result;
[0009] Step 4: training the image segmentation network according to the comparison result between the network predicted segmentation result and the corresponding true segmentation result, wherein the image segmentation network includes a feature extraction network and an upsampling network;
[0010] Step 5: Based on the final segmentation result, the segmentation effect of the image segmentation network is evaluated and the evaluation result is visualized.
[0011] Furthermore, in step 1, the specific steps of using the image acquisition device to obtain the original image are:
[0012] The raw file data stream is obtained using the musculoskeletal ultrasound acquisition device;
[0013] Then, the original file data stream is preprocessed to convert the original file data stream into an original image in an image format that can be processed by a computer;
[0014] The specific steps of preprocessing the original image to obtain the input image are:
[0015] A two-stage target detection algorithm is used to detect and crop the region of interest of the original image, remove the remaining invalid information, and extract effective information for the segmentation task through downsampling operation through hybrid pooling to obtain the input image.
[0016] Furthermore, in step 2, the specific steps of the feature extraction network for extracting features of the input image layer by layer are as follows:
[0017] Step 2.1, using a feature extraction network containing multiple deformable convolutional blocks to extract features of the input image layer by layer, and mapping the input image into a high-dimensional feature vector;
[0018] Step 2.2, using repeated feature extraction and mapping operations to enrich the semantic information contained in the extracted high-dimensional feature vector;
[0019] In step 2.3, the high-dimensional feature vector learned by the feature extraction network is projected into a pixel space consistent with the resolution of the original image for dense prediction and classification.
[0020] Furthermore, in step 2.1, the deformable convolution block includes a deformable convolution kernel, and the deformable convolution kernel is provided with learnable parameters for adjusting its internal parameters for different data sets to dynamically adjust the size and shape of the deformable convolution kernel.
[0021] Furthermore, in step 2.1, the expression for feature extraction by the deformable convolutional layer is:
[0022] O=Conv(LN(GELU(Dconv(LN(GELU(Conv(I)))))))
[0023] Where I represents the input image, Conv is the normal convolution operation, Dconv is the deformable convolution operation, GELU is the activation function, LN represents layer normalization, and O represents the extracted features.
[0024] Furthermore, in step 3, the specific steps of restoring the image size layer by layer by combining the image features and semantic information of each layer by the upsampling network are:
[0025] Step 3.1, the high-dimensional feature vectors of different levels obtained by the feature extraction network mapping are cropped and spliced through the cross-connection and the generated information of the corresponding level of the upsampling network to achieve feature fusion;
[0026] Step 3.2, use the upsampling method to gradually restore the image size and obtain the upsampled features;
[0027] In step 3.3, the feature vector extracted by the feature extraction network is upsampled to a fixed resolution to achieve end-to-end training, thereby completing pixel-level dense prediction classification and generating a segmentation mask for the corresponding image.
[0028] Furthermore, in step 3.2, the specific steps of gradually restoring the image size using the upsampling method are as follows:
[0029] Step 3.2.1, using cropping and splicing operations to perform feature fusion on the high-dimensional feature vector extracted by the feature extraction network and the up-sampled features extracted by the up-sampling network;
[0030] Step 3.2.2, use the deconvolution operation to restore the image size, repeat the feature fusion operation in step 3.2.1 multiple times until the segmentation result with the same size as the input image is obtained, the expression is:
[0031] P = deconv(concat(I, O))
[0032] In the formula, deconv represents the deconvolution operation, concat represents the concatenation operation, O represents the generated information of the current layer, I represents the high-dimensional feature vector used by the current layer, and P is the output result of the current layer.
[0033] Furthermore, in step 4, the specific steps for training the image segmentation network are:
[0034] Step 4.1, compare the actual segmentation result with the network predicted segmentation result, and use the loss function to quantitatively describe the gap between the network predicted segmentation result and the actual segmentation result;
[0035] Step 4.2, adjust the image segmentation network parameters using the back propagation algorithm according to the loss function to optimize the network segmentation performance.
[0036] Furthermore, in step 5, the specific steps for evaluating the segmentation effect of the image segmentation network are:
[0037] Step 5.1, input the test set images into the trained image segmentation network, and output the network predicted segmentation results corresponding to each image;
[0038] In step 5.2, the actual segmentation result is compared with the network predicted segmentation result, and the predicted segmentation result is evaluated using the Accuracy and IoU evaluation indicators.
[0039] Furthermore, in step 5, the specific steps for visualizing the evaluation results are as follows:
[0040] Draw the predicted segmentation mask on the original image and set the transparency of the segmentation mask to 50% to generate the final visualization result for visualization display.
[0041] Compared with the prior art, the present invention has the following beneficial effects: in the image segmentation method based on deformable convolution provided by the present invention, the image segmentation network is provided with a deformable convolution kernel, which can dynamically adjust the size and shape of the convolution kernel through learnable parameters according to different segmentation scenarios, thereby effectively improving the problems of the current image segmentation algorithm and providing a solution for adapting to different segmentation scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is a schematic diagram of the method flow of the image segmentation method of the present invention;
[0043] Figure 2 A network structure diagram of the image segmentation method of the present invention;
[0044] Figure 3 This is a structural diagram of a deformable convolution unit of the present invention;
[0045] Figure 4 It is a schematic diagram of the pretreatment process of the present invention. DETAILED DESCRIPTION
[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.
[0047] The image segmentation method based on deformable convolution described in this embodiment is described in detail in Figure 1This embodiment includes the following steps:
[0048] Step 1, using an image acquisition device to obtain an original image, and preprocessing the original image to obtain an input image;
[0049] Step 2: The input image obtained after image preprocessing is used as the input of the feature extraction network. The feature extraction network extracts features of the input image layer by layer, and then outputs the image features and semantic information extracted from each layer to the corresponding layer of the upsampling network through the cross connection;
[0050] Step 3: The upsampling network restores the image size layer by layer by combining the image features and semantic information of each layer to obtain the network prediction segmentation result;
[0051] Step 4: training the image segmentation network according to the comparison result between the network predicted segmentation result and the corresponding true segmentation result, wherein the image segmentation network includes a feature extraction network and an upsampling network;
[0052] Step 5: Based on the final segmentation result, the segmentation effect of the image segmentation network is evaluated and the evaluation result is visualized.
[0053] Furthermore, in step 1, the specific steps of using the image acquisition device to obtain the original image are:
[0054] The raw file data stream is obtained using the musculoskeletal ultrasound acquisition device;
[0055] Then, the original file data stream is preprocessed by the pydicom toolkit in Python, and the original file data stream is converted into an original image in an image format that can be processed by a computer;
[0056] The specific steps of preprocessing the original image to obtain the input image are:
[0057] A two-stage target detection algorithm (DetectoRS) is used to detect and crop the region of interest of the original image, remove the remaining invalid information, and perform downsampling operations through hybrid pooling to extract effective information for the segmentation task, such as the texture feature information of the image, to obtain the input image.
[0058] like Figure 2 As shown, it is a network structure diagram of the image segmentation method of the present invention. The image segmentation network consists of a feature extraction network and an upsampling network, wherein the feature extraction network includes four deformable convolution blocks, and the upsampling network includes four decoders. Information transmission and feature fusion are achieved between the layers of the feature extraction network and the upsampling network through cross-connections.
[0059] Furthermore, in step 2, the specific steps of the feature extraction network for extracting features of the input image layer by layer are as follows:
[0060] Step 2.1, using a feature extraction network containing multiple deformable convolutional blocks to extract features of the input image layer by layer, and mapping the input image into a high-dimensional feature vector;
[0061] Step 2.2, repeat the feature extraction and mapping operations multiple times. As the depth of the feature extraction network layer increases, the semantic information contained in the extracted high-dimensional feature vector becomes richer;
[0062] In step 2.3, the high-dimensional feature vector learned by the feature extraction network is projected to the pixel space with the same resolution as the original image for dense prediction and classification. This is the process of generating a mask, i.e., segmentation. Dense classification prediction refers to the task of classifying whether each pixel in a space with the same size as the original image is the target area, thereby generating a segmentation mask with the same size as the original image.
[0063] Furthermore, in step 2.1, the deformable convolution block includes a deformable convolution kernel, and the deformable convolution kernel is set with learnable parameters for adjusting its internal parameters for different data sets to dynamically adjust the size and shape of the deformable convolution kernel, which can better adapt to the image characteristics of different data sets, thereby eliminating the interference of some background noise and more effectively extracting effective information in the image.
[0064] like Figure 3 As shown, it is a structural diagram of the deformable convolution unit of the present invention, which includes basic units such as ordinary convolution operation Conv, deformable convolution operation Dconv, activation function GELU, etc. When the image is processed by the deformable convolution unit, it is first processed by the residual connection ordinary convolution operation Conv, activation function GELU and layer normalization LN in sequence, and then enters the residual connection deformable convolution operation Dconv, activation function GELU and layer normalization LN for processing, and finally passes through the feedforward neural network layer FFN and the ordinary convolution operation Conv to obtain the output of the current unit.
[0065] Furthermore, in step 2.1, the expression for feature extraction by the deformable convolutional layer is:
[0066] O=Conv(LN(GELU(Dconv(LN(GELU(Conv(I)))))))
[0067] In the formula, I represents the input image, Conv is the normal convolution operation, Dconv is the deformable convolution, GELU is the activation function, LN represents layer normalization, and O represents the extracted features.
[0068] Furthermore, in step 3, the specific steps of restoring the image size layer by layer by combining the image features and semantic information of each layer by the upsampling network are:
[0069] Step 3.1, the high-dimensional feature vectors of different levels obtained by the feature extraction network mapping are cropped and spliced through the cross-connection and the generated information of the corresponding level of the upsampling network to achieve feature fusion;
[0070] Step 3.2, use the upsampling method to gradually restore the image size and obtain the upsampled features;
[0071] In step 3.3, the feature vector extracted by the feature extraction network is upsampled to a fixed resolution to achieve end-to-end training, thereby completing pixel-level dense prediction classification and generating a segmentation mask for the corresponding image.
[0072] Furthermore, in step 3.2, the specific steps of gradually restoring the image size using the upsampling method are as follows:
[0073] Step 3.2.1, using cropping and splicing operations to perform feature fusion on the high-dimensional feature vector extracted by the feature extraction network and the up-sampled features extracted by the up-sampling network;
[0074] Step 3.2.2, use the deconvolution operation to restore the image size, repeat the feature fusion operation in step 3.2.1 multiple times until the segmentation result with the same size as the input image is obtained, the expression is:
[0075] P = deconv(concat(I, O))
[0076] In the formula, deconv represents the deconvolution operation, concat represents the concatenation operation, O represents the generated information of the current layer, I represents the high-dimensional feature vector used by the current layer, and P is the output result of the current layer.
[0077] Furthermore, in step 4, the specific steps for training the image segmentation network are:
[0078] Step 4.1, compare the actual segmentation result with the network predicted segmentation result, and use the loss function to quantitatively describe the gap between the network predicted segmentation result and the actual segmentation result;
[0079] Step 4.2, according to the loss function, use the back propagation algorithm to adjust the image segmentation network parameters to optimize the network segmentation performance and improve the accuracy of the network segmentation results.
[0080] Furthermore, in step 5, the specific steps for evaluating the segmentation effect of the image segmentation network are:
[0081] Step 5.1, input the test set images into the trained image segmentation network, and output the network predicted segmentation results corresponding to each image;
[0082] Step 5.2: Compare the actual segmentation result with the network predicted segmentation result, and evaluate the predicted segmentation result using the Accuracy and IoU evaluation indicators. Accuracy is calculated as the ratio of the number of correctly classified pixels to the total number of pixels in the segmentation result, and IoU is calculated as the ratio of the intersection and union between the predicted segmentation result and the true segmentation result.
[0083] Furthermore, in step 5, the specific steps for visualizing the evaluation results are as follows:
[0084] Draw the predicted segmentation mask on the original image and set the transparency of the segmentation mask to 50% to generate the final visualization result for visualization display.
[0085] like Figure 4 As shown, it is a schematic diagram of the preprocessing process of the present invention. The original file data stream is processed by the pydicom toolkit in Python and converted into an image format that can be processed by a computer, and then the input image of the input image segmentation network is obtained through a cropping operation.
[0086] The steps in the method of the present invention can be adjusted in order, merged and deleted according to actual needs. Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.
[0087] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.
[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. An image segmentation method based on deformable convolution, It is characterized in that The steps include: Step 1, using an image acquisition device to obtain an original image, and preprocessing the original image to obtain an input image; Step 2: The input image obtained after image preprocessing is used as the input of the feature extraction network. The feature extraction network extracts features of the input image layer by layer, and then outputs the image features and semantic information extracted from each layer to the corresponding layer of the upsampling network through the cross connection; Step 3: The upsampling network restores the image size layer by layer by combining the image features and semantic information of each layer to obtain the network prediction segmentation result; Step 4: training the image segmentation network according to the comparison result between the network predicted segmentation result and the corresponding true segmentation result, wherein the image segmentation network includes a feature extraction network and an upsampling network; Step 5: Based on the final segmentation result, the segmentation effect of the image segmentation network is evaluated, and the evaluation result is visualized; In step 2, the specific steps of the feature extraction network to extract features of the input image layer by layer are: Step 2.1, using a feature extraction network containing multiple deformable convolutional blocks to extract features of the input image layer by layer, and mapping the input image into a high-dimensional feature vector; Step 2.2, using repeated feature extraction and mapping operations to enrich the semantic information contained in the extracted high-dimensional feature vector; Step 2.3, projecting the high-dimensional feature vector learned by the feature extraction network to a pixel space consistent with the original image resolution for dense prediction and classification; In step 2.1, the deformable convolution block includes a deformable convolution kernel, and the deformable convolution kernel is set with learnable parameters, which are used to adjust its internal parameters for different data sets to dynamically adjust the size and shape of the deformable convolution kernel; In step 2.1, the expression for feature extraction by the deformable convolution block is: , where I Denoted as the input image, Conv is a normal convolution operation, Dconv is a deformable convolution operation, GELU is the activation function, LN represents layer normalization, and O represents the extracted features.
2. The image segmentation method based on deformable convolution according to claim 1, It is characterized in that In step 1, the specific steps of using the image acquisition device to obtain the original image are: The raw file data stream is obtained using the musculoskeletal ultrasound acquisition device; Then, the original file data stream is preprocessed to convert the original file data stream into an original image in an image format that can be processed by a computer; The specific steps of preprocessing the original image to obtain the input image are: A two-stage target detection algorithm is used to detect and crop the region of interest of the original image, remove the remaining invalid information, and extract effective information for the segmentation task through downsampling operation through hybrid pooling to obtain the input image.
3. The image segmentation method based on deformable convolution according to claim 1, It is characterized in that In step 3, the specific steps of restoring the image size layer by layer by combining the image features and semantic information of each layer by the upsampling network are: Step 3.1, the high-dimensional feature vectors of different levels obtained by the feature extraction network mapping are cropped and spliced through the cross-connection and the generated information of the corresponding level of the upsampling network to achieve feature fusion; Step 3.2, use the upsampling method to gradually restore the image size and obtain the upsampled features; In step 3.3, the feature vector extracted by the feature extraction network is upsampled to a fixed resolution to achieve end-to-end training, thereby completing pixel-level dense prediction classification and generating a segmentation mask for the corresponding image.
4. The image segmentation method based on deformable convolution according to claim 3, It is characterized in that In step 3.2, the specific steps of gradually restoring the image size using the upsampling method are: Step 3.2.1, using cropping and splicing operations to perform feature fusion on the high-dimensional feature vector extracted by the feature extraction network and the up-sampled features extracted by the up-sampling network; Step 3.2.2, use the deconvolution operation to restore the image size, repeat the feature fusion operation in step 3.2.1 multiple times until the segmentation result with the same size as the input image is obtained, the expression is: , where deconv represents the deconvolution operation, concat Represents a splicing operation, O Indicates the generation information of the current layer. I Represents the high-dimensional feature vector used by the current layer, P The output result of the current level.
5. The image segmentation method based on deformable convolution according to claim 1, It is characterized in that In step 4, the specific steps for training the image segmentation network are: Step 4.1, compare the actual segmentation result with the network predicted segmentation result, and use the loss function to quantitatively describe the gap between the network predicted segmentation result and the actual segmentation result; Step 4.2, adjust the image segmentation network parameters using the back propagation algorithm according to the loss function to optimize the network segmentation performance.
6. The image segmentation method based on deformable convolution according to claim 1, It is characterized in that In step 5, the specific steps for evaluating the segmentation effect of the image segmentation network are: Step 5.1, input the test set images into the trained image segmentation network, and output the network predicted segmentation results corresponding to each image; In step 5.2, the actual segmentation result is compared with the network predicted segmentation result, and the predicted segmentation result is evaluated using the Accuracy and IoU evaluation indicators.
7. The image segmentation method based on deformable convolution according to claim 1, It is characterized in that In step 5, the specific steps for visualizing the evaluation results are as follows: Draw the predicted segmentation mask on the original image and set the transparency of the segmentation mask to 50% to generate the final visualization result for visual display.