An improved u-net-based ultrasound image disassembly method

Abstract

The application discloses a kind of based on the improved U-net's ultrasonic image's disassembly method, it is related to ultrasonic image technical field, by combining depth learning technique and image post-processing algorithm, can efficiently handle ultrasonic image type (single figure, double jigsaw, four jigsaw) The identification of operation, ultrasonic regional map contour segmentation and jigsaw disassembly, through the depth learning framework of U-net network, in combination with the feature extraction capability of ResNet50, efficient classification and segmentation of jigsaw image are realized, compared with traditional image processing method, the present application can process large-scale different organ tissue ultrasonic image in short time;Through accurate classification and segmentation, the contour of ultrasonic region in ultrasonic image can be accurately extracted, and jigsaw is effectively disassembled, the integrity of ultrasonic image is restored, avoids the case of misremoval or loss of information, and has wide clinical application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ultrasound images, in particular to a method for disassembling ultrasound images based on an improved U-net. BACKGROUND

[0002] With the rapid development of medical imaging technology, ultrasound imaging, as a non-invasive and real-time imaging detection method, has been widely used in the diagnosis of thyroid diseases.

[0003] Ultrasound images have high resolution and can provide detailed information about thyroid tissue; however, with the increase in the size and complexity of ultrasound images, how to efficiently process and analyze these images has become a difficult point in medical image analysis.

[0004] In existing ultrasound image processing methods, problems such as ultrasound region splicing, region segmentation, and disassembling and splicing pictures are often encountered, and the existing technology has the problems of low precision and efficiency when processing multiple picture images. SUMMARY

[0005] To solve the above technical problems, the present application provides a method for disassembling ultrasound images based on an improved U-net, comprising the following steps:

[0006] S1, inputting an ultrasound image;

[0007] S2, extracting a feature map of the ultrasound image through an encoder of a U-net network;

[0008] S3, inputting the feature map into a classification head to identify the image type of the ultrasound image, the ultrasound image type including single picture, double picture, and four picture; and inputting the feature map into a decoder of the U-net network to predict the ultrasound image region contour, separate the organ tissue of the ultrasound image from other background regions, and obtain a mask image after segmenting the ultrasound region contour;

[0009] S4, when the classification head identifies that the image type of the ultrasound image is a single picture and the mask image output by the U-net network has only one contour, step S5 is performed; when the classification head identifies that the image type of the ultrasound image is a double picture and the mask image output by the U-net network has only one contour, step S6 is performed; when the classification head identifies that the image type of the ultrasound image is a double picture and the mask image output by the U-net network has two contours, step S7 is performed; and when the classification head identifies that the image type of the ultrasound image is a four picture and the mask image output by the U-net network has four contours, step S8 is performed;

[0010] S5, obtain the contour circumscribed rectangular frame and the corresponding coordinate position (x, y, w, h), split the ultrasound region of the ultrasound image according to the coordinate position, and execute step S9;

[0011] S6, obtain the contour circumscribed rectangular frame and the corresponding coordinate position (x, y, w, h), calculate the coordinate position of the two symmetric sub-rectangular frames obtained by splitting the contour circumscribed rectangular frame with the vertical center line, that is, (x, y, w / 2, h) and (x+w / 2, y, w / 2, h), split according to the coordinate position of the two symmetric sub-rectangular frames, and execute step S9;

[0012] S7, obtain two contour circumscribed rectangular frames and the corresponding coordinate positions (x1, y1, w1, h1) and (x2, y2, w2, h2), split the ultrasound regions on the left and right sides of the ultrasound image according to the coordinate positions, and execute step S9;

[0013] S8, obtain four contour circumscribed rectangular frames and the corresponding coordinate positions (x1, y1, w1, h1), (x2, y2, w2, h2), (x3, y3, w3, h3) and (x4, y4, w4, h4), split the ultrasound region of the ultrasound image in the order of top left, top right, bottom right and bottom left according to the coordinate positions, and execute step S9;

[0014] S9, complete the splitting.

[0015] The further defined technical solution of the application is:

[0016] Further, in step S1, the ultrasound image is resized to 512x512.

[0017] As described above, the method for splitting the ultrasound image based on the improved U-net, in step S2, a Resnet50 neural network is used as the backbone network of the U-net network encoder to extract features of the ultrasound image, and the feature learning process is optimized through deep residual connection.

[0018] As the improved U-net-based ultrasound image disassembly method described above, in step S3, after the ultrasound image is input to the encoder of the U-net network, the output image feature size is [1, 2048, 16, 16], wherein 1 represents that 1 picture is input, 2048 represents the channel number of the feature map, and [16, 16] represents that the length and width of the feature map are both 16; the classification head includes a pooling layer, a full connection layer, and an activation layer, and after the feature map enters the classification head, it successively passes through an AdaptiveAvgPool2d layer, a Flatten layer, a first Linear layer, a ReLu layer, a Dropout layer, a second Linear layer, and a Softmax layer; the output of the second Linear layer is converted into a probability distribution through the Softmax layer, and the sum of all probabilities is 1, and the class with the maximum probability is selected as the prediction result; 0 is set as a single picture, 1 as a double puzzle, and 2 as a four puzzle, wherein 0, 1, and 2 all represent index values of the output probability list;

[0019] The decoder of the U-net network segments the ultrasound region contour of the ultrasound image to obtain a mask image with a size of 512x512, and restores the size of the mask image to the original size of the ultrasound image.

[0020] As the improved U-net-based ultrasound image disassembly method described above, the AdaptiveAvgPool2d layer pools the feature map into a spatial dimension of 1x1, that is, the 16x16 feature map of each channel is averaged and pooled, and finally one value is output for each channel.

[0021] As the improved U-net-based ultrasound image disassembly method described above, the Flatten layer flattens the output of the AdaptiveAvgPool2d layer into a one-dimensional vector.

[0022] As the improved U-net-based ultrasound image disassembly method described above, the first Linear layer is a full connection layer, and the first Linear layer maps the input 2048-dimensional feature to a 512-dimensional feature, that is, maps the input size

[12048] to [1, 512].

[0023] As the improved U-net-based ultrasound image disassembly method described above, the ReLu layer is an activation function, which is used to apply the ReLU operation to each element one by one, so that the output shape is the same as the input; the ReLu operation sets negative values to 0 and keeps positive values unchanged.

[0024] As the improved U-net-based ultrasound image disassembly method described above, the Dropout layer randomly sets 50% of the neuron outputs to 0.

[0025] The second Linear layer is a full connection layer, which is used to map the 512-dimensional features to 3 dimensions, i.e., the number of categories of the classification task is 3, which are single image, double puzzle and four puzzle respectively; the 1x3 vector output by the second Linear layer is sent into the Softmax layer to convert the vector into a probability distribution.

[0026] The beneficial effects of the present application are:

[0027] (1) In the present application, the efficient classification and segmentation of puzzle images are realized through the deep learning framework of the U-net network combined with the feature extraction capability of ResNet50, compared with the traditional image processing method, the present application can process large-scale different organ tissue ultrasound images in a short time, thereby having higher efficiency compared with the prior art;

[0028] (2) In the present application, through accurate classification and segmentation, the outline of the ultrasound region in the ultrasound image can be accurately extracted, and the puzzle can be effectively split, the integrity of the ultrasound image is restored, and the situation of misremoval or loss of information is avoided, thereby having higher accuracy compared with the prior art;

[0029] (3) In the present application, different types of ultrasound images such as single image, double puzzle and four puzzle can be processed, the application range is wide, different clinical needs can be met, thereby having higher flexibility compared with the prior art;

[0030] (4) In the present application, it can be easily integrated into the existing ultrasound image processing system, the automation degree is improved, the manual intervention is reduced, and the diagnosis efficiency is improved, thereby having higher scalability compared with the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is the schematic diagram of the overall process of the present application;

[0032] Figure 2 is the structural schematic diagram of the U-net network in the embodiment of the present application;

[0033] Figure 3 is the schematic diagram of the image post-processing algorithm in the embodiment of the present application;

[0034] Figure 4 is the structural schematic diagram of the classification head in the embodiment of the present application;

[0035] Figure 5Figures (a), (b), (c) and (d) are schematic diagrams of the disassembly results of different types of ultrasound images in the embodiments of the present application, wherein (a) is a schematic diagram of the disassembly result when the image type is a single image and the mask image has only one contour; (b) is a schematic diagram of the disassembly result when the image type is a double puzzle and the mask image has only one contour; (c) is a schematic diagram of the disassembly result when the image type is a double puzzle and the mask image has two contours; (d) is a schematic diagram of the disassembly result when the image type is a four-puzzle and the mask image has four contours. DETAILED DESCRIPTION

[0036] The embodiment provides an ultrasound image disassembly method based on an improved U-net, as shown in the figure, comprising the following steps: Figure 1

[0037] Step 1, input the ultrasound image, and resize the ultrasound image to 512x512 size, these ultrasound images can be single image, double puzzle or four puzzle, these different types of ultrasound images will have different effects on the subsequent segmentation process and disassembly operation.

[0038] Step 2, extracting the feature map of the ultrasound image through the encoder encoder of the U-net network; using Resnet50 neural network as the backbone network of the U-net network encoder, extracting the features of the ultrasound image, ResNet50 as the backbone network of the U-net network encoder, can effectively capture the high-level features in the ultrasound image, and optimize the feature learning process through its deep residual connection, avoiding the problem of gradient disappearance.

[0039] Step 3, input the feature map into the classification head to identify the image type of the ultrasound image, the ultrasound image type includes single image, double puzzle and four puzzle; at the same time, input the feature map into the decoder of the U-net network, predict the ultrasound image region contour, separate the organ tissue of the ultrasound image from other background region, and get the mask image after segmenting the ultrasound region contour.

[0040] At the same time of identifying the image type of the ultrasound image, the feature map obtained by inputting the ultrasound image into the U-net network encoder is sent into the decoder to predict the ultrasound image region contour, and the decoder can effectively restore the details of the image through upsampling, jump connection and convolution operation, and produce high-quality ultrasound region segmentation image; as shown in the figure, the U-net network is a typical image segmentation network, which has encoder and decoder structure, and can restore the details of the image while extracting high-level features. Figure 2

[0041] ​​The ultrasound image is input into the U-net network, and the segmentation network can accurately extract the boundary of the ultrasound region; due to the fact that the ultrasound image often contains complex tissue structure and noise, the U-net network can improve the segmentation accuracy through efficient capture of context information and detail recovery; the segmentation process can separate the organ tissue of the ultrasound image from other background regions, and prepare for the subsequent splitting of the ultrasound region (single image, double puzzle, four puzzle) of the ultrasound image.

[0042] The decoder of the U-net network gradually restores the image resolution through the deconvolution layer, ensures that the ultrasound region outline of the ultrasound image is accurately segmented, and in the segmentation process, the U-net network not only focuses on global information, but also pays attention to the preservation of local details, which is particularly important for ultrasound image segmentation, because the boundary between different organ tissues and background regions of the ultrasound image may be relatively blurred.

[0043] The decoder of the U-net network segments the ultrasound region outline of the ultrasound image to obtain a mask image with a size of 512x512, restores the size of the mask image to the original ultrasound image size, as shown in Figure 3 The image post-processing algorithm is used to obtain the circumscribed rectangle coordinates of the ultrasound region outline in the mask image.

[0044] A classification head is connected after the decoder of the U-net network to identify the image type of the ultrasound image. The classification head will divide the input image into three types, single image, double puzzle and four puzzle. The significance of this classification step is that the subsequent processing flow will adopt different strategies according to the different image types, for example, double puzzle and four puzzle may involve more complex puzzle relationship, and additional processing steps are needed to accurately split the splicing region; through this step, the puzzle type can be quickly and accurately identified, providing necessary pre-information for subsequent image processing.

[0045] After the ultrasound image is input into the encoder of the U-net network, the output image feature size is [1, 2048, 16, 16], where 1 represents that 1 picture is input, 2048 represents the channel number of the feature map, and [16, 16] represents that the length and width of the feature map are both 16.

[0046] The classification head framework is as shown in Figure 4As shown, the classification head is mainly composed of a pooling layer and a fully connected layer and an activation layer, wherein AdaptiveAvgPool2d(1) pools the feature map into a spatial dimension of 1x1, that is, the 16x16 feature map of each channel is averaged and pooled, and finally one value is output for each channel; the Flatten layer flattens the output into a one-dimensional vector; the first Linear layer is a fully connected layer that maps the input 2048-dimensional features to 512-dimensional features, that is, the input size

[12048] is mapped to [1, 512]; ReLu is an activation function that applies the ReLu operation to each element, ReLu sets negative values to 0 and keeps positive values unchanged, and this operation is applied to each element one by one, so the output shape is the same as the input; Dropout(0.5) is a regularization technique that randomly sets 50% of the neuron outputs to 0, thereby reducing overfitting; the second Linear layer is also a fully connected layer that maps the 512-dimensional features to 3 dimensions, that is, the number of classes of the classification task is 3, which are single image, double puzzle and four puzzle respectively.

[0047] Finally, the 1x3 vector output by the second Linear layer is sent to the softmax, which converts the output of the fully connected layer into a probability distribution and makes the sum of all probabilities equal to 1. In this embodiment, 0 is set as single image, 1 as double puzzle, and 2 as four puzzle. The above numbers 0, 1 and 2 represent the index values of the output probability list. For example, if the output probability is [0.8, 0.15, 0.05], the class (0) with the maximum probability (0.8) is selected as the prediction result, so the image type of the ultrasound image is single image.

[0048] Step four, when the classification head identifies that the image type of the ultrasound image is single image and the mask image output by the U-net network has only one contour, the image post-processing algorithm as shown in Figure 3 is used to obtain the contour circumscribed rectangle and the corresponding coordinate position (x, y, w, h). The ultrasound region of the ultrasound image is split according to the coordinate position, and the split image result is shown in Figure 5 (a) of the figure.

[0049] When the classification head identifies that the image type of the ultrasound image is double puzzle and the mask image output by the U-net network has only one contour, the image post-processing algorithm processes the mask image to obtain the contour circumscribed rectangle and the corresponding coordinate position (x, y, w, h), which indicates that the ultrasound regions of the double puzzle are close together. When the ultrasound regions are split, the contour circumscribed rectangle needs to be calculated to split the two symmetric sub-rectangular boxes with the vertical center line to obtain the coordinate positions (x, y, w / 2, h) and (x+w / 2, y, w / 2, h). The two symmetric sub-rectangular boxes are split according to the coordinate positions, and the split image result is shown in Figure 5As shown in (b) of FIG.

[0050] When the classification head identifies that the image type of the ultrasound image is a double-puzzle image, and the mask image output by the U-net network has two contours, the image post-processing algorithm processes the mask image and calculates the circumscribed rectangular frame of the contours to obtain the coordinate positions (x1, y1, w1, h1) and (x2, y2, w2, h2) of the circumscribed rectangular frames of the two contours, and directly splits the ultrasound regions on the left and right sides of the ultrasound image according to the two coordinate positions respectively, and the splitting result is as shown in (b) of FIG. Figure 5

[0051] When the classification head identifies that the image type of the ultrasound image is a four-puzzle image, and the mask image output by the U-net network has four contours, the image post-processing algorithm processes the mask image and calculates the circumscribed rectangular frame of the contours to obtain the coordinate positions (x1, y1, w1, h1), (x2, y2, w2, h2), (x3, y3, w3, h3) and (x4, y4, w4, h4) of the circumscribed rectangular frames of the four contours, and splits the ultrasound regions of the ultrasound image in the order of top left, top right, bottom right and bottom left according to the four coordinate positions, and the splitting result is as shown in (d) of FIG. Figure 5

[0052] The method of the embodiment realizes the image type recognition, segmentation and ultrasound region splitting of the ultrasound image through the deep learning technology and the image post-processing algorithm, greatly improves the efficiency and accuracy of the ultrasound puzzle splitting, reduces the subjectivity and workload of manual screening, can effectively screen the ultrasound puzzle and accurately split the ultrasound puzzle, restores the integrity of the ultrasound image, avoids mis-splitting or missing information, and has a wide clinical application prospect.

[0053] Through the deep learning framework of the U-net network and the feature extraction capability of ResNet50, efficient classification and segmentation of the puzzle image are realized, compared with the traditional image processing method, the present application can process large-scale different organ tissue ultrasound images in a short time, so that the present application has higher efficiency compared with the prior art; the encoder structure of the U-net network combined with ResNet50 can fully utilize the advantages of ResNet50 in image feature extraction, and combine the U-net network for image segmentation, and improve the accuracy of puzzle image processing.

[0054] ​​By introducing a classification head in the U-net network, automatic recognition of different puzzle types (single image, double puzzle, four puzzle) is realized, and the subsequent image processing process is optimized. At the same time, the image post-processing algorithm is used to accurately correct the segmentation contour, accurately find the splicing position of the puzzle, and the image post-processing algorithm is used to split the puzzle, restore the integrity of the ultrasound image; Through accurate classification and segmentation, the outline of the ultrasound region in the ultrasound image can be accurately extracted, and the puzzle can be effectively split to restore the integrity of the ultrasound image, avoid the situation of misremoval or loss of information, thereby having higher accuracy compared with the prior art;

[0055] The method of the embodiment can also process different types of ultrasound images such as single image, double puzzle and four puzzle, has wide application range, can cope with different clinical needs, and has higher flexibility compared with the prior art; The method of the embodiment can also be easily integrated into the existing ultrasound image processing system, improve the automation degree, reduce manual intervention, and improve the diagnosis efficiency, thereby having higher scalability compared with the prior art.

[0056] In addition to the above embodiments, the present application can also have other implementation manners. Any technical solution formed by equivalent substitution or equivalent transformation falls within the protection scope required by the present application.

Claims

1. An improved U-net based method for ultrasound image unzipping, characterized in that: The method comprises the following steps: S1, inputting an ultrasound image; S2, extracting a feature map of the ultrasound image through an encoder of a U-net network; S3, inputting the feature map into a classification head to identify the image type of the ultrasound image, wherein the ultrasound image type comprises a single image, a double-puzzle image, and a four-puzzle image; and inputting the feature map into a decoder of the U-net network to predict an ultrasound region contour, separate an organ tissue of the ultrasound image from other background regions, and obtain a mask image after segmenting the ultrasound region contour; S4, when the classification head identifies that the image type of the ultrasound image is a single image and the mask image output by the U-net network has only one contour, performing step S5; when the classification head identifies that the image type of the ultrasound image is a double-puzzle image and the mask image output by the U-net network has only one contour, performing step S6; when the classification head identifies that the image type of the ultrasound image is a double-puzzle image and the mask image output by the U-net network has two contours, performing step S7; and when the classification head identifies that the image type of the ultrasound image is a four-puzzle image and the mask image output by the U-net network has four contours, performing step S8; S5, obtaining a contour circumscribed rectangle and a corresponding coordinate position (x, y, w, h), splitting the ultrasound region of the ultrasound image according to the coordinate position, and performing step S9; S6, obtaining a contour circumscribed rectangle and a corresponding coordinate position (x, y, w, h), calculating the coordinate position of two symmetrical sub-rectangular frames obtained by splitting the contour circumscribed rectangle along a vertical center line, i.e., (x, y, w / 2, h) and (x+w / 2, y, w / 2, h), and splitting the ultrasound image according to the coordinate position of the two symmetrical sub-rectangular frames, and performing step S9; S7, obtaining two contour circumscribed rectangles and corresponding coordinate positions (x1, y1, w1, h1) and (x2, y2, w2, h2), splitting the ultrasound regions on the left and right sides of the ultrasound image according to the coordinate positions, and performing step S9; S8, obtaining four contour circumscribed rectangles and corresponding coordinate positions (x1, y1, w1, h1), (x2, y2, w2, h2), (x3, y3, w3, h3), and (x4, y4, w4, h4), and splitting the ultrasound regions of the ultrasound image in the order of top left, top right, bottom right, and bottom left according to the coordinate positions, and performing step S9; S9, completing the splitting of the image.

2. The method of claim 1, wherein: In the step S1, the ultrasound image is resized to 512x512.

3. The method of claim 1, wherein: In the step S2, a Resnet50 neural network is used as a backbone network of the encoder of the U-net network to extract features of the ultrasound image, and a deep residual connection is used to optimize the feature learning process.

4. The method of claim 1, wherein: In the step S3, after the ultrasound image is input to the encoder of the U-net network, the output image feature size is [1, 2048, 16, 16], wherein 1 represents that one picture is input, 2048 represents the channel number of the feature map, and [16, 16] represents that the length and width of the feature map are both 16; the classification head includes a pooling layer, a full connection layer and an activation layer, and after the feature map enters the classification head, sequentially passes through an AdaptiveAvgPool2d layer, a Flatten layer, a first Linear layer, a ReLu layer, a Dropout layer, a second Linear layer and a Softmax layer; the output of the second Linear layer is converted into a probability distribution through the Softmax layer, and the sum of all probabilities is 1, and the class with the maximum probability is selected as the prediction result; 0 is set as a single picture, 1 is set as a double picture, and 2 is set as a four picture, wherein 0, 1 and 2 all represent index values of the output probability list; The decoder of the U-net network performs segmentation on the ultrasound region contour of the ultrasound image to obtain a mask image with a size of 512x512, and restores the size of the mask image to the original size of the ultrasound image.

5. The method of claim 4, wherein: The AdaptiveAvgPool2d layer pools the feature map into a spatial dimension of 1x1, that is, the 16x16 feature map of each channel is averaged and pooled, and finally one value is output for each channel.

6. The method of claim 4, wherein: The Flatten layer flattens the output of the AdaptiveAvgPool2d layer into a one-dimensional vector.

7. The method of claim 4, wherein: The first Linear layer is a full connection layer, and the first Linear layer maps the input 2048-dimensional feature to a 512-dimensional feature, that is, maps the input size [12048] to [1, 512].

8. The method of claim 4, wherein: The ReLu layer is an activation function, which is used to apply the ReLU operation to each element one by one, so that the output shape is the same as the input; the ReLu operation sets the negative value to 0 and keeps the positive value unchanged.

9. The method of claim 4, wherein: The Dropout layer randomly sets 50% of the neuron outputs to 0.

10. The method of claim 4, wherein: The second Linear layer is a full connection layer, which is used to map the 512-dimensional feature to 3 dimensions, that is, the class number of the classification task is 3, which are a single picture, a double picture and a four picture respectively; the 1x3 vector output by the second Linear layer is input into the Softmax layer to convert the vector into a probability distribution.

Images