Ultrasonic image splitting method based on improved U-net

Abstract

The invention discloses an ultrasonic image splitting method based on improved U-net, relates to the technical field of ultrasonic images, and can efficiently process the operations of identification of ultrasonic image types (single image, double puzzles and four puzzles), ultrasonic region image contour segmentation and puzzle splitting by combining a deep learning technology and an image post-processing algorithm. Through a deep learning framework of a U-net network and in combination with the feature extraction capability of ResNet50, efficient classification and segmentation of jigsaw images are realized, and compared with a traditional image processing method, the method can process large-scale different organ tissue ultrasonic images in a short time; through accurate classification and segmentation, the contour of the ultrasonic area in the ultrasonic image can be accurately extracted, the puzzle is effectively split, the integrity of the ultrasonic image is recovered, the situation of mistaken dismantling or information loss is avoided, and the method has wide clinical application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of ultrasonic images, and in particular to an ultrasonic image disassembly method based on an improved U-net. Background Art

[0002] With the rapid development of medical imaging technology, ultrasound imaging, as a non-invasive and real-time imaging detection method, has been widely used in the diagnosis of thyroid diseases.

[0003] Ultrasound images have high resolution and can provide detailed information about thyroid tissue; however, as the size and complexity of ultrasound images increase, how to efficiently process and analyze these images has become a difficult point in medical image analysis.

[0004] In the existing ultrasound image processing methods, problems such as ultrasound region stitching, region segmentation and splitting puzzles are usually encountered, and the existing technology still has problems of low accuracy and efficiency when processing multi-puzzle images. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an ultrasound image decomposition method based on an improved U-net, comprising the following steps: S1, input ultrasound image; S2, extracting the feature map of the ultrasound image through the encoder of the U-net network; S3, input the feature map into the classification head to identify the image type of the ultrasound image, which includes single image, double puzzle and four puzzle; at the same time, input the feature map into the decoder of the U-net network to predict the contour of the ultrasound image area, separate the organ tissue of the ultrasound image from other background areas, and obtain a mask image after segmenting the ultrasound area contour; S4. When the classification head recognizes that the image type of the ultrasound image is a single image, and the mask image output by the U-net network has only one contour, execute step S5; when the classification head recognizes that the image type of the ultrasound image is a double puzzle, and the mask image output by the U-net network has only one contour, execute step S6; when the classification head recognizes that the image type of the ultrasound image is a double puzzle, and the mask image output by the U-net network has two contours, execute step S7; when the classification head recognizes that the image type of the ultrasound image is a quad puzzle, and the mask image output by the U-net network has four contours, execute step S8; S5, obtaining the outline circumscribed rectangular frame and the corresponding coordinate positions (x, y, w, h), splitting the ultrasound region of the ultrasound image according to the coordinate positions, and executing step S9; S6, obtain the outline circumscribed rectangular frame and the corresponding coordinate position (x, y, w, h), calculate the coordinate positions of the two symmetrical sub-rectangular frames obtained by splitting the outline circumscribed rectangular frame along the vertical center line, namely (x, y, w / 2, h) and (x+w / 2, y, w / 2, h), split the two symmetrical sub-rectangular frames according to the coordinate positions, and execute step S9; S7, obtaining two outline circumscribed rectangular frames and corresponding coordinate positions (x1, y1, w1, h1) and (x2, y2, w2, h2), splitting the ultrasound regions on the left and right sides of the ultrasound image according to the coordinate positions, and executing step S9; S8, obtaining four outline circumscribed rectangular frames and corresponding coordinate positions (x1, y1, w1, h1), (x2, y2, w2, h2), (x3, y3, w3, h3) and (x4, y4, w4, h4), and splitting the ultrasound region of the ultrasound image in the order of upper left, upper right, lower right and lower left according to the coordinate positions, and executing step S9; S9, complete the disassembly.

[0006] The technical solution further defined in the present invention is: Furthermore, in step S1, the ultrasound image is resized to a size of 512x512.

[0007] As described above, in the ultrasound image disassembly method based on the improved U-net, in step S2, the Resnet50 neural network is used as the backbone network of the U-net network encoder to extract features of the ultrasound image, and the feature learning process is optimized through deep residual connections.

[0008] As described above, in a method for disassembling ultrasound images based on an improved U-net, in step S3, after the ultrasound image is input into the encoder of the U-net network, the output image feature size is [1,2048,16,16], where 1 indicates that the input is 1 picture, 2048 indicates the number of channels of the feature map, and [16,16] indicates that the length and width of the feature map are both 16; the classification head includes a pooling layer, a fully connected layer, and an activation layer. After the feature map enters the classification head, it passes through the AdaptiveAvgPool2d layer, the Flatten layer, the first Linear layer, the ReLu layer, the Dropout layer, the second Linear layer, and the Softmax layer in sequence; the output of the second Linear layer is converted into a probability distribution through the Softmax layer, and the sum of all probabilities is 1, and the category with the largest probability is selected as the prediction result; 0 is set as a single image, 1 is a double puzzle, and 2 is set as a four-puzzle, where 0, 1, and 2 all represent the index values ​​of the output probability list; The decoder of the U-net network segments the ultrasonic region contour of the ultrasonic image to obtain a mask image of size 512x512, and restores the size of the mask image to the size of the original ultrasonic image.

[0009] As described above, in the ultrasound image decomposition method based on the improved U-net, the AdaptiveAvgPool2d layer pools the feature map into a 1x1 spatial dimension, that is, average pooling is performed on the 16x16 feature map of each channel, and finally a value is output for each channel.

[0010] As described above, the unpacking method of ultrasound images based on improved U-net, the Flatten layer flattens the output of the AdaptiveAvgPool2d layer into a one-dimensional vector.

[0011] As described above, in the ultrasound image decomposition method based on the improved U-net, the first Linear layer is a fully connected layer, and the first Linear layer maps the input 2048-dimensional features to 512-dimensional features, that is, the input size

[12048] is mapped to [1,512].

[0012] As described above, in the ultrasound image disassembly method based on the improved U-net, the ReLu layer is an activation function used to apply the ReLU operation to each element one by one so that the output shape is the same as the input; the ReLu operation sets negative values ​​to 0 and keeps positive values ​​unchanged.

[0013] As described above, a method for disassembling ultrasound images based on an improved U-net, the Dropout layer randomly sets 50% of the neuron outputs to 0.

[0014] As described above, in the ultrasound image decomposition method based on the improved U-net, the second Linear layer is a fully connected layer, which is used to map the 512-dimensional features to 3 dimensions, that is, the number of categories of the classification task is 3, namely single image, double puzzle and four puzzle; the 1x3 vector output by the second Linear layer is sent to the Softmax layer to convert the vector into a probability distribution.

[0015] The beneficial effects of the present invention are: (1) In the present invention, the deep learning framework of the U-net network is combined with the feature extraction capability of ResNet50 to achieve efficient classification and segmentation of puzzle images. Compared with traditional image processing methods, the present invention can process large-scale ultrasound images of different organ tissues in a short time, thus having higher efficiency than the existing technology; (2) In the present invention, through precise classification and segmentation, the contour of the ultrasonic area in the ultrasonic image can be accurately extracted, and the jigsaw puzzle can be effectively split to restore the integrity of the ultrasonic image, thereby avoiding the occurrence of erroneous removal or loss of information, thereby having higher accuracy than the existing technology; (3) The present invention can process different types of ultrasound images such as single image, double image and quadruple image, has a wide range of applications, and can meet different clinical needs, thus having higher flexibility than the existing technology; (4) The present invention can be easily integrated into the existing ultrasound image processing system to improve its automation level, reduce manual intervention, and improve diagnostic efficiency, thereby having higher scalability than the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the overall process of the present invention; Figure 2 A schematic diagram of the structure of a U-net network in an embodiment of the present invention; Figure 3 Schematic diagram of an image post-processing algorithm in an embodiment of the present invention; Figure 4 It is a structural schematic diagram of a classification head in an embodiment of the present invention; Figure 5 Schematic diagrams of the decomposition results of different types of ultrasound images in an embodiment of the present invention, wherein (a) is a schematic diagram of the decomposition results when the image type is a single image and the mask image has only one contour; (b) is a schematic diagram of the decomposition results when the image type is a double puzzle and the mask image has only one contour; (c) is a schematic diagram of the decomposition results when the image type is a double puzzle and the mask image has two contours; (d) is a schematic diagram of the decomposition results when the image type is a quad puzzle and the mask image has four contours. DETAILED DESCRIPTION

[0017] This embodiment provides an ultrasound image disassembly method based on an improved U-net, such as Figure 1 As shown, the following steps are included: Step 1: Input ultrasound images and resize them to 512x512. These ultrasound images may be single images, double images, or quad images. These different types of ultrasound images will have different effects on the subsequent segmentation process and splitting operations.

[0018] Step 2: Extract the feature map of the ultrasound image through the encoder of the U-net network; use the Resnet50 neural network as the backbone network of the U-net network encoder to extract the features of the ultrasound image. ResNet50, as the backbone network of the U-net network encoder, can effectively capture the high-level features in the ultrasound image, and optimize the feature learning process through its deep residual connection, avoiding the gradient disappearance problem.

[0019] Step 3: Input the feature map into the classification head to identify the image type of the ultrasound image, which includes single image, double puzzle and quad puzzle. At the same time, input the feature map into the decoder of the U-net network to predict the contour of the ultrasound image area, separate the organ tissue of the ultrasound image from other background areas, and obtain the mask image after segmenting the ultrasound area contour.

[0020] While the ultrasound image is being recognized, the feature map obtained by sending the ultrasound image to the U-net network encoder is sent to the decoder to predict the contour of the ultrasound image area. The decoder can effectively restore the details of the image through upsampling, skip connection and convolution operations, and generate high-quality ultrasound area segmentation images; Figure 2 As shown in the figure, the U-net network is a typical image segmentation network with an encoder and decoder structure, which can restore the detailed information of the image while extracting high-level features.

[0021] The ultrasound image is sent to the U-net network, and the segmentation network can accurately extract the boundaries of the ultrasound area. Since ultrasound images often contain complex tissue structures and noise, the U-net network can improve the segmentation accuracy by efficiently capturing contextual information and restoring details. The segmentation process can separate the organ tissues of the ultrasound image from other background areas, preparing for the subsequent segmentation of the ultrasound area (single image, double puzzle, quad puzzle) of the ultrasound image.

[0022] The decoder of the U-net network gradually restores the image resolution through the deconvolution layer to ensure that the contour of the ultrasonic area of ​​the ultrasound image is accurately segmented. During the segmentation process, the U-net network not only pays attention to the global information, but also pays attention to the preservation of local details, which is especially important for ultrasound image segmentation because the boundaries between the ultrasound images of different organ tissues and the background areas may be blurred.

[0023] The decoder of the U-net network segments the contour of the ultrasound region of the ultrasound image to obtain a mask image of size 512x512, and restores the size of the mask image to the size of the original ultrasound image, such as Figure 3 As shown, the image post-processing algorithm is used to obtain the coordinates of the circumscribed rectangular frame of the ultrasound area contour in the mask image.

[0024] A classification head is connected after the decoder of the U-net network to identify the image type of the ultrasound image. The classification head will divide the input image into three types: single image, double puzzle and quad puzzle. The significance of this classification step is that the subsequent processing flow will adopt different strategies according to the different image types. For example, double puzzle and quad puzzle may involve more complex puzzle relationships, and additional processing steps are required to accurately split the splicing area. Through this step, the puzzle type can be identified quickly and accurately, providing the necessary pre-information for subsequent image processing.

[0025] After the ultrasound image is input into the encoder of the U-net network, the output image feature size is [1,2048,16,16], where 1 means that the input is 1 image, 2048 means the number of channels of the feature map, and [16,16] means that the length and width of the feature map are both 16.

[0026] Classification header framework such as Figure 4 As shown in the figure, the classification head mainly consists of a pooling layer, a fully connected layer and an activation layer. Among them, AdaptiveAvgPool2d (1) pools the feature map into a 1x1 spatial dimension, that is, average pooling is performed on the 16x16 feature map of each channel, and finally each channel outputs a value; the Flatten layer flattens the output into a one-dimensional vector; the first Linear layer is a fully connected layer, which maps the input 2048-dimensional features to 512-dimensional features, that is, the input size

[12048] is mapped to [1,512]; ReLu is an activation function, which applies the ReLU operation to each element. ReLu sets negative values ​​to 0 and keeps positive values ​​unchanged. This operation is applied to each element one by one, so the output shape is the same as the input; Dropout (0.5) is a regularization technique, which randomly sets 50% of the neuron outputs to 0 to reduce overfitting; the second Linear layer is also a fully connected layer, which maps the 512-dimensional features to 3 dimensions, that is, the number of categories of the classification task is 3, namely single image, double puzzle and four puzzle.

[0027] Finally, the 1x3 vector output by the second Linear layer is sent to the softmax, which converts the output of the fully connected layer into a probability distribution and makes the sum of all probabilities 1. In this embodiment, 0 is set to a single image, 1 is a double puzzle, and 2 is a four-puzzle puzzle. The above numbers 0, 1, and 2 represent the index values ​​of the output probability list; for example, the output probability is [0.8, 0.15, 0.05], and the category (0) with the largest probability (0.8) is selected as the prediction result, so the image type of the ultrasound image is a single image.

[0028] Step 4: When the classification head recognizes that the image type of the ultrasound image is a single image, and the mask image output by the U-net network has only one contour, directly use Figure 3 The image post-processing algorithm shown in FIG. 1 obtains the contour circumscribed rectangular frame and the corresponding coordinate position (x, y, w, h), and splits the ultrasonic region of the ultrasonic image according to the coordinate position. The splitting result is shown in FIG. Figure 5 As shown in Figure (a).

[0029] When the classification head recognizes that the image type of the ultrasound image is a double puzzle, and the mask image output by the U-net network has only one outline, the image post-processing algorithm processes the mask image to obtain the outline circumscribed rectangular frame and the corresponding coordinate position (x, y, w, h), indicating that the ultrasound regions of the double puzzle are close together. When splitting the ultrasound region, it is necessary to calculate the coordinate positions of the two symmetrical sub-rectangular frames obtained by splitting the outline circumscribed rectangular frame with the vertical center line, namely (x, y, w / 2, h) and (x+w / 2, y, w / 2, h). The split is performed according to the coordinate positions of the two symmetrical sub-rectangular frames. The splitting result is as follows: Figure 5 As shown in Figure (b).

[0030] When the classification head recognizes that the image type of the ultrasound image is a double puzzle, and the mask image output by the U-net network has two contours, the image post-processing algorithm processes the mask image and calculates the circumscribed rectangular frame of the contours to obtain the corresponding coordinate positions (x1, y1, w1, h1) and (x2, y2, w2, h2) of the two contour circumscribed rectangular frames. The ultrasound regions on the left and right sides of the ultrasound image are directly split according to the two coordinate positions. The splitting results are shown in Figure 2. Figure 5 As shown in Figure (c).

[0031] When the classification head recognizes that the image type of the ultrasound image is a four-puzzle image, and the mask image output by the U-net network has four contours, the image post-processing algorithm processes the mask image and calculates the circumscribed rectangular frame of the contours to obtain the coordinate positions of the circumscribed rectangular frames of the four contours (x1, y1, w1, h1), (x2, y2, w2, h2), (x3, y3, w3, h3) and (x4, y4, w4, h4). The ultrasound region of the ultrasound image is split in the order of upper left, upper right, lower right and lower left according to the four coordinate positions. The splitting results are shown in Figure 2. Figure 5 As shown in Figure (d).

[0032] The method of this embodiment realizes image type recognition, segmentation and ultrasonic region splitting of ultrasound images through deep learning technology and image post-processing algorithm, which greatly improves the efficiency and accuracy of ultrasound puzzle splitting and reduces the subjectivity and workload of manual screening. It can effectively screen ultrasound puzzles and accurately split ultrasound puzzles, restore the integrity of ultrasound images, avoid accidental removal or loss of information, and has broad clinical application prospects.

[0033] Through the deep learning framework of the U-net network and the feature extraction capability of ResNet50, efficient classification and segmentation of puzzle images are achieved. Compared with traditional image processing methods, the present invention can process large-scale ultrasound images of different organ tissues in a short time, thus having higher efficiency than the existing technology; the U-net network combined with the encoder structure of ResNet50 can make full use of the advantages of ResNet50 in image feature extraction, and combine with the U-net network for image segmentation to improve the accuracy of puzzle image processing.

[0034] By introducing a classification head into the U-net network, the automatic recognition of different puzzle types (single, double, and quad puzzles) is realized, and the subsequent image processing process is optimized. At the same time, the segmentation contour is accurately corrected through the image post-processing algorithm, the puzzle joint position is accurately found, and the puzzle is split through the image post-processing algorithm to restore the integrity of the ultrasound image; through precise classification and segmentation, the contour of the ultrasound area in the ultrasound image can be accurately extracted, and the puzzle can be effectively split to restore the integrity of the ultrasound image, avoiding the occurrence of erroneous removal or loss of information, thereby having higher accuracy than the existing technology; The method of this embodiment can also process different types of ultrasound images such as single image, double image and quadruple image, and has a wide range of applications and can meet different clinical needs, thus having higher flexibility than the existing technology; the method of this embodiment can also be easily integrated into the existing ultrasound image processing system to improve its degree of automation, reduce manual intervention, and improve diagnostic efficiency, thus having higher scalability than the existing technology.

[0035] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solution formed by equivalent replacement or equivalent transformation falls within the protection scope required by the present invention.

Claims

1. A method for disassembling ultrasound images based on an improved U-net, characterized in that: The following steps are involved: S1, input ultrasound image; S2, extracting the feature map of the ultrasound image through the encoder of the U-net network; S3, input the feature map into the classification head to identify the image type of the ultrasound image, which includes single image, double puzzle and four puzzle; at the same time, input the feature map into the decoder of the U-net network to predict the contour of the ultrasound image area, separate the organ tissue of the ultrasound image from other background areas, and obtain a mask image after segmenting the ultrasound area contour; S4. When the classification head recognizes that the image type of the ultrasound image is a single image, and the mask image output by the U-net network has only one contour, execute step S5; when the classification head recognizes that the image type of the ultrasound image is a double puzzle, and the mask image output by the U-net network has only one contour, execute step S6; when the classification head recognizes that the image type of the ultrasound image is a double puzzle, and the mask image output by the U-net network has two contours, execute step S7; when the classification head recognizes that the image type of the ultrasound image is a quad puzzle, and the mask image output by the U-net network has four contours, execute step S8; S5, obtaining the outline circumscribed rectangular frame and the corresponding coordinate positions (x, y, w, h), splitting the ultrasound region of the ultrasound image according to the coordinate positions, and executing step S9; S6, obtain the outline circumscribed rectangular frame and the corresponding coordinate position (x, y, w, h), calculate the coordinate positions of the two symmetrical sub-rectangular frames obtained by splitting the outline circumscribed rectangular frame along the vertical center line, namely (x, y, w / 2, h) and (x+w / 2, y, w / 2, h), split the two symmetrical sub-rectangular frames according to the coordinate positions, and execute step S9; S7, obtaining two outline circumscribed rectangular frames and corresponding coordinate positions (x1, y1, w1, h1) and (x2, y2, w2, h2), splitting the ultrasound regions on the left and right sides of the ultrasound image according to the coordinate positions, and executing step S9; S8, obtaining four outline circumscribed rectangular frames and corresponding coordinate positions (x1, y1, w1, h1), (x2, y2, w2, h2), (x3, y3, w3, h3) and (x4, y4, w4, h4), and splitting the ultrasound region of the ultrasound image in the order of upper left, upper right, lower right and lower left according to the coordinate positions, and executing step S9; S9, complete the disassembly.

2. The method for disassembling ultrasound images based on improved U-net according to claim 1, characterized in that: In the step S1, the ultrasound image is resized to a size of 512x512.

3. The ultrasonic image decomposition method based on improved U-net according to claim 1, characterized in that: In step S2, the Resnet50 neural network is used as the backbone network of the U-net network encoder to extract features from the ultrasound image, and the feature learning process is optimized through deep residual connections.

4. The method for disassembling ultrasound images based on improved U-net according to claim 1, characterized in that: In the step S3, after the ultrasound image is input into the encoder of the U-net network, the output image feature size is [1,2048,16,16], where 1 indicates that the input is 1 picture, 2048 indicates the number of channels of the feature map, and [16,16] indicates that the length and width of the feature map are both 16; the classification head includes a pooling layer, a fully connected layer, and an activation layer. After the feature map enters the classification head, it passes through the AdaptiveAvgPool2d layer, the Flatten layer, the first Linear layer, the ReLu layer, the Dropout layer, the second Linear layer, and the Softmax layer in sequence; the output of the second Linear layer is converted into a probability distribution through the Softmax layer, and the sum of all probabilities is 1, and the category with the largest probability is selected as the prediction result; 0 is set as a single image, 1 is a double puzzle, and 2 is a four-puzzle, where 0, 1, and 2 all represent the index values ​​of the output probability list; The decoder of the U-net network segments the ultrasonic region contour of the ultrasonic image to obtain a mask image of size 512x512, and restores the size of the mask image to the size of the original ultrasonic image.

5. The method for disassembling ultrasound images based on improved U-net according to claim 4, characterized in that: The AdaptiveAvgPool2d layer pools the feature map into a 1x1 spatial dimension, that is, average pooling is performed on the 16x16 feature map of each channel, and finally each channel outputs a value.

6. The ultrasonic image decomposition method based on improved U-net according to claim 4, characterized in that: The Flatten layer flattens the output of the AdaptiveAvgPool2d layer into a one-dimensional vector.

7. The method for disassembling ultrasound images based on improved U-net according to claim 4, characterized in that: The first Linear layer is a fully connected layer. The first Linear layer maps the input 2048-dimensional features to 512-dimensional features, that is, maps the input size [12048] to [1,512].

8. The method for disassembling ultrasound images based on improved U-net according to claim 4, characterized in that: The ReLu layer is an activation function for applying the ReLU operation to each element one by one so that the output shape is the same as the input; the ReLu operation sets negative values ​​to 0 and keeps positive values ​​unchanged.

9. The method for disassembling ultrasound images based on improved U-net according to claim 4, characterized in that: The Dropout layer randomly sets 50% of the neuron outputs to 0.

10. The method for disassembling ultrasound images based on improved U-net according to claim 4, characterized in that: The second Linear layer is a fully connected layer, which is used to map the 512-dimensional features to 3 dimensions, that is, the number of categories of the classification task is 3, namely single image, double puzzle and four puzzle; the 1x3 vector output by the second Linear layer is sent to the Softmax layer to convert the vector into a probability distribution.

Images