Ultrasonic image thyroid nodule segmentation method based on edge attention

Abstract

The invention provides an ultrasonic image thyroid nodule segmentation method and device based on edge attention, and relates to the technical field of image segmentation. The ultrasound image thyroid nodule segmentation method based on edge attention comprises the following steps: acquiring a thyroid ultrasound image data set; inputting the thyroid ultrasound image data set into a pre-trained Faster R-CNN network to obtain thyroid nodule position information; according to the thyroid nodule position information, cutting the thyroid ultrasound image data set to obtain a processed thyroid ultrasound image data set; constructing an initial thyroid nodule segmentation network; according to the processed thyroid ultrasound image data set, training an initial thyroid nodule segmentation network to obtain a trained thyroid nodule segmentation network; and obtaining a to-be-detected thyroid ultrasound image and inputting the to-be-detected thyroid ultrasound image into the trained thyroid nodule segmentation network to obtain a thyroid nodule segmentation prediction result. According to the invention, the accuracy of thyroid nodule segmentation can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and in particular to a method and device for segmenting thyroid nodules in ultrasonic images based on edge attention. Background Art

[0002] Thyroid nodules refer to lumps filled with fluid or solid inside the thyroid gland. They are a common clinical disease, and their appearance often represents thyroid abnormalities. In the general population, the incidence rate is about 65%, among which the prevalence rate in women is much higher than that in men, and in recent years, the prevalence rate in women has continued to increase. Ultrasound has become the most popular and reliable method for detecting thyroid nodules in clinical practice due to its advantages of being painless, non-invasive, effective and fast. Segmenting thyroid nodules from thyroid ultrasound images is the key to diagnosing thyroid nodules or thyroid cancer. However, the most commonly used segmentation methods in clinical practice often rely on manual annotation by doctors, which requires a lot of clinical practice training and skills, and it is difficult to remove subjective factors from the diagnosis process, resulting in misdiagnosis and missed diagnosis. Therefore, computer-aided diagnosis has become the key. Using computer technology to complete nodule segmentation can assist doctors in analyzing the condition, reduce the rate of misdiagnosis and missed diagnosis, and improve diagnostic efficiency.

[0003] Currently, most thyroid nodule segmentation methods are based on deep learning methods, which perform segmentation by continuously learning the deep feature representation of images; however, most deep learning methods do not explicitly take into account the characteristics of blurred structural boundaries and texture heterogeneity in medical images. Since the target boundaries in thyroid ultrasound images are blurred and the grayscale difference inside and outside the boundaries is small, the segmentation results of previous methods often have unclear boundaries, resulting in poor segmentation accuracy. Summary of the invention

[0004] In order to solve the problem of unclear boundaries in the segmentation results of the prior art, resulting in poor segmentation accuracy, the embodiment of the present invention provides a method and device for segmenting thyroid nodules in ultrasound images based on edge attention. The technical solution is as follows:

[0005] On the one hand, a method for segmenting thyroid nodules in an ultrasound image based on edge attention is provided. The method is implemented by a device for segmenting thyroid nodules in an ultrasound image based on edge attention. The method comprises:

[0006] S1. Obtain a thyroid ultrasound image dataset;

[0007] S2, inputting the thyroid ultrasound image dataset into a pre-trained Faster R-CNN network to obtain thyroid nodule location information;

[0008] S3. According to the thyroid nodule location information, the thyroid ultrasound image dataset is cropped to obtain a processed thyroid ultrasound image dataset;

[0009] S4, constructing the initial thyroid nodule segmentation network;

[0010] S5. Training the initial thyroid nodule segmentation network according to the processed thyroid ultrasound image data set to obtain a trained thyroid nodule segmentation network;

[0011] S6. Acquire a thyroid ultrasound image to be detected; input the thyroid ultrasound image to be detected into the trained thyroid nodule segmentation network to obtain a thyroid nodule segmentation prediction result.

[0012] Optionally, the step S2 of inputting the thyroid ultrasound image dataset into a pre-trained Faster R-CNN network to obtain thyroid nodule location information includes:

[0013] S21, inputting the thyroid ultrasound image dataset into a pre-trained Faster R-CNN network, performing feature extraction through a ResNet-50 network, and obtaining a feature map;

[0014] S22, generating a candidate region of interest using an RPN network according to the feature map;

[0015] S23, using the non-maximum suppression method to eliminate candidate regions of interest with low classification scores, and obtain regions of interest with high scores;

[0016] S24. Perform classification and position regression operations on the region of interest with a high score simultaneously according to the pooling layer and the fully connected layer to obtain position information of a boundary detection box; wherein the position information of the boundary detection box is the position information of the thyroid nodule.

[0017] Optionally, the initial thyroid nodule segmentation network of S4 includes:

[0018] encoder, edge prediction module, skip connection structure module, multi-scale context fusion module and decoder; wherein,

[0019] The encoder is used to extract features from the input image using a pre-trained VGG-19 network;

[0020] The edge prediction module is used to extract the boundary perception features of the thyroid nodule area by using the edge detail information contained in the low-level features to generate an edge prediction map;

[0021] The skip connection structure module is used to combine the convolutional features of multiple network layers to learn local and full-text context information;

[0022] The multi-scale context fusion module is used to perceive target features of different scales and obtain multi-scale fusion features;

[0023] The decoder is used to decode the features of the encoder's output and output the processing results.

[0024] Optionally, the step of S5 training the initial thyroid nodule segmentation network according to the processed thyroid ultrasound image dataset to obtain a trained thyroid nodule segmentation network comprises:

[0025] S51, inputting the processed thyroid ultrasound image data set into the initial thyroid nodule segmentation network, performing feature extraction through an encoder, and obtaining a feature map of the input image;

[0026] S52, inputting the feature map of the input image into the edge prediction module to perform feature extraction to obtain boundary perception features;

[0027] S53, inputting the feature map of the input image into the skip connection structure module to extract context information, obtaining multi-scale context information, and processing it through a convolution layer to obtain fusion features;

[0028] S54, inputting the feature map of the input image into the multi-scale context fusion module to perform multi-scale feature fusion to obtain multi-scale fusion features;

[0029] S55, inputting the boundary perception feature, the fusion feature and the multi-scale fusion feature into the decoder for processing to obtain a final segmentation result.

[0030] Optionally, after the step of acquiring a thyroid ultrasound image to be detected in S6; inputting the thyroid ultrasound image to be detected into the trained thyroid nodule segmentation network to obtain a thyroid nodule segmentation prediction result, the step further includes:

[0031] According to the final segmentation result, the image segmentation evaluation index is used to evaluate and obtain the evaluation result;

[0032] Optionally, the image segmentation evaluation indicators include: Dice coefficient, duplication rate, overlap rate, F1 coefficient and accuracy rate.

[0033] On the other hand, a device for segmenting thyroid nodules in an ultrasound image based on edge attention is provided. The device is applied to a method for segmenting thyroid nodules in an ultrasound image based on edge attention. The device comprises:

[0034] A first acquisition unit is used to acquire a thyroid ultrasound image data set;

[0035] A second acquisition unit is used to input the thyroid ultrasound image dataset into a pre-trained Faster R-CNN network to obtain thyroid nodule location information;

[0036] a processing unit, configured to perform cropping processing on the thyroid ultrasound image data set according to the thyroid nodule position information to obtain a processed thyroid ultrasound image data set;

[0037] A construction unit for constructing an initial thyroid nodule segmentation network;

[0038] A training unit, configured to train the initial thyroid nodule segmentation network according to the processed thyroid ultrasound image data set to obtain a trained thyroid nodule segmentation network;

[0039] The prediction unit is used to obtain a thyroid ultrasound image to be detected; input the thyroid ultrasound image to be detected into the trained thyroid nodule segmentation network to obtain a thyroid nodule segmentation prediction result.

[0040] Optionally, the second acquiring unit is used to:

[0041] The thyroid ultrasound image dataset is input into a pre-trained Faster R-CNN network, and feature extraction is performed through a ResNet-50 network to obtain a feature map;

[0042] According to the feature map, a RPN network is used to generate a candidate region of interest;

[0043] The non-maximum suppression method is used to eliminate candidate regions of interest with low classification scores and obtain regions of interest with high scores;

[0044] According to the pooling layer and the fully connected layer, classification and position regression operations are simultaneously performed on the region of interest with a high score to obtain the position information of the boundary detection box; wherein the position information of the boundary detection box is the position information of the thyroid nodule.

[0045] Optionally, the initial thyroid nodule segmentation network includes:

[0046] encoder, edge prediction module, skip connection structure module, multi-scale context fusion module and decoder; wherein,

[0047] The encoder is used to extract features from the input image using a pre-trained VGG-19 network;

[0048] The edge prediction module is used to extract the boundary perception features of the thyroid nodule area by using the edge detail information contained in the low-level features to generate an edge prediction map;

[0049] The skip connection structure module is used to combine the convolutional features of multiple network layers to learn local and full-text context information;

[0050] The multi-scale context fusion module is used to perceive target features of different scales and obtain multi-scale fusion features;

[0051] The decoder is used to decode the features of the encoder's output and output the processing results.

[0052] Optionally, the training unit is used to:

[0053] Inputting the processed thyroid ultrasound image data set into the initial thyroid nodule segmentation network, performing feature extraction through an encoder, and obtaining a feature map of the input image;

[0054] Inputting the feature map of the input image into the edge prediction module to extract features and obtain boundary perception features;

[0055] Inputting the feature map of the input image into the skip connection structure module to extract context information, obtain multi-scale context information, and process it through a convolution layer to obtain fusion features;

[0056] Inputting the feature map of the input image into the multi-scale context fusion module to perform multi-scale feature fusion to obtain multi-scale fusion features;

[0057] The boundary perception feature, the fusion feature and the multi-scale fusion feature are input into the decoder for processing to obtain a final segmentation result.

[0058] Optionally, after the steps of acquiring a thyroid ultrasound image to be detected; inputting the thyroid ultrasound image to be detected into the trained thyroid nodule segmentation network to obtain a thyroid nodule segmentation prediction result, the method further includes:

[0059] According to the final segmentation result, the image segmentation evaluation index is used to evaluate and obtain the evaluation result;

[0060] Optionally, the image segmentation evaluation indicators include: Dice coefficient, duplication rate, overlap rate, F1 coefficient and accuracy rate.

[0061] On the other hand, a device for segmenting thyroid nodules in ultrasound images based on edge attention is provided, and the device for segmenting thyroid nodules in ultrasound images based on edge attention comprises: a processor; and a memory, wherein computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned methods for segmenting thyroid nodules in ultrasound images based on edge attention is implemented.

[0062] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned edge attention-based ultrasound image thyroid nodule segmentation methods.

[0063] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0064] According to an embodiment of the present invention, a thyroid ultrasound image dataset is first obtained; the thyroid ultrasound image dataset is input into a pre-trained Faster R-CNN network to obtain thyroid nodule location information; the thyroid ultrasound image dataset is cropped according to the thyroid nodule location information to obtain a processed thyroid ultrasound image dataset; secondly, an initial thyroid nodule segmentation network is constructed; the initial thyroid nodule segmentation network is trained according to the processed thyroid ultrasound image dataset to obtain a trained thyroid nodule segmentation network; a thyroid ultrasound image to be detected is obtained; and finally, the thyroid ultrasound image to be detected is input into the trained thyroid nodule segmentation network to obtain a thyroid nodule segmentation prediction result.

[0065] The embodiment of the present invention combines target detection with image segmentation, which enables the thyroid nodule segmentation network to better learn thyroid nodule features while reducing interference from other tissues or features in thyroid ultrasound images; considering the constraint problem of boundaries, a thyroid nodule segmentation network is constructed by combining multi-scale context information to perform thyroid nodule segmentation; the traditional skip connection structure is improved, and the convolutional features of multiple network layers are combined to learn local and global context information, and provide a global context information flow for decoding features; the embodiment of the present invention can be used to automatically segment thyroid ultrasound image nodules; compared with the prior art, the embodiment of the present invention can automatically and efficiently segment thyroid nodule areas in ultrasound images, assist doctors in diagnosing and treating related diseases, and improve diagnostic efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0067] Figure 1 is a flow chart of a method for segmenting thyroid nodules in ultrasound images based on edge attention provided by an embodiment of the present invention;

[0068] Figure 2is a structural schematic diagram of a thyroid nodule segmentation network for ultrasound images based on edge attention provided by an embodiment of the present invention;

[0069] Figure 3 It is a block diagram of a device for segmenting thyroid nodules in ultrasound images based on edge attention provided by an embodiment of the present invention;

[0070] Figure 4 It is a structural schematic diagram of an ultrasonic image thyroid nodule segmentation device based on edge attention provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0072] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0073] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0074] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0075] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0076] The embodiment of the present invention provides a method for segmenting thyroid nodules in ultrasound images based on edge attention. The method can be implemented by a device for segmenting thyroid nodules in ultrasound images based on edge attention. The device for segmenting thyroid nodules in ultrasound images based on edge attention can be a terminal or a server. Figure 1 The flowchart of the method for segmenting thyroid nodules in ultrasound images based on edge attention is shown in FIG. The processing flow of the method may include the following steps:

[0077] S1. Obtain a thyroid ultrasound image dataset.

[0078] Among them, thyroid ultrasound images can be obtained from the hospital's ultrasound department and organized to form a data set.

[0079] S2. Input the thyroid ultrasound image dataset into the pre-trained Faster R-CNN network to obtain the location information of thyroid nodules.

[0080] Among them, the Faster R-CNN network includes: RPN network, feature pyramid network FPN, pooling layer and fully connected layer.

[0081] Among them, the use of Faster R-CNN network can reduce the amount of calculation, improve the detection speed, and accurately locate the target by sharing convolution features.

[0082] Optionally, the specific implementation process of S2 may include S21-S24:

[0083] S21, input the thyroid ultrasound image dataset into the pre-trained Faster R-CNN network, perform feature extraction through the ResNet-50 network, and obtain a feature map;

[0084] S22, according to the feature map, using the RPN network to generate candidate regions of interest;

[0085] S23, using the non-maximum suppression method to eliminate candidate regions of interest with low classification scores, and obtain regions of interest with high scores;

[0086] S24. Classify and regress the regions of interest with high scores according to the pooling layer and the fully connected layer to obtain the position information of the boundary detection box; wherein the position information of the boundary detection box is the position information of the thyroid nodule.

[0087] S3. According to the location information of the thyroid nodules, the thyroid ultrasound image dataset is cropped to obtain a processed thyroid ultrasound image dataset.

[0088] S4. Construct an initial thyroid nodule segmentation network.

[0089] Among them, the encoder-decoder structure is used to construct the thyroid nodule segmentation network.

[0090] Optionally, the initial thyroid nodule segmentation network of S4 includes:

[0091] encoder, edge prediction module, skip connection structure module, multi-scale context fusion module and decoder; wherein,

[0092] Encoder, used to extract features from the input image using a pre-trained VGG-19 network;

[0093] An edge prediction module is used to extract boundary-aware features of the thyroid nodule region using edge detail information contained in low-level features and generate an edge prediction map;

[0094] Among them, the edge prediction module can input the extracted boundary-aware features into the decoder for fusion, which can overcome the inaccurate prediction of the target boundary and improve the segmentation performance.

[0095] The skip connection structure module is used to combine the convolutional features of multiple network layers to learn local and full-text context information;

[0096] Among them, the skip connection structure module can provide the decoder with a global context information flow.

[0097] Multi-scale context fusion module, used to perceive target features of different scales and obtain multi-scale fusion features;

[0098] The decoder is used to decode the features of the encoder's output and output the processing results.

[0099] S5. According to the processed thyroid ultrasound image dataset, the initial thyroid nodule segmentation network is trained to obtain a trained thyroid nodule segmentation network.

[0100] Optionally, the specific implementation process of S5 may include S51-S55:

[0101] S51, inputting the processed thyroid ultrasound image data set into the initial thyroid nodule segmentation network, performing feature extraction through an encoder, and obtaining a feature map of the input image;

[0102] In one feasible implementation, the encoder uses a pre-trained VGG-19 network, which includes five convolutional layers to extract features from the input image, so that the resolution of the processed image is reduced to 1, , , as well as ; Among them, the decoder consists of 5 stages, which improve the resolution of feature maps through upsampling and convolution operations.

[0103] S52, inputting the feature map of the input image into the edge prediction module for feature extraction to obtain boundary perception features;

[0104] In a feasible implementation, based on the output feature map of the first layer and the output feature map of the second layer of the encoder, the edge prediction module uses the edge detail information contained in the feature map to extract the boundary perception features of the thyroid nodule. The specific implementation process may include:

[0105] S521, using 1 The convolution layer processes the output feature map of the first layer of the encoder to obtain the first feature; 1 The convolution layer processes the output feature map of the second layer of the encoder to obtain the second feature, and connects the first feature and the second feature through the batch normalization function and the activation function to obtain the splicing feature; wherein the splicing feature can be expressed by the following formula (1):

[0106] (1)

[0107] in, Represents the output feature map of the first layer of the encoder; Represents the output feature map of the second layer of the encoder; represents a 3×3 convolutional layer; Represents a splicing operation; Represents the features after splicing.

[0108] S522. According to the splicing features, two independent 1×1 convolutional layers are used to calculate the combined weights to obtain a first weight calculation result and a second weight calculation result; the first weight calculation result and the second weight calculation result are spliced ​​to obtain a spliced ​​result; according to the spliced ​​result, a Softmax activation function is used to calculate to obtain a first weight matrix and a second weight matrix; wherein the first weight matrix and the second weight matrix can be expressed by the following formula (2):

[0109] (2)

[0110] in, represents a 1×1 convolutional layer; Represents the Softmax activation function; represents the first weight matrix; represents the second weight matrix.

[0111] S523, according to the output feature map of the first layer of the encoder, the output feature map of the second layer of the encoder, the first weight matrix and the second weight matrix, a gating strategy is used to calculate and obtain the fused edge feature, which can be expressed by the following formula (3):

[0112] (3)

[0113] in, Indicates the fused edge features.

[0114] S524, inputting the fused edge features into three convolution blocks for processing to obtain edge attention features, i.e., edge perception features;

[0115] Each convolution block includes: a 3×3 convolution layer, a batch normalization layer, and a ReLU activation function.

[0116] Among them, the edge-aware features include four different levels of boundary features; among them, the four different levels of boundary features are the output features after passing through different numbers of convolution blocks; among them, the edge-aware features can be fused with the corresponding regional features of different levels to guide the representation learning of the thyroid nodule area.

[0117] S53, inputting the feature map of the input image into the skip connection structure module to extract context information, obtaining multi-scale context information, and processing it through the convolution layer to obtain fusion features;

[0118] In a feasible implementation manner, the specific implementation process of the skip connection structure module may include:

[0119] S531, input the feature map of the input image into three convolutional layers respectively, map all features into the same channel space, and obtain processed features; wherein each convolutional layer includes: a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function;

[0120] S532, the processed features are sampled to the same size and spliced ​​to obtain spliced ​​features; three separable convolutions with different void rates are used to process the spliced ​​features in parallel to obtain global context information of multi-level features;

[0121] S533. According to the global context information of the multi-level features, a 1×1 convolution layer is used to process the features to obtain the final fusion features.

[0122] In a feasible implementation manner, the embodiment of the present invention uses the third layer of the encoder To the third layer of the decoder Taking the skip connection on as an example, the process of obtaining the final fusion feature can be expressed by the following formula (4) and formula (5):

[0123] (4)

[0124] (5)

[0125] in, represents a 3×3 convolutional layer; represents a 1×1 convolutional layer; Represents a 3×3 separable convolutional layer; represents upsampling; Represents a splicing operation; represents the final fusion feature; Represents an intermediate variable.

[0126] S54, inputting the feature map of the input image into a multi-scale context fusion module to perform multi-scale feature fusion to obtain a multi-scale fusion feature;

[0127] The multi-scale context fusion module includes a scale-aware fusion module; the multi-scale context fusion module can perceive target features of different scales; the encoder output features are used as input, and convolution kernels with different receptive fields are constructed using three different dilation rates of 1, 2 and 4, which can effectively expand the receptive field to collect richer multi-scale context information; the scale-aware fusion module uses a spatial attention mechanism and a self-learning method to fuse the feature maps of two adjacent scales, and dynamically adjusts the size of the receptive field, and finally outputs the final perceptual features according to two cascaded scale-aware fusion modules. The feature maps of adjacent scales are the outputs of three separable convolutions with different dilation rates, and the feature maps of adjacent scales are between each other.

[0128] In a feasible implementation manner, the specific implementation process of multi-scale context fusion includes:

[0129] S541, splicing the output feature map of the first layer of the encoder and the output feature map of the second layer to obtain a splicing result;

[0130] S542, input the splicing result into the first branch, perform convolution processing using 1×1 convolution, calculate through softmax activation function, and obtain the first attention map; perform convolution processing using two layers of 3×3 convolution, and calculate through softmax activation function to obtain the second attention map;

[0131] S543, according to the output feature map of the first layer, the output feature map of the second layer, the first attention map and the second attention map, a weighted sum method is used to process and obtain the features of the first branch output; wherein the process of obtaining the features of the first branch output can be expressed by the following formula (6):

[0132] (6)

[0133] in, Represents the output feature map of the first layer of the encoder; Represents the output feature map of the second layer of the encoder; represents the first attention map; represents the second attention map; Represents the features of the first branch output; Represents element-wise multiplication.

[0134] S544, input the concatenated result into the second branch, perform convolution processing by 1×1 convolution, map it to the range of 0 to 1 by Sigmoid activation function, and obtain the processing result; multiply the processing result by the concatenated result to obtain the features output by the second branch; wherein the features output by the second branch can be expressed by the following formula (7):

[0135] (7)

[0136] in, Represents the Sigmoid activation function; Represents the features of the second branch output.

[0137] S545, adding the features output by the first branch to the features output by the second branch to obtain a multi-scale fusion feature, which can be expressed by the following formula (8):

[0138] (8)

[0139] in, Represents multi-scale fusion features.

[0140] S55, inputting the boundary perception feature, the fusion feature and the multi-scale fusion feature into the decoder for processing to obtain the final segmentation result.

[0141] S6. Obtain a thyroid ultrasound image to be detected; input the thyroid ultrasound image to be detected into a trained thyroid nodule segmentation network to obtain a thyroid nodule segmentation prediction result.

[0142] Optionally, after the step of obtaining a thyroid ultrasound image to be detected in S6; inputting the thyroid ultrasound image to be detected into a trained thyroid nodule segmentation network to obtain a thyroid nodule segmentation prediction result, the step further includes:

[0143] According to the final segmentation result, the image segmentation evaluation index is used to evaluate and obtain the evaluation result;

[0144] Optionally, the image segmentation evaluation indicators include: Dice coefficient, duplicate detection rate, overlap rate, F1 coefficient and accuracy rate.

[0145] According to an embodiment of the present invention, a thyroid ultrasound image dataset is first obtained; the thyroid ultrasound image dataset is input into a pre-trained Faster R-CNN network to obtain thyroid nodule location information; the thyroid ultrasound image dataset is cropped according to the thyroid nodule location information to obtain a processed thyroid ultrasound image dataset; secondly, an initial thyroid nodule segmentation network is constructed; the initial thyroid nodule segmentation network is trained according to the processed thyroid ultrasound image dataset to obtain a trained thyroid nodule segmentation network; a thyroid ultrasound image to be detected is obtained; and finally, the thyroid ultrasound image to be detected is input into the trained thyroid nodule segmentation network to obtain a thyroid nodule segmentation prediction result.

[0146] The embodiment of the present invention combines target detection with image segmentation, which enables the thyroid nodule segmentation network to better learn thyroid nodule features while reducing interference from other tissues or features in thyroid ultrasound images; considering the constraint problem of boundaries, a thyroid nodule segmentation network is constructed by combining multi-scale context information to perform thyroid nodule segmentation; the traditional skip connection structure is improved, and the convolutional features of multiple network layers are combined to learn local and global context information, and provide a global context information flow for decoding features; the embodiment of the present invention can be used to automatically segment thyroid ultrasound image nodules; compared with the prior art, the embodiment of the present invention can automatically and efficiently segment thyroid nodule areas in ultrasound images, assist doctors in diagnosing and treating related diseases, and improve diagnostic efficiency.

[0147] Figure 3 1 is a block diagram of a device for segmenting thyroid nodules in ultrasound images based on edge attention according to an exemplary embodiment. The device is used in a method for segmenting thyroid nodules in ultrasound images based on edge attention. Figure 3 The device includes a first acquisition unit 310, a second acquisition unit 320, a processing unit 330, a construction unit 340, a training unit 350 and a prediction unit 360. Wherein:

[0148] A first acquisition unit 310 is used to acquire a thyroid ultrasound image dataset;

[0149] A second acquisition unit 320 is used to input the thyroid ultrasound image dataset into a pre-trained Faster R-CNN network to obtain thyroid nodule location information;

[0150] The processing unit 330 is used to perform cropping processing on the thyroid ultrasound image data set according to the thyroid nodule location information to obtain a processed thyroid ultrasound image data set;

[0151] A construction unit 340 is used to construct an initial thyroid nodule segmentation network;

[0152] A training unit 350 is used to train the initial thyroid nodule segmentation network according to the processed thyroid ultrasound image data set to obtain a trained thyroid nodule segmentation network;

[0153] The prediction unit 360 is used to obtain a thyroid ultrasound image to be detected; input the thyroid ultrasound image to be detected into the trained thyroid nodule segmentation network to obtain a thyroid nodule segmentation prediction result.

[0154] Optionally, the second acquiring unit 320 is configured to:

[0155] The thyroid ultrasound image dataset is input into a pre-trained Faster R-CNN network, and feature extraction is performed through a ResNet-50 network to obtain a feature map;

[0156] According to the feature map, a RPN network is used to generate a candidate region of interest;

[0157] The non-maximum suppression method is used to eliminate candidate regions of interest with low classification scores and obtain regions of interest with high scores;

[0158] According to the pooling layer and the fully connected layer, classification and position regression operations are simultaneously performed on the region of interest with a high score to obtain the position information of the boundary detection box; wherein the position information of the boundary detection box is the position information of the thyroid nodule.

[0159] Optionally, the initial thyroid nodule segmentation network includes:

[0160] encoder, edge prediction module, skip connection structure module, multi-scale context fusion module and decoder; wherein,

[0161] The encoder is used to extract features from the input image using a pre-trained VGG-19 network;

[0162] The edge prediction module is used to extract the boundary perception features of the thyroid nodule area by using the edge detail information contained in the low-level features to generate an edge prediction map;

[0163] The skip connection structure module is used to combine the convolutional features of multiple network layers to learn local and full-text context information;

[0164] The multi-scale context fusion module is used to perceive target features of different scales and obtain multi-scale fusion features;

[0165] The decoder is used to decode the features of the encoder's output and output the processing results.

[0166] Optionally, the training unit 350 is used to:

[0167] Inputting the processed thyroid ultrasound image data set into the initial thyroid nodule segmentation network, performing feature extraction through an encoder, and obtaining a feature map of the input image;

[0168] Inputting the feature map of the input image into the edge prediction module to extract features and obtain boundary perception features;

[0169] Inputting the feature map of the input image into the skip connection structure module to extract context information, obtain multi-scale context information, and process it through a convolution layer to obtain fusion features;

[0170] Inputting the feature map of the input image into the multi-scale context fusion module to perform multi-scale feature fusion to obtain multi-scale fusion features;

[0171] The boundary perception feature, the fusion feature and the multi-scale fusion feature are input into the decoder for processing to obtain a final segmentation result.

[0172] Optionally, after the steps of acquiring a thyroid ultrasound image to be detected; inputting the thyroid ultrasound image to be detected into the trained thyroid nodule segmentation network to obtain a thyroid nodule segmentation prediction result, the method further includes:

[0173] According to the final segmentation result, the image segmentation evaluation index is used to evaluate and obtain the evaluation result;

[0174] Optionally, the image segmentation evaluation indicators include: Dice coefficient, duplication rate, overlap rate, F1 coefficient and accuracy rate.

[0175] According to an embodiment of the present invention, a thyroid ultrasound image dataset is first obtained; the thyroid ultrasound image dataset is input into a pre-trained Faster R-CNN network to obtain thyroid nodule location information; the thyroid ultrasound image dataset is cropped according to the thyroid nodule location information to obtain a processed thyroid ultrasound image dataset; secondly, an initial thyroid nodule segmentation network is constructed; the initial thyroid nodule segmentation network is trained according to the processed thyroid ultrasound image dataset to obtain a trained thyroid nodule segmentation network; a thyroid ultrasound image to be detected is obtained; and finally, the thyroid ultrasound image to be detected is input into the trained thyroid nodule segmentation network to obtain a thyroid nodule segmentation prediction result.

[0176] The embodiment of the present invention combines target detection with image segmentation, which enables the thyroid nodule segmentation network to better learn thyroid nodule features while reducing interference from other tissues or features in thyroid ultrasound images; considering the constraint problem of boundaries, a thyroid nodule segmentation network is constructed by combining multi-scale context information to perform thyroid nodule segmentation; the traditional skip connection structure is improved, and the convolutional features of multiple network layers are combined to learn local and global context information, and provide a global context information flow for decoding features; the embodiment of the present invention can be used to automatically segment thyroid ultrasound image nodules; compared with the prior art, the embodiment of the present invention can automatically and efficiently segment thyroid nodule areas in ultrasound images, assist doctors in diagnosing and treating related diseases, and improve diagnostic efficiency.

[0177] Figure 4 is a schematic diagram of the structure of a device for segmenting thyroid nodules in ultrasound images based on edge attention provided by an embodiment of the present invention, such as Figure 4 As shown, the ultrasound image thyroid nodule segmentation device based on edge attention may include the above Figure 3 The device for segmenting thyroid nodules in ultrasound images based on edge attention is shown. Optionally, the device for segmenting thyroid nodules in ultrasound images based on edge attention 410 may include a first processor 2001 .

[0178] Optionally, the edge attention-based ultrasound image thyroid nodule segmentation device 410 may further include a memory 2002 and a transceiver 2003 .

[0179] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0180] Combine the following Figure 4 The components of the edge attention-based ultrasound image thyroid nodule segmentation device 410 are specifically introduced:

[0181] The first processor 2001 is the control center of the edge attention-based ultrasound image thyroid nodule segmentation device 410, which can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0182] Optionally, the first processor 2001 may perform various functions of the edge attention-based ultrasound image thyroid nodule segmentation device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0183] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0184] In a specific implementation, as an embodiment, the ultrasound image thyroid nodule segmentation device 410 based on edge attention may also include multiple processors, such as Figure 4 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0185] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0186] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0187] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0188] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0189] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be transmitted through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0190] It should be noted that Figure 4 The structure of the edge attention-based ultrasound image thyroid nodule segmentation device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0191] In addition, the technical effects of the edge-attention-based ultrasound image thyroid nodule segmentation device 410 can refer to the technical effects of the edge-attention-based ultrasound image thyroid nodule segmentation method described in the above method embodiment, which will not be repeated here.

[0192] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0193] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0194] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0195] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0196] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0197] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0198] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0199] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0200] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0201] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0202] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0203] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0204] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for segmenting thyroid nodules in ultrasound images based on edge attention, characterized in that: The method comprises: S1. Obtain a thyroid ultrasound image dataset; S2, inputting the thyroid ultrasound image dataset into a pre-trained Faster R-CNN network to obtain thyroid nodule location information; S3. According to the thyroid nodule location information, the thyroid ultrasound image dataset is cropped to obtain a processed thyroid ultrasound image dataset; S4, constructing the initial thyroid nodule segmentation network; S5. Training the initial thyroid nodule segmentation network according to the processed thyroid ultrasound image data set to obtain a trained thyroid nodule segmentation network; S6. Acquire a thyroid ultrasound image to be detected; input the thyroid ultrasound image to be detected into the trained thyroid nodule segmentation network to obtain a thyroid nodule segmentation prediction result.

2. The method for segmenting thyroid nodules in ultrasonic images based on edge attention according to claim 1, characterized in that: The step S2 inputs the thyroid ultrasound image dataset into a pre-trained Faster R-CNN network to obtain thyroid nodule location information, including: S21, inputting the thyroid ultrasound image dataset into a pre-trained Faster R-CNN network, performing feature extraction through a ResNet-50 network, and obtaining a feature map; S22, generating a candidate region of interest using an RPN network according to the feature map; S23, using the non-maximum suppression method to eliminate candidate regions of interest with low classification scores, and obtain regions of interest with high scores; S24. Perform classification and position regression operations on the region of interest with a high score simultaneously according to the pooling layer and the fully connected layer to obtain position information of a boundary detection box; wherein the position information of the boundary detection box is the position information of the thyroid nodule.

3. The method for segmenting thyroid nodules in ultrasonic images based on edge attention according to claim 1, characterized in that: The initial thyroid nodule segmentation network of S4 includes: encoder, edge prediction module, skip connection structure module, multi-scale context fusion module and decoder; wherein, The encoder is used to extract features from the input image using a pre-trained VGG-19 network; The edge prediction module is used to extract the boundary perception features of the thyroid nodule area by using the edge detail information contained in the low-level features to generate an edge prediction map; The skip connection structure module is used to combine the convolutional features of multiple network layers to learn local and full-text context information; The multi-scale context fusion module is used to perceive target features of different scales and obtain multi-scale fusion features; The decoder is used to decode the features of the encoder's output and output the processing results.

4. The method for segmenting thyroid nodules in ultrasonic images based on edge attention according to claim 3, characterized in that: The step S5 of training the initial thyroid nodule segmentation network according to the processed thyroid ultrasound image dataset to obtain a trained thyroid nodule segmentation network includes: S51, inputting the processed thyroid ultrasound image data set into the initial thyroid nodule segmentation network, performing feature extraction through an encoder, and obtaining a feature map of the input image; S52, inputting the feature map of the input image into the edge prediction module to perform feature extraction to obtain boundary perception features; S53, inputting the feature map of the input image into the skip connection structure module for processing, obtaining multi-scale context information features, and processing through a convolution layer to obtain fusion features; S54, inputting the feature map of the input image into the multi-scale context fusion module to perform multi-scale feature fusion to obtain multi-scale fusion features; S55, inputting the boundary perception feature, the fusion feature and the multi-scale fusion feature into the decoder for processing to obtain a final segmentation result.

5. The method for segmenting thyroid nodules in ultrasonic images based on edge attention according to claim 1, characterized in that: After the step of obtaining the thyroid ultrasound image to be detected in S6; inputting the thyroid ultrasound image to be detected into the trained thyroid nodule segmentation network to obtain the thyroid nodule segmentation prediction result, the method further includes: According to the final segmentation result, the image segmentation evaluation index is used to evaluate and obtain the evaluation result.

6. The method for segmenting thyroid nodules in ultrasonic images based on edge attention according to claim 5, characterized in that: The image segmentation evaluation indicators include: Dice coefficient, duplicate detection rate, overlap rate, F1 coefficient and accuracy.

7. A device for segmenting thyroid nodules in ultrasound images based on edge attention, wherein the device for segmenting thyroid nodules in ultrasound images based on edge attention is used to implement the method for segmenting thyroid nodules in ultrasound images based on edge attention as claimed in any one of claims 1 to 6, characterized in that: The device comprises: A first acquisition unit is used to acquire a thyroid ultrasound image data set; A second acquisition unit is used to input the thyroid ultrasound image dataset into a pre-trained Faster R-CNN network to obtain thyroid nodule location information; a processing unit, configured to perform cropping processing on the thyroid ultrasound image data set according to the thyroid nodule position information to obtain a processed thyroid ultrasound image data set; A construction unit for constructing an initial thyroid nodule segmentation network; A training unit, configured to train the initial thyroid nodule segmentation network according to the processed thyroid ultrasound image data set to obtain a trained thyroid nodule segmentation network; The prediction unit is used to obtain a thyroid ultrasound image to be detected; input the thyroid ultrasound image to be detected into the trained thyroid nodule segmentation network to obtain a thyroid nodule segmentation prediction result.

8. The method for segmenting thyroid nodules in ultrasound images based on edge attention according to claim 7, characterized in that: The second acquiring unit is used for: The thyroid ultrasound image dataset is input into a pre-trained Faster R-CNN network, and feature extraction is performed through a ResNet-50 network to obtain a feature map; According to the feature map, a RPN network is used to generate a candidate region of interest; The non-maximum suppression method is used to eliminate candidate regions of interest with low classification scores and obtain regions of interest with high scores; According to the pooling layer and the fully connected layer, classification and position regression operations are simultaneously performed on the region of interest with a high score to obtain the position information of the boundary detection box; wherein the position information of the boundary detection box is the position information of the thyroid nodule.

9. An ultrasonic image thyroid nodule segmentation device based on edge attention, characterized in that: The edge attention-based ultrasound image thyroid nodule segmentation device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 6.

Images