A regularized residual learning method for thyroid ultrasound nodule segmentation

Abstract

This paper discloses a method for thyroid ultrasound nodule segmentation using regularized residual learning. The method utilizes a semi-supervised learning strategy based on a coding-decoding network structure, regularized residual learning, and a segmentation strategy for thyroid ultrasound nodules. The method can effectively generate segmentation results for thyroid ultrasound nodules. Due to the use of a semi-supervised learning strategy, the method can achieve thyroid ultrasound nodule segmentation with a small amount of label learning, providing an effective solution for the diagnosis of thyroid diseases in clinical medicine.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology and relates to the problem of ultrasound image lesion target segmentation in medical auxiliary diagnosis. It is a method that uses the powerful function of residual network, utilizes the regularized learning strategy of artificial intelligence, adopts the latest cutting-edge residual coding technology, establishes an effective network structure, and realizes the segmentation of ultrasound image targets. Background Art

[0002] In recent years, artificial intelligence technology has rapidly advanced. With the rapid development of deep learning, complex multi-layer perceptron networks have achieved more satisfactory medical image segmentation performance, including liver tumor segmentation, brain tumor segmentation, and lung nodule segmentation. To obtain stable features from multi-layer perceptron learning, early segmentation algorithms were primarily based on supervised learning, such as U-Net, fully convolutional networks, and residual convolutional neural networks. While these methods achieved good performance, they required experienced experts to label the sampled data, which posed a challenge for ultrasound nodule segmentation.

[0003] How to establish an effective deep learning network coding model and make full use of the semi-supervised learning mechanism to achieve ultrasound image nodule segmentation in the case of few labels is a practical problem that needs to be solved urgently. Summary of the Invention

[0004] The purpose of this invention is to propose a thyroid ultrasound image nodule segmentation method based on regularized residual learning. This method is characterized by employing a semi-supervised learning strategy based on a coding-decoding network structure, regularized residual learning, and thyroid ultrasound image nodule segmentation. This method can effectively predict the segmentation results of thyroid ultrasound image nodules. Due to the use of a semi-supervised strategy, this method can achieve thyroid ultrasound image nodule segmentation with a small amount of label learning, providing an effective solution for the diagnosis of thyroid diseases in clinical medicine.

[0005] The specific technical solution for achieving the purpose of the present invention is:

[0006] A regularized residual learning method for thyroid ultrasound image nodule segmentation is disclosed. The method is characterized in that the method inputs a thyroid ultrasound image sequence with an image resolution of M×N, 100≤M≤1000, 100≤N≤1000, and the thyroid ultrasound image nodule segmentation includes the following steps:

[0007] Step 1: Build the dataset

[0008] Sample a sequences of thyroid ultrasound images to construct a sampled data set, where the number of frames in each sequence is b, and 10 ≤ a ≤ 50, 40 ≤ b ≤ 80. Randomly select c frames of images from the b frames of images in each sequence for annotation, and give the labels of thyroid nodules. Use the selected c frames of images and labels to construct an annotated data set C, where 20 ≤ c ≤ 30. Then use the remaining images in the sampled data set to construct an unlabeled data set D. Randomly select n sequence images from the annotated data set C, 3 ≤ n ≤ 10. Randomly select 75% of the images from each of the n sequence images. Use the selected 75% of the images and the corresponding labels to construct a total training set. Use the images and labels remaining after removing the data of the total training set from the annotated data set C to construct a test data set G. Further randomly select m sequence data and the corresponding labels from the total training set for constructing a training set E, where 2 ≤ m < n. Then use the remaining images and the corresponding labels in the total training set to construct a training set F. Further randomly select 10% of the images and the corresponding label data from the test data set G and supplement them to the training set E and the training set F respectively;

[0009] Step 2: Construct a neural network

[0010] The resolution of the images input to the neural network is p × q, where p is the width and q is the height, in pixels, and 100 ≤ p ≤ 1000, 100 ≤ q ≤ 1000;

[0011] (1) Basic segmentation network A

[0012] Take the tensor G as the input, with a scale of α × o × p × 1, and the tensor g as the output, with a scale of α × o × p × 1, where α is the number of batches;[[ID=,13]]

[0013] The basic segmentation network A consists of an encoder and a decoder. For the tensor G, after successive encoding and decoding processes, the output tensor g is obtained;

[0014] The encoder consists of 28 units. The first unit is a convolution unit, which contains 16 convolution kernels, all of which are 3×3 in shape, and the horizontal and vertical strides of the convolution are 2. The structures of the second to 13th units are the same, including activation processing, 1 convolution operation, 12 convolution kernels, all of which are 3×3 in shape, and the horizontal and vertical strides of the convolution are 1. The 14th unit is a convolution operation, including activation processing, 1 convolution operation, 160 convolution kernels, all of which are 1×1 in shape, and a spatial pooling process with a pooling scale of 2×2. The structures of the 15th to 26th units are the same, including activation processing, 1 convolution operation, 12 convolution kernels, all of which are 3×3 in shape. The 27th unit is a batch normalization process. The 28th unit is a convolution unit, which contains 304 convolution kernels, all of which are 1×1 in shape, and the horizontal and vertical strides of the convolution are 1.

[0015] The decoder consists of three units. The first unit is a deconvolution unit, which contains 128 convolution kernels. The shape of the convolution kernel is 4×4, and the horizontal and vertical strides of the convolution are both 2. The second unit is a deconvolution unit, which contains 64 convolution kernels. The shape of the convolution kernel is 4×4, and the horizontal and vertical strides of the convolution are both 2. The third unit is a convolution unit, which contains 2 convolution kernels. The shape of the convolution kernel is 1×1, and the horizontal and vertical strides of the convolution are both 1.

[0016] (2) Residual Learning Network B

[0017] Tensors G and g are used as input, with scales of α×o×p×1, and tensor h is used as output, with scales of α×o×p×1, where α is the batch size;

[0018] The residual learning network B consists of 3 layers of residual encoding units and 3 layers of residual decoding units;

[0019] For the tensor G, after one convolution operation, specifically, including three convolution kernels, the shape of the convolution kernel is 1×1, and the step size of the horizontal and vertical directions of the convolution is 1. The obtained result is concatenated with the tensor g according to the grayscale channel to obtain the tensor I1; the tensor I1 is sequentially processed by the following three layers of residual coding unit and three layers of residual decoding unit:

[0020] The first-layer residual coding unit is processed as a convolution operation, which includes 64 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical strides of the convolution are both 1. After activation processing, another convolution operation is performed, which includes 64 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical strides of the convolution are both 1, and the tensor I2 is obtained; then the tensor I1 is convolved once, which includes 64 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical strides of the convolution are both 1, batch normalization processing, and activation processing to obtain tensor I3; tensors I2 and I3 are added to obtain tensor I4; tensor I4 is input into the following second-layer residual coding unit processing;

[0021] When the second-layer residual coding unit is processed, the tensor I4 is batch normalized and activated. After that, a convolution operation is performed, including 128 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical steps of the convolution are both 2. Then normalization and activation are performed. After that, a convolution operation is performed, including 128 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical steps of the convolution are both 1, to obtain tensor I5; then a convolution operation is performed on tensor I4, including 128 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical steps of the convolution are both 2, and then batch normalization and activation are performed to obtain tensor I6; tensors I5 and I6 are added to obtain tensor I7, and tensor I7 is input into the following third-layer residual coding unit processing;

[0022] When processing the third-layer residual coding unit, I7 is batch normalized and activated. After that, a convolution operation is performed, which includes 256 convolution kernels. The shape of the convolution kernels is 3×3, and the horizontal and vertical steps of the convolution are 2. Then normalization and activation are performed again, and then a convolution operation is performed again, which includes 256 convolution kernels. The shape of the convolution kernels is 3×3, and the horizontal and vertical steps of the convolution are 1, to obtain tensor I8; ​​then tensor I7 is convolved once, which includes 256 convolution kernels. The shape of the convolution kernels is 3×3, and the horizontal and vertical steps of the convolution are 2. Then normalization and activation are performed again to obtain tensor I9, and tensor I8 and tensor I9 are further added to obtain tensor I10;

[0023] The tensor I10 is batch normalized and activated, and then a convolution operation is performed, which includes 512 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical step sizes of the convolution are both 1. Then it is normalized and activated again, and then a convolution operation is performed again, which includes 512 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical step sizes of the convolution are both 1. The tensor I11 is then processed by the first layer residual decoding unit as follows:

[0024] When processing the first-layer residual decoding unit, the tensors I10 and I11 are concatenated according to the third channel to obtain the tensor I12. The tensor I12 is batch normalized and activated, and then a convolution operation is performed, which includes 256 convolution kernels. The shape of the convolution kernels is 3×3, and the horizontal and vertical step sizes of the convolution are 1. Then the normalization and activation processing are performed again, and a convolution operation is performed, which includes 512 convolution kernels. The shape of the convolution kernels is 3×3. The horizontal and vertical step sizes of the convolution are 1. The convolution operation is 3×3, and the horizontal and vertical strides of the convolution are both 1, resulting in a tensor I13. The tensor I12 is then convolved once, containing 256 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical strides of the convolution are both 1. Then it is normalized and activated to obtain a tensor I14. The tensors I13 and I14 are added together to obtain a tensor I15. The tensor I15 is then processed by the second-layer residual decoding unit as follows:

[0025] When processing the second-layer residual decoding unit, the result of upsampling I15 is concatenated with tensor I7 to obtain tensor I16, and the result is batch normalized and activated before convolution operation, which includes 128 convolution kernels, the shape of which is 3×3, and the horizontal and vertical step sizes of the convolution are both 1. The result is batch normalized and activated before convolution operation, which includes 128 convolution kernels, the shape of which is 3×3, and the horizontal and vertical step sizes of the convolution are both 1. The horizontal and vertical strides of the product are both 1, resulting in a tensor I17. Tensor I16 is then convolved, containing 128 convolution kernels, each of which has a shape of 3×3 and a horizontal and vertical stride of 1. The result is batch normalized and activated before being convolved to obtain a tensor I18. Tensor I17 and I18 are added together to obtain a tensor I19. Tensor I19 is then processed by the third-layer residual decoding unit as follows:

[0026] When processing the third-layer residual decoding unit, the result of upsampling I19 is concatenated with the tensor I4 to obtain the tensor I20. The result is batch normalized and activated, and then a convolution operation is performed, which includes 64 convolution kernels. The shape of the convolution kernel is 3×3, and the step size of the horizontal and vertical directions of the convolution is 1. The result is batch normalized and activated, and then a convolution operation is performed, which includes 64 convolution kernels. The shape of the convolution kernel is 3×3, and the step size of the horizontal and vertical directions of the convolution is 1, to obtain the tensor I21. Then the tensor I20 is convolved. Convolution operation, including 64 convolution kernels, the shape of the convolution kernels are all 3×3, the horizontal and vertical strides of the convolution are both 1, the result is convolution operation, including 64 convolution kernels, the shape of the convolution kernels are all 3×3, the horizontal and vertical strides of the convolution are both 1, the result is batch normalized and activated to obtain tensor I22; tensor I21 is added to tensor I22, and the obtained tensor is convolved, including 2 convolution kernels, the shape of the convolution kernels are all 3×3, the horizontal and vertical strides of the convolution are both 1, to obtain tensor h;

[0027] Step 3: Training the Neural Network

[0028] During neural network training, the cross entropy loss for each batch is calculated using the segmentation results output by the network and the labels of the network input data. Specifically:

[0029]

[0030] Among them, i represents the i-th pixel, S i represents the i-th pixel label, t i represents the probability that the i-th pixel in the network output tensor belongs to the thyroid nodule area, and W represents the number of pixels;

[0031] 1) Train the basic segmentation network A 20,000 times

[0032] Get the training data from the training set E, uniformly scale it to a resolution of p×o, and input it to the network A for training. Iteratively optimize and continuously modify the model parameters of the network A to minimize the loss of each batch. Based on the output of the network A and the label of the input data of the network A, use formula (1) to calculate the cross entropy loss of each batch.

[0033] 2) Train the residual learning network B 20,000 times

[0034] Initialize the network using the parameters of network A obtained in step 1), obtain training data from the training set F, input it into network A, obtain tensor g, then input tensor G and tensor g into network B, iterate and optimize, and minimize the loss of each batch by continuously modifying the model parameters of network B. Calculate the cross entropy loss of each batch according to formula (1) based on the output of network B and the label of the input data of network A;

[0035] 3) Jointly train the basic segmentation network A and network B 20,000 times

[0036] Initialize the network using the parameters of network A and network B trained in step 1) and step 2), take a batch of data, wherein the first half of the batch of supervised data of the batch of data is obtained from the training set F, and the second half of the batch of unsupervised data of the batch of data is obtained from the unlabeled data set D, and then concatenate the first half of the batch of supervised data and the second half of the batch of unsupervised data according to the grayscale channel, input the resulting tensor into network A, and then divide the tensor g output by network A into the supervised data output result of network A and the unsupervised data output result of network A according to the grayscale channel, and then input the tensor G and the tensor g into network B, and then divide the tensor h output by network B into the supervised data output result of network B and the unsupervised data output result of network B according to the grayscale channel;

[0037] The training process is iteratively optimized, and the model parameters of network A and network B are continuously modified to minimize the regularized residual loss Z of each batch. The regularized residual loss Z is calculated using formula (2):

[0038] Z=L+U+V (2)

[0039] Wherein, L is the cross entropy loss calculated by formula (1) based on the supervised data output results of network A and the labels of the supervised data, U is the cross entropy loss calculated by formula (1) based on the supervised data output results of network B and the labels of the supervised data, and V is the regularized residual consistency loss calculated by structural similarity based on the unsupervised data output results of network A and the unsupervised data output results of network B;

[0040] Step 4: Segmentation of thyroid ultrasound image nodules

[0041] Take out a batch of image data of α from the thyroid ultrasound image acquisition data, and scale the image resolution to p×q, where p is the width and q is the height, in pixels, 100≤p≤1000, 100≤q≤1000. Construct a tensor G as the input of network A, use the model parameters obtained in training step 3) to initialize model A, use model A to perform prediction, and obtain tensor g, which is the thyroid ultrasound image nodule region segmentation result.

[0042] Beneficial effects of the present invention:

[0043] The present invention designs a deep learning model based on a residual network, establishes a neural network, learns contextual features in medical images, and fully utilizes the mechanism of deep learning to realize nodule segmentation in thyroid ultrasound images. The present invention can effectively obtain nodule segmentation information in thyroid ultrasound images, which helps to realize the segmentation function of thyroid lesion areas in clinical diagnosis. It can provide an effective solution for lesion target segmentation for artificial intelligence medical-assisted diagnosis, and improve the efficiency of artificial intelligence-assisted medical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is the result diagram of thyroid ultrasound image nodule segmentation in the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] Example

[0047] This example was implemented on a PC running Windows 10 64-bit. The hardware configuration included an i7-9700F CPU, 16GB of RAM, and an NVIDIA GeForce GTX 2060 8GB GPU. Tensorflow 1.14 was used as the deep learning library, and Python 3.7 was used for programming. Ultrasound imaging was acquired using a Mindray Resona 7EXP color Doppler duplex scanner and a broadband 114-5MHz linear array transducer. Ultrasound parameters were preset to center TGC and a 135dB dynamic range, and these parameters remained constant throughout the acquisition process.

[0048] In this embodiment, a thyroid ultrasound image sequence is input, and the image resolution is M×N, where M is 224 and N is 224. The segmentation of thyroid ultrasound image nodules includes the following steps:

[0049] Step 1: Build the dataset

[0050] Sampling a sequence of thyroid ultrasound images to construct a sampling data set, where the number of frames in each sequence is b, and a is 11, b is 50, randomly selecting c frames of images from the b frames of each sequence for annotation, giving the label of the thyroid nodule, using the selected c frames of images and labels to construct a labeled data set C, where c is 20, and then using the remaining images in the sampling data set to construct an unlabeled data set D, randomly selecting n sequence images from the labeled data set C, n is 3, randomly selecting 75% of the images from each sequence of the n sequence images, using the selected 75% images and corresponding labels to construct a total training set, using the remaining images and labels after removing the total training set data from the labeled data set C to construct a test data set G, further randomly selecting m sequence data and corresponding labels from the total training set to construct a training set E, where m is 2, and then using the remaining images and corresponding labels in the total training set to construct a training set F, and further randomly selecting 10% of the images and corresponding label data from the test data set G to supplement the training sets E and training set F respectively;

[0051] Step 2: Build the Neural Network

[0052] The resolution of the image or video input to the neural network is p×q, where p is the width and q is the height, both in pixels.

[0053] (1) Basic segmentation network A

[0054] Tensor G is used as input, with a scale of α×o×p×1, and tensor g is used as output, with a scale of α×o×p×1, where α is the batch size and α is 4;

[0055] The basic segmentation network A consists of an encoder and a decoder. For the tensor G, after encoding and decoding, the output tensor g is obtained;

[0056] The encoder consists of 28 units. The first unit is a convolution unit, which contains 16 convolution kernels, all of which are 3×3 in shape, and the horizontal and vertical strides of the convolution are 2. The structures of the second to 13th units are the same, including activation processing, 1 convolution operation, 12 convolution kernels, all of which are 3×3 in shape, and the horizontal and vertical strides of the convolution are 1. The 14th unit is a convolution operation, including activation processing, 1 convolution operation, 160 convolution kernels, all of which are 1×1 in shape, and a spatial pooling process with a pooling scale of 2×2. The structures of the 15th to 26th units are the same, including activation processing, 1 convolution operation, 12 convolution kernels, all of which are 3×3 in shape. The 27th unit is a batch normalization process. The 28th unit is a convolution unit, which contains 304 convolution kernels, all of which are 1×1 in shape, and the horizontal and vertical strides of the convolution are 1.

[0057] The decoder consists of three units. The first unit is a deconvolution unit, which contains 128 convolution kernels. The shape of the convolution kernel is 4×4, and the horizontal and vertical strides of the convolution are both 2. The second unit is a deconvolution unit, which contains 64 convolution kernels. The shape of the convolution kernel is 4×4, and the horizontal and vertical strides of the convolution are both 2. The third unit is a convolution unit, which contains 2 convolution kernels. The shape of the convolution kernel is 1×1, and the horizontal and vertical strides of the convolution are both 1.

[0058] (2) Residual Learning Network B

[0059] Tensor G and tensor g are used as input, with scales of α×o×p×1, and tensor h is used as output, with scales of α×o×p×1, where α is the batch size and α is 4;

[0060] The residual learning network B consists of 3 layers of residual encoding units and 3 layers of residual decoding units;

[0061] For the tensor G, after one convolution operation, specifically, including three convolution kernels, the shape of the convolution kernel is 1×1, and the step size of the horizontal and vertical directions of the convolution is 1. The obtained result is concatenated with the tensor g according to the grayscale channel to obtain the tensor I1; the tensor I1 is sequentially processed by the following three layers of residual coding unit and three layers of residual decoding unit:

[0062] The first-layer residual coding unit is processed as a convolution operation, which includes 64 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical strides of the convolution are both 1. After activation processing, another convolution operation is performed, which includes 64 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical strides of the convolution are both 1, and the tensor I2 is obtained; then the tensor I1 is convolved once, which includes 64 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical strides of the convolution are both 1, batch normalization processing, and activation processing to obtain tensor I3; tensors I2 and I3 are added to obtain tensor I4; tensor I4 is input into the following second-layer residual coding unit processing;

[0063] When the second-layer residual coding unit is processed, the tensor I4 is batch normalized and activated. After that, a convolution operation is performed, including 128 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical steps of the convolution are both 2. Then normalization and activation are performed. After that, a convolution operation is performed, including 128 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical steps of the convolution are both 1, to obtain tensor I5; then a convolution operation is performed on tensor I4, including 128 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical steps of the convolution are both 2, and then batch normalization and activation are performed to obtain tensor I6; tensors I5 and I6 are added to obtain tensor I7, and tensor I7 is input into the following third-layer residual coding unit processing;

[0064] When processing the third-layer residual coding unit, I7 is batch normalized and activated. After that, a convolution operation is performed, which includes 256 convolution kernels. The shape of the convolution kernels is 3×3, and the horizontal and vertical steps of the convolution are 2. Then normalization and activation are performed again, and then a convolution operation is performed again, which includes 256 convolution kernels. The shape of the convolution kernels is 3×3, and the horizontal and vertical steps of the convolution are 1, to obtain tensor I8; ​​then tensor I7 is convolved once, which includes 256 convolution kernels. The shape of the convolution kernels is 3×3, and the horizontal and vertical steps of the convolution are 2. Then normalization and activation are performed again to obtain tensor I9, and tensor I8 and tensor I9 are further added to obtain tensor I10;

[0065] The tensor I10 is batch normalized and activated, and then a convolution operation is performed, which includes 512 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical step sizes of the convolution are both 1. Then it is normalized and activated again, and then a convolution operation is performed again, which includes 512 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical step sizes of the convolution are both 1. The tensor I11 is then processed by the first layer residual decoding unit as follows:

[0066] When processing the first-layer residual decoding unit, the tensors I10 and I11 are concatenated according to the third channel to obtain the tensor I12. The tensor I12 is batch normalized and activated, and then a convolution operation is performed, which includes 256 convolution kernels. The shape of the convolution kernels is 3×3, and the horizontal and vertical step sizes of the convolution are 1. Then the normalization and activation processing are performed again, and a convolution operation is performed, which includes 512 convolution kernels. The shape of the convolution kernels is 3×3. The horizontal and vertical step sizes of the convolution are 1. The convolution operation is 3×3, and the horizontal and vertical strides of the convolution are both 1, resulting in a tensor I13. The tensor I12 is then convolved once, containing 256 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical strides of the convolution are both 1. Then it is normalized and activated to obtain a tensor I14. The tensors I13 and I14 are added together to obtain a tensor I15. The tensor I15 is then processed by the second-layer residual decoding unit as follows:

[0067] When processing the second-layer residual decoding unit, the result of upsampling I15 is concatenated with tensor I7 to obtain tensor I16, and the result is batch normalized and activated before convolution operation, which includes 128 convolution kernels, the shape of which is 3×3, and the horizontal and vertical step sizes of the convolution are both 1. The result is batch normalized and activated before convolution operation, which includes 128 convolution kernels, the shape of which is 3×3, and the horizontal and vertical step sizes of the convolution are both 1. The horizontal and vertical strides of the product are both 1, resulting in a tensor I17. Tensor I16 is then convolved, containing 128 convolution kernels, each of which has a shape of 3×3 and a horizontal and vertical stride of 1. The result is batch normalized and activated before being convolved to obtain a tensor I18. Tensor I17 and I18 are added together to obtain a tensor I19. Tensor I19 is then processed by the third-layer residual decoding unit as follows:

[0068] When processing the third-layer residual decoding unit, the result of upsampling I19 is concatenated with the tensor I4 to obtain the tensor I20. The result is batch normalized and activated, and then a convolution operation is performed, which includes 64 convolution kernels. The shape of the convolution kernel is 3×3, and the step size of the horizontal and vertical directions of the convolution is 1. The result is batch normalized and activated, and then a convolution operation is performed, which includes 64 convolution kernels. The shape of the convolution kernel is 3×3, and the step size of the horizontal and vertical directions of the convolution is 1, to obtain the tensor I21. Then the tensor I20 is convolved. Convolution operation, including 64 convolution kernels, the shape of the convolution kernels are all 3×3, the horizontal and vertical strides of the convolution are both 1, the result is convolution operation, including 64 convolution kernels, the shape of the convolution kernels are all 3×3, the horizontal and vertical strides of the convolution are both 1, the result is batch normalized and activated to obtain tensor I22; tensor I21 is added to tensor I22, and the obtained tensor is convolved, including 2 convolution kernels, the shape of the convolution kernels are all 3×3, the horizontal and vertical strides of the convolution are both 1, to obtain tensor h;

[0069] Step 3: Training the Neural Network

[0070] During neural network training, the cross entropy loss for each batch is calculated using the segmentation results output by the network and the labels of the network input data. Specifically:

[0071]

[0072] Among them, i represents the i-th pixel, S i represents the i-th pixel label, t i represents the probability that the i-th pixel in the network output tensor belongs to the thyroid nodule area, W represents the number of pixels, and W is calculated as 224×224;

[0073] 1) Train the basic segmentation network A 20,000 times

[0074] Get the training data from the training set E, uniformly scale it to a resolution of 224×224, and input it to the network A for training. Iterative optimization is performed by continuously modifying the model parameters of the network A to minimize the loss of each batch. Based on the output of the network A and the label of the input data of the network A, the cross entropy loss of each batch is calculated using formula (1).

[0075] 2) Train the residual learning network B 20,000 times

[0076] Initialize the network using the parameters of network A obtained in step 1), obtain training data from the training set F, uniformly scale it to a resolution of 224×224, input it to network A, obtain tensor g, and then input tensor G and tensor g to network B. Iterate and optimize, and minimize the loss of each batch by continuously modifying the model parameters of network B. Calculate the cross entropy loss of each batch according to formula (1) based on the output of network B and the label of the input data of network A.

[0077] 3) Jointly train the basic segmentation network A and network B 20,000 times

[0078] Initialize the network using the parameters of network A and network B trained in step 1) and step 2), take a batch of data, wherein the first half of the batch of supervised data of the batch of data is obtained from the training set F, and the second half of the batch of unsupervised data of the batch of data is obtained from the unlabeled data set D, then concatenate the first half of the batch of supervised data and the second half of the batch of unsupervised data according to the grayscale channel, uniformly scale them to a resolution of 224×224, input the resulting tensor into network A, and then divide the tensor g output by network A into the supervised data output result of network A and the unsupervised data output result of network A according to the grayscale channel, then input the tensor G and the tensor g into network B, and then divide the tensor h output by network B into the supervised data output result of network B and the unsupervised data output result of network B according to the grayscale channel;

[0079] The training process is iteratively optimized, and the model parameters of network A and network B are continuously modified to minimize the regularized residual loss Z of each batch. The regularized residual loss Z is calculated using formula (2):

[0080] Z=L+U+V (2)

[0081] Wherein, L is the cross entropy loss calculated by formula (1) based on the supervised data output results of network A and the labels of the supervised data, U is the cross entropy loss calculated by formula (1) based on the supervised data output results of network B and the labels of the supervised data, and V is the regularized residual consistency loss calculated by structural similarity based on the unsupervised data output results of network A and the unsupervised data output results of network B;

[0082] Step 4: Segmentation of thyroid ultrasound image nodules

[0083] Take a batch of image data from the thyroid ultrasound image acquisition data, instantiate and set the number of images in a batch to 4, and scale the image resolution to 224×224, with p as the width and q as the height in pixels. Construct a tensor G as the input of network A, use the model parameters obtained in training step 3) to initialize model A, use model A to perform prediction, and obtain tensor g, which is the thyroid ultrasound image nodule region segmentation result.

[0084] In this embodiment, the hyperparameters of the experiment are: the optimizer uses the Adam optimizer, the network learning rate is 0.0002, and the momentum coefficient is 0.9. This embodiment uses the training set constructed by sampling ultrasound image data to perform network training. Figure 1 is the visual segmentation result.

[0085] The Dice coefficient and Jaccard index were used to quantitatively evaluate the method of the present invention. When calculating the Dice coefficient, the number of pixels in the intersection of the predicted area and the label area of ​​the thyroid nodule was recorded as R, and the number of pixels in the union of the predicted area and the label area of ​​the thyroid nodule was recorded as Q. Then the Dice coefficient was calculated as The Jaccard index was calculated as The method of the present invention was used for statistics, and a Dice coefficient of 0.822 and a Jaccard index of 0.697 were obtained. Since the embodiment of the present invention uses three cases for labeling and additionally uses 10% of the test data as training data, such statistical results were obtained, indicating that the method of the present invention is very effective when there are few labels.

Claims

1. A thyroid ultrasound image nodule segmentation method based on regularized residual learning, characterized in that: The method inputs a sequence of thyroid ultrasound images with an image resolution of M×N, where 100≤M≤1000 and 100≤N≤1000. The thyroid ultrasound image nodule segmentation includes the following steps: Step 1: Construct a dataset Sample a sequences of thyroid ultrasound images to construct a sampled dataset, where the number of frames in each sequence is b, and 10≤a≤50, 40≤b≤80. Randomly select c frames of images from the b frames of images in each sequence for annotation, and give the labels of thyroid nodules. Use the selected c frames of images and labels to construct an annotated dataset C, where 20≤c≤30. Then use the remaining images in the sampled dataset to construct an unlabeled dataset D. Randomly select n sequences of images from the annotated dataset C, where 3≤n≤10. Randomly select 75% of the images from each of the n sequences of images. Use the selected 75% of the images and the corresponding labels to construct a total training set. Use the remaining images and labels in the annotated dataset C after removing the data in the total training set to construct a test dataset G. Further randomly select m sequence data and the corresponding labels from the total training set for constructing a training set E, where 2≤m<n. Then use the remaining images and the corresponding labels in the total training set to construct a training set F. Further randomly select 10% of the images and the corresponding label data from the test dataset G and supplement them to the training set E and the training set F respectively; Step 2: Construct a neural network The image resolution input to the neural network is p×q, where p is the width and q is the height, in pixels, and 100≤p≤l000, 100≤q≤1000; (1) Basic segmentation network A Use tensor G as the input, with a scale of α×o×p×1, and tensor g as the output, with a scale of α×o×p×1, where α is the number of batches; The basic segmentation network A consists of an encoder and a decoder. For tensor G, after passing through encoding and decoding processes in sequence, the output tensor g is obtained; The encoder consists of 28 units. The first unit is a convolutional unit, containing 16 convolutional kernels, and the shape of each convolutional kernel is 3×3. The stride in the horizontal and vertical directions of the convolution is 2. The structures of the second to the 13th units are the same, including activation processing and one convolution operation, containing 12 convolutional kernels, and the shape of each convolutional kernel is 3×3. The stride in the horizontal and vertical directions of the convolution is 1. The 14th unit is a convolution operation, including activation processing and one convolution operation, containing 160 convolutional kernels, and the shape of each convolutional kernel is 1×1. There is one spatial pooling process, and the pooling scale is 2×2. The structures of the 15th to the 26th units are the same, including activation processing and one convolution operation, containing 12 convolutional kernels, and the shape of each convolutional kernel is l×l. The 27th unit is batch normalization processing. The 28th unit is a convolutional unit, containing 304 convolutional kernels, and the shape of each convolutional kernel is 1×1. The stride in the horizontal and vertical directions of the convolution is 1; The decoder consists of three units. The first unit is a deconvolution unit, which contains 128 convolution kernels. The shape of the convolution kernel is 4×4, and the horizontal and vertical strides of the convolution are both 2. The second unit is a deconvolution unit, which contains 64 convolution kernels. The shape of the convolution kernel is 4×4, and the horizontal and vertical strides of the convolution are both 2. The third unit is a convolution unit, which contains 2 convolution kernels. The shape of the convolution kernel is 1×1, and the horizontal and vertical strides of the convolution are both 1. (2) Residual Learning Network B Tensors G and g are used as input, with scales of α×o×p×1, and tensor h is used as output, with scales of α×o×p×1, where α is the batch size; The residual learning network B consists of 3 layers of residual encoding units and 3 layers of residual decoding units; For the tensor G, after one convolution operation, specifically, including three convolution kernels, the shape of the convolution kernel is 1×1, and the step size of the horizontal and vertical directions of the convolution is 1. The obtained result is concatenated with the tensor g according to the grayscale channel to obtain the tensor I1; the tensor I1 is sequentially processed by the following three layers of residual coding unit and three layers of residual decoding unit: The first-layer residual coding unit is processed as a convolution operation, which includes 64 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical strides of the convolution are both 1. After activation processing, another convolution operation is performed, which includes 64 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical strides of the convolution are both 1, and the tensor I2 is obtained; then the tensor I1 is convolved once, which includes 64 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical strides of the convolution are both 1, batch normalization processing, and activation processing to obtain tensor I3; tensors I2 and I3 are added to obtain tensor I4; tensor I4 is input into the following second-layer residual coding unit processing; When the second-layer residual coding unit is processed, the tensor I4 is batch normalized and activated, and then a convolution operation is performed, including 128 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical steps of the convolution are both 2. Then normalization and activation are performed, and then a convolution operation is performed, including 128 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical steps of the convolution are both 1, to obtain tensor I5; then a convolution operation is performed on tensor I4, including 128 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical steps of the convolution are both 2, and then batch normalization and activation are performed to obtain tensor I6; tensors I5 and I6 are added to obtain tensor I7, and tensor I7 is input into the following third-layer residual coding unit processing; When processing the third-layer residual coding unit, batch normalization and activation processing are performed on I7. After that, a convolution operation is performed, which includes 256 convolution kernels. The shape of the convolution kernels is 3×3, and the horizontal and vertical strides of the convolution are 2. Then normalization and activation processing are performed again, and a convolution operation is performed again, which includes 256 convolution kernels. The shape of the convolution kernels is 3×3, and the horizontal and vertical strides of the convolution are 1, to obtain tensor I8; ​​then a convolution operation is performed on tensor I7, which includes 256 convolution kernels. The shape of the convolution kernels is 3×3, and the horizontal and vertical strides of the convolution are 2. Then normalization and activation processing are performed again to obtain tensor I9. Tensor I8 and tensor I9 are further added to obtain tensor I10; The tensor I10 is batch normalized and activated, and then a convolution operation is performed, which includes 512 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical step sizes of the convolution are both 1. Then it is normalized and activated again, and then a convolution operation is performed again, which includes 512 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical step sizes of the convolution are both 1. The tensor I11 is then processed by the first layer residual decoding unit as follows: When processing the first-layer residual decoding unit, the tensors I10 and I11 are concatenated according to the third channel to obtain the tensor I12. The tensor I12 is batch normalized and activated, and then a convolution operation is performed, which includes 256 convolution kernels. The shape of the convolution kernels is 3×3, and the horizontal and vertical step sizes of the convolution are 1. Then the normalization and activation processing are performed again, and a convolution operation is performed, which includes 512 convolution kernels. The shape of the convolution kernels is 3×3. The horizontal and vertical step sizes of the convolution are 1. The convolution operation is 3×3, and the horizontal and vertical strides of the convolution are both 1, resulting in a tensor I13. The tensor I12 is then convolved once, containing 256 convolution kernels, the shape of the convolution kernels are all 3×3, and the horizontal and vertical strides of the convolution are both 1. Then it is normalized and activated to obtain a tensor I14. The tensors I13 and I14 are added together to obtain a tensor I15. The tensor I15 is then processed by the second-layer residual decoding unit as follows: When processing the second-layer residual decoding unit, the result of upsampling I15 is concatenated with tensor I7 to obtain tensor I16, and the result is batch normalized and activated before convolution operation, which includes 128 convolution kernels, the shape of which is 3×3, and the horizontal and vertical step sizes of the convolution are both 1. The result is batch normalized and activated before convolution operation, which includes 128 convolution kernels, the shape of which is 3×3, and the horizontal and vertical step sizes of the convolution are both 1. The horizontal and vertical strides of the product are both 1, resulting in a tensor I17. Tensor I16 is then convolved, containing 128 convolution kernels, each of which has a shape of 3×3 and a horizontal and vertical stride of 1. The result is batch normalized and activated before being convolved to obtain a tensor I18. Tensor I17 and I18 are added together to obtain a tensor I19. Tensor I19 is then processed by the third-layer residual decoding unit as follows: When processing the third-layer residual decoding unit, the result of upsampling I19 is concatenated with the tensor I4 to obtain the tensor I20. The result is batch normalized and activated, and then a convolution operation is performed, which includes 64 convolution kernels. The shape of the convolution kernel is 3×3, and the step size of the horizontal and vertical directions of the convolution is 1. The result is batch normalized and activated, and then a convolution operation is performed, which includes 64 convolution kernels. The shape of the convolution kernel is 3×3, and the step size of the horizontal and vertical directions of the convolution is 1, to obtain the tensor I21. Then the tensor I20 is convolved. Convolution operation, including 64 convolution kernels, the shape of the convolution kernels are all 3×3, the horizontal and vertical strides of the convolution are both 1, the result is convolution operation, including 64 convolution kernels, the shape of the convolution kernels are all 3×3, the horizontal and vertical strides of the convolution are both 1, the result is batch normalized and activated to obtain tensor I22; tensor I21 is added to tensor I22, and the obtained tensor is convolved, including 2 convolution kernels, the shape of the convolution kernels are all 3×3, the horizontal and vertical strides of the convolution are both 1, to obtain tensor h; Step 3: Training the Neural Network During neural network training, the cross entropy loss for each batch is calculated using the segmentation results output by the network and the labels of the network input data. Specifically: Among them, i represents the i-th pixel, S i represents the i-th pixel label, t i represents the probability that the i-th pixel in the network output tensor belongs to the thyroid nodule area, and W represents the number of pixels; 1) Train the basic segmentation network A 20,000 times Get the training data from the training set E, uniformly scale it to a resolution of p×o, and input it to the network A for training. Iteratively optimize and continuously modify the model parameters of the network A to minimize the loss of each batch. Based on the output of the network A and the label of the input data of the network A, use formula (1) to calculate the cross entropy loss of each batch. 2) Train the residual learning network B 20,000 times Initialize the network using the parameters of network A obtained in step 1), obtain training data from the training set F, input it into network A, obtain tensor g, then input tensor G and tensor g into network B, iterate and optimize, and minimize the loss of each batch by continuously modifying the model parameters of network B. Calculate the cross entropy loss of each batch according to formula (1) based on the output of network B and the label of the input data of network A; 3) Jointly train the basic segmentation network A and network B 20,000 times Initialize the network using the parameters of network A and network B trained in step 1) and step 2), take a batch of data, wherein the first half of the batch of supervised data of the batch of data is obtained from the training set F, and the second half of the batch of unsupervised data of the batch of data is obtained from the unlabeled data set D, and then concatenate the first half of the batch of supervised data and the second half of the batch of unsupervised data according to the grayscale channel, input the resulting tensor into network A, and then divide the tensor g output by network A into the supervised data output result of network A and the unsupervised data output result of network A according to the grayscale channel, and then input the tensor G and the tensor g into network B, and then divide the tensor h output by network B into the supervised data output result of network B and the unsupervised data output result of network B according to the grayscale channel; The training process is iteratively optimized, and the model parameters of network A and network B are continuously modified to minimize the regularized residual loss Z of each batch. The regularized residual loss Z is calculated using formula (2): Z=L+U+V (2) Wherein, L is the cross entropy loss calculated by formula (1) based on the supervised data output results of network A and the labels of the supervised data, U is the cross entropy loss calculated by formula (1) based on the supervised data output results of network B and the labels of the supervised data, and V is the regularized residual consistency loss calculated by structural similarity based on the unsupervised data output results of network A and the unsupervised data output results of network B; Step 4: Segmentation of thyroid ultrasound image nodules Take out a batch of image data of α from the thyroid ultrasound image acquisition data, and scale the image resolution to p×q, where p is the width and q is the height, in pixels, 100≤p≤1000, 100≤q≤1000. Construct a tensor G as the input of network A, use the model parameters obtained in training step 3) to initialize model A, use model A to perform prediction, and obtain tensor g, which is the thyroid ultrasound image nodule region segmentation result.

Images