Thyroid nodule localization method based on multi-level weighted neural network

Abstract

The present invention relates to a thyroid nodule positioning method based on a multi-level weighted neural network. A thyroid nodule positioning model based on a PyTorch deep learning framework is constructed, wherein the constructed thyroid nodule positioning model includes a nodule contour coarse positioning network model and a nodule contour fine positioning network model; for any thyroid ultrasound image to be identified and positioned, the thyroid ultrasound image is preprocessed, and the preprocessed thyroid ultrasound image is sequentially segmented and positioned by the nodule contour coarse positioning network model and the nodule contour fine positioning network model to obtain a thyroid nodule position contour feature map, and according to the thyroid nodule position contour feature map, the position and contour of the thyroid nodule are depicted on the identified and positioned thyroid ultrasound image. The present invention can quickly and accurately locate nodules on thyroid ultrasound images, making the diagnostic results more reliable.

Description

Technical Field

[0001] The invention relates to a thyroid nodule positioning method, in particular to a thyroid nodule positioning method based on a multi-level weighted neural network. Background Art

[0002] In recent years, the incidence of thyroid diseases has continued to rise, and has attracted more and more attention. Thyroid nodules refer to an abnormal growth in the thyroid tissue, which is a common clinical disease. Thyroid nodules can be benign or malignant. Early detection of lesions and correct judgment of their benign or malignant nature are of great guiding significance for subsequent treatment plans.

[0003] Benign and malignant thyroid nodules have identifiable imaging features on ultrasound images. Ultrasound examination has the advantages of low cost, no radiation, safety and reliability, and has become one of the common means of diagnosing thyroid diseases. Therefore, it is of great clinical application value to study how to assist doctors to accurately locate nodules and determine the contours of nodules on ultrasound images. However, thyroid ultrasound images often have problems such as low resolution, uneven grayscale, blurred edges, and severe speckle noise. In addition, the boundary contours, sizes, and positions of thyroid nodules in different populations vary. Therefore, the morphology of nodules on ultrasound images varies greatly, which greatly affects the accuracy of nodule contour judgment.

[0004] With the continuous development of computer technology and algorithms, artificial intelligence technology with deep learning as the core has been deeply studied and widely used in various fields. Assisted analysis of medical images through artificial intelligence technology can reduce the workload of doctors, and at the same time reduce the impact of factors such as differences in imaging equipment and doctor level on diagnostic results.

[0005] In the past decade, researchers have proposed a variety of thyroid nodule segmentation and localization algorithms, but due to the complexity of nodule morphology in thyroid ultrasound images, it is difficult to obtain accurate segmentation and localization results, which makes it difficult to meet the actual requirements of medical assistance. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a thyroid nodule localization method based on a multi-level weighted neural network, which can quickly and accurately locate nodules on thyroid ultrasound images. At the same time, the accurate nodule edge contour helps to judge whether the nodules are benign or malignant, avoids too many subjective factors, and makes the diagnosis results more reliable.

[0007] According to the technical solution provided by the present invention, a thyroid nodule localization method based on a multi-level weighted neural network is constructed to construct a thyroid nodule localization model based on a PyTorch deep learning framework, wherein the constructed thyroid nodule localization model includes a nodule contour coarse localization network model and a nodule contour fine localization network model;

[0008] For any thyroid ultrasound image to be identified and located, the thyroid ultrasound image is preprocessed, and the preprocessed thyroid ultrasound image is segmented and positioned by a nodule contour coarse positioning network model and a nodule contour fine positioning network model in turn to obtain a thyroid nodule position contour feature map, and based on the thyroid nodule position contour feature map, the position and contour of the thyroid nodule are depicted on the identified and located thyroid ultrasound image.

[0009] The preprocessing of the thyroid ultrasound image includes converting the thyroid ultrasound image to a target size specification and normalizing the pixel values ​​of the converted image.

[0010] The following steps are involved in building a thyroid nodule localization model based on the PyTorch deep learning framework:

[0011] Step 1: Provide a target localization model based on the PyTorch deep learning framework and configure hyperparameters for training the target localization model;

[0012] Step 2: Prepare a training data set and a test data set for training the target positioning model. When using the prepared data set to train the target positioning model, the FocalLoss loss function is used as the loss function. The Adam optimizer is used to optimize the model parameters of the current target positioning model. After the training is terminated, a thyroid nodule positioning model based on the PyTorch deep learning framework is obtained.

[0013] The nodule contour rough positioning network model processes the preprocessed thyroid ultrasound image, including the following steps:

[0014] Step 10: Preprocess the thyroid ultrasound image to be identified and located to obtain the feature Figure 1 , and the obtained features Figure 1 The input is to the nodule contour rough positioning network model, which uses a convolution operation with a step size of 2 to achieve downsampling, and uses a bilinear interpolation method to gradually expand the feature map, and finally obtains a feature with a channel number of 2. Figure 2 ;

[0015] Step 11: Features Figure 2 The element values ​​in the two channels are compared to obtain the features that characterize the nodule location indication features. Figure 3 ;

[0016] Step 12: Based on the characteristics Figure 3, determine the coarse nodule positioning contour and the contour expansion rectangle that matches the determined coarse nodule positioning contour, intercept image blocks on the thyroid ultrasound image according to the determined contour expansion rectangle, use the intercepted image blocks to generate feature map four, and input the feature map four into the nodule contour precise positioning network model.

[0017] The nodule contour precise positioning network model includes a nodule contour first positioning sub-network, a nodule contour second positioning sub-network and a nodule contour third positioning sub-network, wherein the feature image four is processed by the nodule contour first positioning sub-network to obtain a feature image five with a channel number of 2; the feature image five and the feature image four are processed by the nodule contour second positioning sub-network to obtain a feature image six with a channel number of 2; the feature image six and the feature image four are processed by the nodule contour third positioning sub-network to obtain a feature image seven with a channel number of 2.

[0018] Compare the element values ​​in the two channels in the feature map 7, and obtain the feature map 8 after the comparison;

[0019] Adjust the size of feature map 8 to the same size as the outline expansion rectangle, and overlay the resized feature map 8 onto the feature Figure 3 In order to obtain the feature map nine that characterizes the precise location indication characteristics of the nodule, the feature map nine is used to depict the position and outline of the thyroid nodule on the identified and located thyroid ultrasound image.

[0020] 8. According to claim 6, the thyroid nodule positioning method based on a multi-level weighted neural network is characterized in that, when feature map 4 is processed by the first nodule contour positioning sub-network to obtain feature map 5, the number of channels of feature map 5 is 2, wherein the first channel feature value of any feature point represents the probability value of the feature point being a non-thyroid nodule, and the second channel feature value of any feature point represents the probability value of the feature point being a thyroid nodule.

[0021] When the second nodule contour localization sub-network processes feature map 5 and feature map 4, the following steps are included:

[0022] Step 20: Evenly divide feature map 4 into feature sub-maps F i (i=1, 2, 3, 4), the four feature maps are processed in sequence to obtain four feature maps M′ with 2 channels i (i=1, 2, 3, 4);

[0023] Step 21: Get the feature sub-graph F according to the four-division of the feature graph i The feature graph 5 is evenly divided in the manner of (i=1, 2, 3, 4) to obtain the feature sub-graph N i (i=1, 2, 3, 4), and the feature map N i (i=1, 2, 3, 4) and feature map M′i (i=1, 2, 3, 4) are spliced ​​by channel to obtain a feature splicing map with 4 channels;

[0024] Step 22: Perform feature fusion on the feature splicing map to obtain a feature map P with 2 channels. i , where for the feature map P i , the first channel characteristic value of any feature point represents the probability value of the feature point being a non-thyroid nodule, and the second channel characteristic value of any feature point represents the probability value of the feature point being a thyroid nodule;

[0025] Step 23: feature map P i For any feature point of the feature point, the credibility s of the prediction result of the feature point is calculated, and the credibility feature map P of the prediction result is obtained according to the credibility s of the prediction results of all feature points i c ; According to the calculation method of the prediction result credibility feature map, the feature sub-map N i (i=1, 2, 3, 4) Calculate the prediction result credibility feature map N i c ;

[0026] Step 24: Use the prediction result credibility feature map P i c And the prediction result credibility feature map N i c , calculate the optimized sorting feature graph C i ;

[0027] Step 25: Optimize the sorting feature graph C i The eigenvalues ​​in are arranged from large to small to determine the coordinates of the first k feature points; according to the coordinates of the first k feature points determined, the feature subgraph N i The eigenvalues ​​of (i=1, 2, 3, 4) are processed by weighted summation to output the feature subgraph M i (i=1, 2, 3, 4);

[0028] Step 26: Get the feature sub-graph F according to the four-division of the feature graph i The coordinate parameters of (i=1, 2, 3, 4) are used to transform the feature subgraph M i (i=1, 2, 3, 4) are spliced ​​at different positions to obtain a feature map 6 with a channel number of 2 and a size of 128*128.

[0029] The method for locating thyroid nodules based on a multi-level weighted neural network according to claim 6, wherein in step 24, the prediction result credibility feature map P is used i c And the prediction result credibility feature map N ic , get the sorting feature map C i When , we have: in, Represents point-wise multiplication.

[0030] When the third nodule contour localization subnetwork processes the feature map 6 and the feature map 4 to obtain the feature map 7 with a channel number of 2, the specific steps include:

[0031] Step 30: Evenly divide the feature graph F4 into feature sub-graphs, and perform j (j=1, 2, 3, ..., 16) are processed in sequence to obtain 16 feature maps M′ with 2 channels j (j=1, 2, 3, ..., 16);

[0032] Step 31: Obtain feature sub-graph F′ based on the four-division feature graph j The feature graph 6 is evenly divided in the manner of (j=1, 2, 3, ..., 16) to obtain the feature sub-graph N j (j=1, 2, 3, ..., 16), and the feature subgraph N j (j=1, 2, 3, ..., 16) and feature map M′ j (j=1, 2, 3, ..., 16) are spliced ​​by channel to obtain a feature splicing map with 4 channels;

[0033] Step 32: perform feature fusion on the above feature splicing map to obtain a feature map P′ with 2 channels. j (j=1, 2, 3, ..., 16), where for the feature map P′ j (j=1, 2, 3, ..., 16), the first channel characteristic value of any feature point represents the probability value of the feature point being a non-thyroid nodule, and the second channel characteristic value of any feature point represents the probability value of the feature point being a thyroid nodule;

[0034] Step 33: feature map P′ j For any feature point of (j=1, 2, 3, ..., 16), the credibility s of the prediction result of the feature point is calculated, and the credibility feature map of the prediction result is obtained according to the credibility s of the prediction results of all feature points. According to the calculation method of the prediction result credibility feature map, the feature sub-map N j (j=1, 2, 3, ..., 16) Calculate the prediction result credibility feature map

[0035] Step 34: Use the prediction result credibility feature map And the prediction result credibility feature map Calculate the optimized sorting feature graph C′ j (j=1, 2, 3, ..., 16);

[0036] Step 35: Optimize the sorting feature graph C′ j Arrange the eigenvalues ​​in from large to small, determine the coordinates of the first q feature points; according to the coordinates of the first q feature points determined, j The eigenvalues ​​of (j=1, 2, 3, ..., 16) are processed by weighted summation to output the feature subgraph M j (j=1, 2, 3, ..., 16);

[0037] Step 36: Obtain feature sub-graph F′ based on the four-division feature graph j The coordinate parameters of (j=1, 2, 3, ..., 16) are used to transform the feature subgraph M j (j=1, 2, 3, ..., 16) are spliced ​​at different positions to obtain a feature map 7 with a channel number of 2 and a size of 128*128.

[0038] The advantages of the present invention are as follows: for the thyroid ultrasound image to be identified and located, the thyroid ultrasound image is preprocessed, and the preprocessed thyroid ultrasound image is segmented and positioned by a nodule contour coarse positioning network model and a nodule contour fine positioning network model in turn to obtain a thyroid nodule position contour feature map, and according to the thyroid nodule position contour feature map, the position and contour of the thyroid nodule are depicted on the identified and located thyroid ultrasound image. During the segmentation and positioning processing, the edge information of the segmented image is more refined, and the obtained contour line is more accurate. It has good adaptability to thyroid ultrasound image nodules, especially small nodules, so that the nodules on the thyroid ultrasound image can be quickly and accurately positioned. At the same time, the accurate nodule edge contour is helpful for judging whether the nodule is benign or malignant, avoiding too many subjective factors, and making the diagnosis result more reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the present invention.

[0040] Figure 2 Schematic diagram of the workflow of the nodule contour precise positioning network model of the present invention.

[0041] Figure 3 This is a schematic diagram of the specific segmentation and positioning of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with specific drawings and embodiments.

[0043] like Figure 1As shown: In order to quickly and accurately locate nodules on thyroid ultrasound images, accurate nodule edge contours are helpful for judging whether the nodules are benign or malignant, avoiding too many subjective factors, and making the diagnosis results more reliable, the thyroid nodule positioning method of the present invention is specifically as follows: constructing a thyroid nodule positioning model based on the PyTorch deep learning framework, wherein the constructed thyroid nodule positioning model includes a nodule contour coarse positioning network model and a nodule contour fine positioning network model;

[0044] For any thyroid ultrasound image to be identified and located, the thyroid ultrasound image is preprocessed, and the preprocessed thyroid ultrasound image is segmented and positioned by a nodule contour coarse positioning network model and a nodule contour fine positioning network model in turn to obtain a thyroid nodule position contour feature map, and based on the thyroid nodule position contour feature map, the position and contour of the thyroid nodule are depicted on the identified and located thyroid ultrasound image.

[0045] Specifically, the PyTorch deep learning framework is an existing commonly used deep learning framework. Therefore, the PyTorch deep learning framework can be used to build a thyroid nodule localization model based on the PyTorch deep learning framework, so as to use the constructed thyroid nodule localization model based on the PyTorch deep learning framework to locate thyroid nodules in thyroid ultrasound images.

[0046] In an embodiment of the present invention, the constructed thyroid nodule positioning model includes a nodule contour coarse positioning network model and a nodule contour fine positioning network model. When working specifically, for any thyroid ultrasound image to be identified and positioned, the thyroid ultrasound image is preprocessed, and the preprocessed thyroid ultrasound image is sequentially segmented and positioned by the nodule contour coarse positioning network model and the nodule contour fine positioning network model to obtain a thyroid nodule position contour feature map, and the position and contour of the thyroid nodule are depicted on the identified and positioned thyroid ultrasound image according to the thyroid nodule position contour feature map. For the specific segmentation and positioning process of the nodule contour coarse positioning network model and the nodule contour fine positioning network model, reference can be made to the above specific description.

[0047] In the embodiment of the present invention, professionals annotate the collected thyroid ultrasound images and convert the annotated images into binary grayscale images, wherein the grayscale value of the thyroid nodule area is set to 255, and the grayscale value of other areas is set to 0. Since the image sizes from different ultrasound devices are different, preprocessing is required before inputting the model, that is, the sizes of all thyroid ultrasound images and annotated images are uniformly converted to the target size specification. In specific implementation, the target size specification is 512*512; at the same time, all pixel values ​​are divided by 255, that is, the pixel values ​​are normalized to [0,1].

[0048] Furthermore, the construction of a thyroid nodule localization model based on the PyTorch deep learning framework includes the following steps:

[0049] Step 1: Provide a target localization model based on the PyTorch deep learning framework and configure hyperparameters for training the target localization model;

[0050] A target localization model based on the PyTorch deep learning framework is provided, and the model parameters are trained using the FocalLoss loss function and the Adam optimizer, and the hyperparameters for training the target localization model are configured. After reaching the training termination state, a thyroid nodule localization model based on the PyTorch deep learning framework can be obtained.

[0051] Specifically, the configured hyperparameters include learning rate, batchsize, weight decay coefficient, etc. The specific method and process of configuring the hyperparameters can adopt the existing commonly used methods, which are well known to those in the technical field and will not be repeated here. The specific method and process of determining the termination state of training can be consistent with the existing ones and are well known to those in the technical field.

[0052] Step 2: Prepare a training data set and a test data set for training the target positioning model. When using the prepared data set to train the target positioning model, the FocalLoss loss function is used as the loss function. The Adam optimizer is used to optimize the model parameters of the current target positioning model. After the training is terminated, a thyroid nodule positioning model based on the PyTorch deep learning framework is obtained.

[0053] Specifically, when making a data set, thyroid ultrasound images are collected from hospitals, and professionals annotate the collected images to form training and test data sets. The specific conditions and specific production processes of training and test data sets are well known to those skilled in the art and will not be described in detail here. Of course, in the specific implementation, the ultrasound images input into the model are preprocessed images. The specific preprocessing can refer to the above description and will not be described in detail here.

[0054] Furthermore, when the nodule contour rough positioning network model processes the preprocessed thyroid ultrasound image, the following steps are included:

[0055] Step 10: Preprocess the thyroid ultrasound image to be identified and located to obtain the feature Figure 1 , and the obtained features Figure 1 The input is to the nodule contour rough positioning network model, which uses a convolution operation with a step size of 2 to achieve downsampling, and uses a bilinear interpolation method to gradually expand the feature map, and finally obtains a feature with a channel number of 2. Figure 2 ;

[0056] Specifically, in the nodule contour rough positioning network model, the convolution kernel size of the convolution operation is 3*3, the step size is 1 or 2, and the convolution operation with a step size of 2 is used to achieve downsampling, so that the size of the feature map is reduced; the specific method of batch normalization operation is:

[0057]

[0058] Among them, x′ is the normalized output data, x is the input data to be normalized; mean(x) and Var(x) are the mean and variance of the batch data respectively; γ and β are learnable scaling parameters and translation parameters, and their initial values ​​are set to 1 and 0 respectively.

[0059] In addition, the activation operation uses the Leaky Relu activation function. The specific situation of the Leaky Relu activation function is consistent with the existing one and will not be repeated here. After preprocessing, the thyroid ultrasound image to be identified and located obtains a feature of size 512*512 Figure 1 .feature Figure 1 After being processed by the nodule contour rough positioning network model, the feature with 2 channels and size of 512*512 is obtained. Figure 2 .

[0060] The process of using the nodule contour coarse positioning network model to implement the convolution operation with a step size of 2 to achieve downsampling and bilinear interpolation is consistent with the existing ones and will not be repeated here.

[0061] Step 11: Features Figure 2 The element values ​​in the two channels are compared to obtain the features that characterize the nodule location indication features. Figure 3 ;

[0062] Specifically, compare features by channel Figure 2 If the element value of the first channel is greater than or equal to the corresponding element value of the second channel, the value is set to 0, otherwise, it is set to 1. After comparing one by one, the number of channels is 1, which is consistent with the feature Figure 2 Features of the same size Figure 3 .

[0063] feature Figure 3 That is, it is an indicative feature map of the location of the coarse nodule. A feature value of 1 indicates that the pixel point at this location in the thyroid ultrasound image is a pixel point of the thyroid nodule, and a feature value of 0 indicates that the pixel point at this location in the thyroid ultrasound image is not a pixel point of the thyroid nodule.

[0064] Step 12: Based on the characteristics Figure 3, determine the coarse nodule positioning contour and the contour expansion rectangle that matches the determined coarse nodule positioning contour, intercept image blocks on the thyroid ultrasound image according to the determined contour expansion rectangle, use the intercepted image blocks to generate feature map four, and input the feature map four into the nodule contour precise positioning network model.

[0065] Specifically, the features Figure 3 The contour composed of all the pixels with a value of 1 is the rough positioning contour of the nodule. According to the rough positioning contour, the minimum bounding rectangle of the rough positioning contour of the nodule is determined, and w and h are used to represent the width and height of the minimum bounding rectangle respectively. Compare the size of rectangle w and h, and record the value of the long side of the rectangle as L. The minimum bounding rectangle is expanded by 0.25×L pixels in the four directions of up, down, left, and right to obtain the contour expansion rectangle of the rough positioning of the nodule. The four coordinate values ​​of the contour expansion rectangle are used to intercept the image block on the thyroid ultrasound image and convert it to 128*128 size; after the size conversion, all pixel values ​​are divided by 255, and the pixel values ​​are normalized to [0,1] to obtain feature map 4, which is the input of the nodule contour precise positioning network model.

[0066] like Figure 2 As shown, the nodule contour precise positioning network model includes a nodule contour first positioning sub-network, a nodule contour second positioning sub-network and a nodule contour third positioning sub-network, wherein the feature map four is processed by the nodule contour first positioning sub-network to obtain the feature map five; the feature map five and the feature map four are processed by the nodule contour second positioning sub-network to obtain the feature map six with the number of channels being 2; the feature map six and the feature map four are processed by the nodule contour third positioning sub-network to obtain the feature map seven with the number of channels being 2.

[0067] For specific implementation, please refer to the specific work of the nodule contour precise positioning network model. Figure 2 Among them, feature map 5 is obtained after feature map 4 is processed by the first positioning sub-network of nodule contour; feature map 5 and feature map 4 are processed by the second positioning sub-network of nodule contour to obtain feature map 6 with 2 channels; feature map 6 and feature map 4 are processed by the third positioning sub-network of nodule contour to obtain feature map 7 with 2 channels.

[0068] Figure 2In the figure, the first positioning subnetwork of the nodule contour includes a Net1 module, wherein a feature map 4 of size 128*128 is input into the Net1 module, and the Net1 module outputs a feature map 5 of size 128*128 and number of channels 2. For feature map 5, the first channel feature value of any feature point represents the probability value of the feature point being a non-thyroid nodule, and the second channel feature value of any feature point represents the probability value of the feature point being a thyroid nodule. The method and process of processing the feature map using the Net1 module to obtain feature map 5 are consistent with the existing ones, are well known to those skilled in the art, and will not be repeated here.

[0069] Figure 2 In the example, the second nodule contour localization subnetwork includes a Net2 module and a Net2 optimization module. When the second nodule contour localization subnetwork processes feature maps 5 and 4, it includes the following steps:

[0070] Step 20: Evenly divide feature map 4 into feature sub-maps F i (i=1, 2, 3, 4), the four feature maps are processed in sequence to obtain four feature maps M′ with 2 channels i (i=1, 2, 3, 4);

[0071] Specifically, the feature image 4 is evenly divided into feature sub-images F by using the commonly used technical means in the technical field. i (i=1, 2, 3, 4), the four feature sub-graphs are input into the Net2 module in sequence, and the output feature graph M′ with the same size of 64*64 and the number of channels of 2 can be obtained. i (i = 1, 2, 3, 4), i.e., feature map M′ i (i=1, 2, 3, 4) and the characteristic subgraph F i (i=1, 2, 3, 4) are in one-to-one correspondence. Using the Net2 module, we get the feature map M′ i The specific methods and processes of (i=1, 2, 3, 4) are consistent with the existing ones and are well known to those skilled in the art, and will not be described in detail here.

[0072] Step 21: Get the feature sub-graph F according to the four-division of the feature graph i The feature image 5 is evenly divided in the manner of (i=1, 2, 3, 4), and the feature image 5 is divided to obtain the feature sub-image N i (i=1, 2, 3, 4), and the feature map N i (i=1, 2, 3, 4) and feature map M′ i (i=1, 2, 3, 4) are spliced ​​by channel to obtain a feature splicing map with 4 channels;

[0073] Specifically, the feature graph is divided into four parts to obtain the feature sub-graph F iWhen , record the characteristic subgraph F i The four position coordinate points corresponding to feature map 4 Use these four coordinate points to split feature map N with a size of 64*64 and a channel number of 2 on feature map 5. i (i=1, 2, 3, 4).

[0074] The feature map N i (i=1, 2, 3, 4) and feature map M′ i (i=1, 2, 3, 4) are spliced ​​by channel to obtain a feature splicing map with 4 channels. The specific channel splicing method and process are consistent with the existing ones and are well known to those skilled in the art, so they will not be repeated here.

[0075] Step 22: Input the feature concatenation map into the feature fusion network for feature fusion to obtain a feature map P with a channel number of 2. i (i=1, 2, 3, 4), where for the feature map P i The first channel characteristic value of any feature point represents the probability value of the feature point being a non-thyroid nodule, and the second channel characteristic value of any feature point represents the probability value of the feature point being a thyroid nodule.

[0076] Specifically, after feature fusion, the feature map P with a size of 64*64 and a channel number of 2 can be output. i ; For feature map P i , the first channel characteristic value of any feature point represents the probability value of the feature point being a non-thyroid nodule, and the second channel characteristic value of any feature point represents the probability value of the feature point being a thyroid nodule;

[0077] Step 23: For feature map P i For any feature point of the feature point, the credibility s of the prediction result of the feature point is calculated, and the credibility feature map P of the prediction result is obtained according to the credibility s of the prediction results of all feature points i c ; According to the calculation method of the prediction result credibility feature map, the feature sub-map N i Calculate the prediction result credibility feature map N i c ;

[0078] Specifically, for the feature map P i The credibility s of the prediction result of the feature point is calculated for each feature point, and the specific calculation method is:

[0079] s=|value1-value2|

[0080] Among them, value1 is the probability value of the feature point being a non-thyroid nodule, and value2 is the probability value of the feature point being a thyroid nodule.

[0081] Feature map P i After the prediction result credibility s of all feature points is calculated, the corresponding prediction result credibility feature map P is obtained. i c Similarly, for the feature subgraph N i Use the same method to calculate the corresponding prediction result credibility feature map N i c , that is, for the feature subgraph N i The credibility of the prediction result of the feature point is calculated based on the feature points. The specific calculation method can refer to the above description and will not be repeated here.

[0082] Step 24: Use the prediction result credibility feature map P i c And the prediction result credibility feature map N i c , get the sorting feature map C i ;

[0083] Specifically, using the prediction result credibility feature map P i c And the prediction result credibility feature map N i c , get the sorting feature map C i When , we have: in, Represents point-wise multiplication.

[0084] Step 25: Sort feature graph C i The eigenvalues ​​in are arranged from large to small to determine the coordinates of the first k feature points; according to the coordinates of the first k feature points determined, the feature subgraph N i The eigenvalues ​​of (i=1, 2, 3, 4) are processed by weighted summation to output the feature subgraph M i (i=1, 2, 3, 4);

[0085] Specifically, for the sorting feature graph C i Sort by eigenvalue from large to small, and determine the coordinates of the k feature points in the front, which is the feature subgraph N i The coordinates of the feature points that need to be optimized on the feature subgraph N i The eigenvalues ​​of are optimized by weighted summation. The specific formula is:

[0086] value′1=0.3×value1(N i )+0.7×value1(M′i )

[0087] value′2=0.3×value2(N i )+0.7×value2(M′ i )

[0088] Among them, value1(N i ) is the characteristic subgraph N i The eigenvalue of the first channel of the previous feature point, value2(N i ) is the characteristic subgraph N i The eigenvalue of the second channel of the feature point, value1(M′ i ) is the feature graph M′ i The eigenvalue of the first channel of the corresponding feature point, value2(M′ i ) is the feature graph M′ i The feature value of the second channel of the feature point corresponding to the above, value'1 is the feature value of the first channel after the feature point is optimized, and value'2 is the feature value of the second channel after the feature point is optimized. When determining the coordinates of the first k feature points, k is an empirical value. The following gives an example where k is 1024. The specific value is well known to those skilled in the art to meet the output of the feature sub-graph M required. i (i=1, 2, 3, 4) shall prevail.

[0089] Step 26: Segment the feature sub-graph F according to the feature graph i The coordinate parameters when (i=1, 2, 3, 4) are used to transform the feature subgraph M i (i=1, 2, 3, 4) are concatenated to obtain feature map 6 with a channel number of 2.

[0090] Specifically, for the feature subgraph N i After the feature values ​​of the top 1024 (k=1024) feature points are optimized, the feature subgraph M output by the Net2 optimization module is i (i=1, 2, 3, 4). Four feature maps M i (i=1, 2, 3, 4) are spliced ​​together according to the positions when they are segmented, and a feature map 6 with a size of 128*128 and a channel number of 2 is obtained.

[0091] Specifically, the feature subgraph M is output in the above steps 23 to 26. i The processes of (i=1, 2, 3, 4) are all processed by the Net2 optimization module in the second nodule contour positioning sub-network.

[0092] Figure 2In the third positioning sub-network for nodule contour, the sub-network includes Net3 module and Net3 optimization module. The specific segmentation and positioning process is as follows:

[0093] When the third nodule contour localization subnetwork processes the feature map 6 and the feature map 4 to obtain the feature map 7 with a channel number of 2, the specific steps include:

[0094] Step 30: Evenly divide the feature graph F4 into feature sub-graphs, and perform j (j=1, 2, 3, ..., 16) are processed in sequence to obtain sixteen feature maps M′ with 2 channels j (j=1, 2, 3, ..., 16);

[0095] Specifically, the feature map is divided into sixteen feature sub-maps F′ of size 32*32. j (j=1, 2, 3, ..., 16), and are input into the Net3 module in sequence. The Net3 module can output a feature map M′ with the same size of 32*32 and 2 channels j (j=1, 2, 3, ..., 16). The feature graph is divided into four parts to obtain the feature sub-graph F′ j (j=1, 2, 3, ..., 16), record the characteristic subgraph F′ j (j=1, 2, 3, ..., 16) The four position coordinate points corresponding to the feature map 4

[0096] Step 31: Obtain feature sub-graph F′ based on the four-division feature graph j The feature graph 6 is evenly divided in the manner of (j=1, 2, 3, ..., 16) to obtain the feature sub-graph N j (j=1, 2, 3, ..., 16), and the feature subgraph N j (j=1, 2, 3, ..., 16) and feature map M′ j (j=1, 2, 3, ..., 16) are spliced ​​by channel to obtain a feature splicing map with 4 channels;

[0097] Specifically, these four coordinate points are used to split feature map 6 into a feature sub-map N with a size of 32*32 and a channel number of 2. j (j=1, 2, 3, ..., 16), the specific segmentation is N j (j=1, 2, 3, ..., 16) and the specific method and process of the feature splicing graph can refer to the above description.

[0098] Step 32: perform feature fusion on the above feature splicing map to obtain a feature map P′ with 2 channels. j (j=1, 2, 3, ..., 16), where for the feature map P′ j(j=1, 2, 3, ..., 16), the first channel feature value of any feature point represents the probability value of the feature point being a non-thyroid nodule, and the second channel feature value of any feature point represents the probability value of the feature point being a thyroid nodule;

[0099] Specifically, we get the feature map P′ j (j=1, 2, 3, ..., 16), and the characteristic graph P′ j The characteristics of (j=1, 2, 3, ..., 16) can all refer to the above description.

[0100] Step 33: feature map P′ j For any feature point of (j=1, 2, 3, ..., 16), the credibility s of the prediction result of the feature point is calculated, and the credibility feature map of the prediction result is obtained according to the credibility s of the prediction results of all feature points. According to the calculation method of the prediction result credibility feature map, the feature sub-map N j (j=1, 2, 3, ..., 16) Calculate the prediction result credibility feature map

[0101] Step 34: Use the prediction result credibility feature map And the prediction result credibility feature map Calculate the optimized sorting feature graph C′ j (j=1, 2, 3, ..., 16);

[0102] Step 35: Optimize the sorting feature graph C′ j Arrange the eigenvalues ​​in from large to small, determine the coordinates of the first q feature points; according to the coordinates of the first q feature points determined, j The eigenvalues ​​of (j=1, 2, 3, ..., 16) are processed by weighted summation to output the feature subgraph M j (j=1, 2, 3, ..., 16);

[0103] Step 36: Obtain feature sub-graph F′ based on the four-division feature graph j The coordinate parameters of (j=1, 2, 3, ..., 16) are used to transform the feature subgraph M j (j=1, 2, 3, ..., 16) are spliced ​​at different positions to obtain a feature map 7 with a channel number of 2 and a size of 128*128.

[0104] Specifically, the above steps 31 to 36 are all completed by the Net3 optimization module. The specific situation of the Net3 optimization module can refer to the working process description of the Net2 optimization module, which will not be repeated here. j(j=1, 2, 3, ..., 16). Sixteen feature maps M j (j=1, 2, 3, ..., 16) are spliced ​​together according to the positions when they are segmented, and a feature map 7 with a size of 128*128 and a channel number of 2 is obtained. q is similar to the above k, both of which are empirical values. For details, please refer to the above description, which will not be repeated here.

[0105] Further, the two channel element values ​​of the feature map 7 are compared, and the feature map 8 is obtained after the comparison;

[0106] Adjust the size of the feature image 8 to the same size as the outer expansion rectangle of the nodule rough positioning outline, and overlay the resized feature image 8 onto the feature Figure 3 In order to obtain the characteristic image 9 representing the location indication characteristics of the sperm nodule, the characteristic image 9 is used to depict the location and outline of the thyroid nodule on the identified and located thyroid ultrasound image.

[0107] In the embodiment of the present invention, the method for comparing elements in two channels in the feature map 7 can refer to the method for comparing elements in two channels in the feature map 7. Figure 2 Get Features Figure 3 The description of the above is omitted here. Feature Figure 9 is the feature figure of the thyroid nodule position contour.

[0108] During the specific implementation, the segmentation and positioning effects of Unet, Unet++, DeepLabv3+ and the present invention are compared and evaluated. The present invention uses four quantitative indicators, namely Dice similarity coefficient, IOU, Sensitivity and Precision, to evaluate the segmentation and positioning effect. The Dice Similarity Coefficient is a set similarity measurement indicator, which is often used to calculate the similarity of two samples; the full name of IOU is Intersection over Union, which represents the ratio of the intersection of the two sets of model prediction results and true labels to the union of the two sets; Sensitivity represents the proportion of correctly predicted positive samples to the total positive samples. In the embodiment of the present invention, thyroid nodules are considered positive, and Precision represents the proportion of correctly predicted samples among the samples predicted to be positive.

[0109] Segmentation and positioning comparison table

[0110] method Dice IOU Sensitivity Precision Unet 0.8805 0.7925 0.8846 0.8920 Unet++ 0.9025 0.8252 0.9021 0.9113 DeepLabv3+ 0.8926 0.8115 0.8943 0.9018 The present invention 0.9169 0.8483 0.9183 0.9204

[0111] It can be seen from the quantitative indicators in the above table that compared with the traditional method, the method of the present invention has obvious improvements in the four indicators of Dice similarity coefficient, IOU, Sensitivity and Precision.

[0112] In summary, if Figure 3 As shown, for the thyroid ultrasound image to be identified and located, the thyroid ultrasound image is preprocessed, and the preprocessed thyroid ultrasound image is segmented and positioned by a nodule contour coarse positioning network model and a nodule contour fine positioning network model in turn to obtain a thyroid nodule position contour feature map, and based on the thyroid nodule position contour feature map, the position and contour of the thyroid nodule are depicted on the identified and located thyroid ultrasound image.

[0113] During segmentation and positioning processing, the edge information of the segmented image is more refined, the obtained contour line is more accurate, and it has good adaptability to thyroid ultrasound image nodules, especially small nodules. Figure 3 In the last picture, the circular area depicts the location and outline of the thyroid nodule.

Claims

1. A thyroid nodule localization method based on a multi-level weighted neural network, characterized in that: Construct a thyroid nodule localization model based on the PyTorch deep learning framework, wherein the constructed thyroid nodule localization model includes a nodule contour coarse localization network model and a nodule contour fine localization network model; For any thyroid ultrasound image to be identified and located, the thyroid ultrasound image is preprocessed, and the preprocessed thyroid ultrasound image is segmented and located by a nodule contour coarse positioning network model and a nodule contour fine positioning network model in sequence to obtain a thyroid nodule position contour feature map, and the position and contour of the thyroid nodule are depicted on the identified and located thyroid ultrasound image according to the thyroid nodule position contour feature map; The nodule contour rough positioning network model processes the preprocessed thyroid ultrasound image, including the following steps: Step 10, preprocessing the thyroid ultrasound image to be identified and located to obtain a feature map 1, and inputting the obtained feature map 1 into a nodule contour coarse positioning network model, wherein the nodule contour coarse positioning network model uses a convolution operation with a step size of 2 to achieve downsampling, and uses a bilinear interpolation method to gradually amplify the feature map, and finally obtains a feature map 2 with a channel number of 2; Step 11, comparing the element values ​​in the two channels in the feature image 2 to obtain a feature image 3 representing the nodule position indication feature; Step 12: according to the feature image 3, determine the coarse nodule positioning contour and the contour expansion rectangle matched with the determined coarse nodule positioning contour, intercept an image block on the thyroid ultrasound image according to the determined contour expansion rectangle, generate a feature image 4 using the intercepted image block, and input the feature image 4 into the nodule contour precise positioning network model; The nodule contour precise positioning network model includes a nodule contour first positioning sub-network, a nodule contour second positioning sub-network and a nodule contour third positioning sub-network, wherein the feature image 4 is processed by the nodule contour first positioning sub-network to obtain a feature image 5 with a channel number of 2; the feature image 5 and the feature image 4 are processed by the nodule contour second positioning sub-network to obtain a feature image 6 with a channel number of 2; the feature image 6 and the feature image 4 are processed by the nodule contour third positioning sub-network to obtain a feature image 7 with a channel number of 2; When the second nodule contour localization sub-network processes feature map 5 and feature map 4, the following steps are included: Step 20: Evenly divide feature map 4 into feature sub-maps F i (i=1, 2, 3, 4), the four feature maps are processed in sequence to obtain four feature maps M′ with 2 channels i (i=1, 2, 3, 4); Step 21: Get the feature sub-graph F according to the four-division of the feature graph i The feature graph 5 is evenly divided in the manner of (i=1, 2, 3, 4) to obtain the feature sub-graph N i (i=1, 2, 3, 4), and the feature subgraph N i (i=1, 2, 3, 4) and feature map M′ i (i=1, 2, 3, 4) are spliced ​​by channel to obtain a feature splicing map with 4 channels; Step 22: Perform feature fusion on the feature splicing map to obtain a feature map P with 2 channels. i , where for the feature map P i , the first channel characteristic value of any feature point represents the probability value of the feature point being a non-thyroid nodule, and the second channel characteristic value of any feature point represents the probability value of the feature point being a thyroid nodule; Step 23: feature map P i For any feature point of the feature point, the credibility s of the prediction result of the feature point is calculated, and the credibility feature map P of the prediction result is obtained according to the credibility s of the prediction results of all feature points i c ; According to the calculation method of the prediction result credibility feature map, the feature sub-map N i (i=1, 2, 3, 4) Calculate the prediction result credibility feature map N i c ; Step 24: Use the prediction result credibility feature map P i c And the prediction result credibility feature map N i c , calculate the optimized sorting feature graph C i (i=1, 2, 3, 4); Step 25: Optimize the sorting feature graph C i The eigenvalues ​​in are arranged from large to small to determine the coordinates of the first k feature points; according to the coordinates of the first k feature points determined, the feature subgraph N i The eigenvalues ​​of (i=1, 2, 3, 4) are processed by weighted summation to output the feature subgraph M i (i=1, 2, 3, 4); Step 26: Get the feature sub-graph F according to the four-division of the feature graph i The coordinate parameters of (i=1, 2, 3, 4) are used to transform the feature subgraph M i (i=1, 2, 3, 4) are spliced ​​at positions to obtain a feature map 6 with 2 channels and a size of 128*128; When the third nodule contour localization subnetwork processes the feature map 6 and the feature map 4 to obtain the feature map 7 with a channel number of 2, the specific steps include: Step 30: Evenly divide the feature graph F4 into feature sub-graphs, and perform j (j=1, 2, 3, ..., 16) are processed in sequence to obtain sixteen feature maps M′ with 2 channels j (j=1, 2, 3, ..., 16); Step 31: Obtain feature sub-graph F′ based on the four-division feature graph j The feature graph 6 is evenly divided in the manner of (j=1, 2, 3, ..., 16) to obtain the feature sub-graph N j (j=1, 2, 3, ..., 16), and the feature subgraph N j (j=1, 2, 3, ..., 16) and feature map M′ j (j=1, 2, 3, ..., 16) are spliced ​​by channel to obtain a feature splicing map with 4 channels; Step 32: perform feature fusion on the above feature splicing map to obtain a feature map P′ with 2 channels. j (j=1, 2, 3, ..., 16), where for the feature map P′ j (j=1, 2, 3, ..., 16), the first channel feature value of any feature point represents the probability value of the feature point being a non-thyroid nodule, and the second channel feature value of any feature point represents the probability value of the feature point being a thyroid nodule; Step 33: feature map P′ j For any feature point of (j=1, 2, 3, ..., 16), the credibility s of the prediction result of the feature point is calculated, and the credibility feature map of the prediction result is obtained according to the credibility s of the prediction results of all feature points. According to the calculation method of the prediction result credibility feature map, the feature sub-map N j (j=1, 2, 3, ..., 16) Calculate the prediction result credibility feature map Step 34: Use the prediction result credibility feature map And the prediction result credibility feature map Calculate the optimized sorting feature graph C′ j (j=1, 2, 3, ..., 16); Step 35: Optimize the sorting feature graph C′ j Arrange the eigenvalues ​​in from large to small, determine the coordinates of the first q feature points; according to the coordinates of the first q feature points determined, j The eigenvalues ​​of (j=1, 2, 3, ..., 16) are processed by weighted summation to output the feature subgraph M j (j=1, 2, 3, ..., 16); Step 36: Obtain feature sub-graph F′ based on the four-division feature graph j The coordinate parameters of (j=1, 2, 3, ..., 16) are used to transform the feature subgraph M j (j=1, 2, 3, ..., 16) are spliced ​​at different positions to obtain a feature map 7 with a channel number of 2 and a size of 128*128.

2. The thyroid nodule localization method based on a multi-level weighted neural network according to claim 1, characterized in that: The preprocessing of the thyroid ultrasound image includes converting the thyroid ultrasound image to a target size specification and normalizing the pixel values ​​of the converted image.

3. The thyroid nodule localization method based on a multi-level weighted neural network according to claim 2, characterized in that: The following steps are involved in building a thyroid nodule localization model based on the PyTorch deep learning framework: Step 1: Provide a target localization model based on the PyTorch deep learning framework and configure hyperparameters for training the target localization model; Step 2: Prepare a training data set and a test data set for training the target positioning model. When using the prepared data set to train the target positioning model, the FocalLoss loss function is used as the loss function. The Adam optimizer is used to optimize the parameters of the current target positioning model. After the training is terminated, a thyroid nodule positioning model based on the PyTorch deep learning framework is obtained.

4. The thyroid nodule localization method based on a multi-level weighted neural network according to claim 1, characterized in that: Compare the element values ​​in the two channels in the feature map 7, and obtain the feature map 8 after the comparison; The size of feature image eight is adjusted to the same size as the outer expansion rectangle of the contour, and the resized feature image eight is overlaid on feature image three to obtain feature image nine that characterizes the precise location indication features of the nodule. Feature image nine is used to depict the position and contour of the thyroid nodule on the identified and located thyroid ultrasound image.

5. The thyroid nodule localization method based on a multi-level weighted neural network according to claim 1, characterized in that: When feature map 4 is processed by the first nodule contour positioning sub-network to obtain feature map 5, the number of channels of feature map 5 is 2, wherein the first channel feature value of any feature point represents the probability value of the feature point being a non-thyroid nodule, and the second channel feature value of any feature point represents the probability value of the feature point being a thyroid nodule.

6. The thyroid nodule localization method based on a multi-level weighted neural network according to claim 1, characterized in that: In step 24, the prediction result credibility feature map P is used i c And the prediction result credibility feature map N i c , get the sorting feature map C i When , we have: in, Represents point-wise multiplication.

Images