Thyroid nodule automatic diagnosis method based on multi-modal deep learning

Abstract

The invention discloses a thyroid nodule automatic diagnosis method based on multi-modal deep learning. After ultrasonic image data, CT image data and clinical data of a patient are collected and preprocessed, spatial alignment of multi-modal data is achieved; extracting deep features from the multi-modal image data by using a convolutional neural network, performing image segmentation on a thyroid nodule region by using an improved ResNet structure and a U-Net structure, and extracting a region of interest; feature extraction and fusion are realized; constructing a deep neural network of a mixed mode, and training and optimizing a deep learning model; the model reasoning comprises the steps of preprocessing new input data, extracting and fusing features and generating a final diagnosis result. According to the invention, the automatic diagnosis of the thyroid nodules is realized by fusing the multi-modal data from different image sources and combining with the advanced deep learning model, so that the accuracy and efficiency of diagnosis are improved.

Description

Background Art

[0001] The present invention belongs to the technical field of medical image processing and computer-aided diagnosis, and in particular relates to an automatic diagnosis method for thyroid nodules based on multimodal deep learning.

[0003] With the rapid development of medical imaging technology, ultrasound imaging has become the main means of diagnosing thyroid nodules. However, traditional thyroid nodule diagnostic methods rely on subjective analysis of ultrasound images, and doctors interpret images based on experience, which may lead to certain risks of misdiagnosis and missed diagnosis. In addition, single-modality image data often cannot fully reflect the pathological characteristics of nodules, affecting the accuracy of diagnosis. In recent years, deep learning technology has made significant progress in the field of medical image analysis, especially in multimodal data fusion. Multimodal data refers to medical data from different image sources or different types. By fusing these data, it can provide richer feature information than single-modality data, thereby improving the accuracy and robustness of diagnosis. However, most of the existing deep learning-based thyroid nodule diagnostic methods only use single-modality ultrasound images, and fail to fully utilize the advantages of multimodal data. In addition, how to effectively fuse data from different modalities and extract features with diagnostic value from them remains a technical problem that needs to be solved urgently. Summary of the invention

[0004] In order to solve the problems of the prior art, the present invention proposes an automatic diagnosis method for thyroid nodules based on multimodal deep learning.

[0005] A method for automatic diagnosis of thyroid nodules based on multimodal deep learning, comprising the following steps:

[0006] Step 1: Multimodal data collection and preprocessing:

[0007] 1.1 Multimodal data acquisition includes the collection of ultrasound image data, CT image data and patient clinical data, and is stored in the DICOM standard format;

[0008] 1.2 Data standardization includes grayscale standardization and spatial resolution standardization to ensure the consistency of the format and scale of multimodal data;

[0009] 1.3 Realize spatial alignment of multimodal data through image registration method based on feature point matching;

[0010] Step 2: Feature extraction and fusion:

[0011] 2.1 Convolutional neural network is used to extract deep features from multimodal image data, and an improved ResNet structure is adopted;

[0012] 2.2 Use the U-Net structure to segment the thyroid nodule area and extract the region of interest;

[0013] 2.3 Feature fusion strategies include splicing fusion, weighted average fusion and attention mechanism fusion;

[0014] Step 3: Training and optimization of deep learning models:

[0015] 3.1 Construction of a mixed-modality deep neural network, including multiple convolutional layers, pooling layers, fully connected layers, and output layers;

[0016] 3.2 The training process of the model uses a gradient descent optimizer to iteratively update the parameters, and the loss function includes cross entropy loss and L2 regularization term;

[0017] 3.3 Model training and optimization;

[0018] 3.4 Model reasoning includes the preprocessing of new input data, feature extraction and fusion, and the generation of final diagnostic results.

[0019] The present invention collects and preprocesses ultrasound image data, CT image data and patient clinical data to achieve spatial alignment of multimodal data; uses convolutional neural networks to extract deep features from multimodal image data, and adopts improved ResNet structure and U-Net structure to perform image segmentation on thyroid nodule areas and extract regions of interest; achieves feature extraction and fusion; constructs a mixed-modality deep neural network to train and optimize deep learning models; combines model reasoning including preprocessing of new input data, feature extraction and fusion, and generation of final diagnostic results.

[0020] The present invention proposes an automatic diagnosis method for thyroid nodules based on multimodal deep learning, which aims to realize automatic diagnosis of thyroid nodules by fusing multimodal data from different imaging sources and combining advanced deep learning models, thereby improving the accuracy and efficiency of diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of the automatic diagnosis method of the present invention. DETAILED DESCRIPTION

[0022] The technical solution of the present invention is described in detail below with reference to the accompanying drawings:

[0023] like Figure 1 As shown, a method for automatic diagnosis of thyroid nodules based on multimodal deep learning is characterized by comprising the following steps:

[0024] Step 1: Multimodal data collection and preprocessing:

[0025] 1.1 Multimodal data acquisition includes the collection of ultrasound image data, CT image data and patient clinical data, and is stored in the DICOM standard format;

[0026] 1.2 Data standardization includes grayscale standardization and spatial resolution standardization to ensure the consistency of the format and scale of multimodal data;

[0027] 1.3 Realize spatial alignment of multimodal data through image registration method based on feature point matching;

[0028] Step 2: Feature extraction and fusion:

[0029] 2.1 Convolutional neural network is used to extract deep features from multimodal image data, and an improved ResNet structure is adopted;

[0030] 2.2 Use the U-Net structure to segment the thyroid nodule area and extract the region of interest;

[0031] 2.3 Feature fusion strategies include splicing fusion, weighted average fusion and attention mechanism fusion;

[0032] Step 3: Training and optimization of deep learning models:

[0033] 3.1 Construction of a mixed-modality deep neural network, including multiple convolutional layers, pooling layers, fully connected layers, and output layers;

[0034] 3.2 The training process of the model uses a gradient descent optimizer to iteratively update the parameters, and the loss function includes cross entropy loss and L2 regularization term;

[0035] 3.3 Model training and optimization;

[0036] 3.4 Model reasoning includes the preprocessing of new input data, feature extraction and fusion, and the generation of final diagnostic results.

[0037] Collecting multimodal data:

[0038] A multimodal diagnostic system is constructed by collecting multimodal data, including but not limited to the following modes of data:

[0039] (1) Ultrasound imaging data: As the main data source for the diagnosis of thyroid nodules, ultrasound imaging is non-invasive and highly real-time.

[0040] (2) CT image data: CT images have high spatial resolution and can provide detailed structural information of thyroid nodules.

[0041] (3) Patient clinical data: including the patient's medical history, blood test results, etc., which can provide auxiliary diagnostic information.

[0042] During the data collection process, it is necessary to ensure the diversity and representativeness of the data to cover different types of thyroid nodules (such as benign, malignant, cystic, etc.) and from different medical institutions and equipment (such as ultrasound equipment, CT scanners, etc.). The collected data should comply with the DICOM standard and be properly stored and managed for subsequent processing.

[0043] Perform data standardization. Since the data formats and scales of different devices and different modalities are inconsistent, it may affect the subsequent fusion processing. Therefore, it is necessary to standardize the collected multimodal data. The specific steps include:

[0044] (1) Grayscale standardization: Convert image data of different modalities into a unified grayscale range. Usually, a linear normalization method is used to map pixel values ​​to the range of [0, 1]. The formula is as follows:

[0045]

[0046] Among them, I(x,y) is the original pixel value, I max and I min are the minimum and maximum pixel values ​​of the image, respectively. For example, the pixel value range of an ultrasound image is [20,180], then I min =20,I max =180, for a pixel value I(x,y)=100, the value after normalization is:

[0047]

[0048] (2) Spatial resolution standardization: By using bilinear interpolation, all modal image data are adjusted to the same spatial resolution for subsequent image registration and fusion. Assume that the target resolution is R t =256*256 pixels, the original resolution is R o =128*128 pixels, the interpolation formula is:

[0049] I′(x,y)=(1-α)(1-β)I(x1,y1)+α(1-β)I(x2,y1)+(1-α)βI(x1,y2)+αβI(x2,y2)

[0050] Among them, α and β are interpolation weights, and x1, y1 and x2, y2 are neighboring pixels of the pixel to be interpolated. For example, assuming that in the original image, the pixel value of (x1, y1) = (50, 50) is 120, the pixel value of (x2, y1) = (51, 50) is 130, the pixel value of (x1, y2) = (50, 51) is 110, and the pixel value of (x2, y2) = (51, 51) is 140. For the interpolation position (x, y) = (50.5, 50.5), that is, α = β = 0.5, the pixel value after interpolation is:

[0051] I'(50.5,50.5)=0.25*120+0.25*130+0.25*110+0.25*140=125

[0052] For image registration and alignment operations, in order to ensure the spatial alignment (co-registration) of multimodal data, the present invention adopts an image registration method based on feature point matching. First, feature point detection is performed on images of different modalities. Commonly used detection algorithms include SIFT (scale-invariant feature transform) and ORB (rapid robust feature). For each image to be registered, the detector will find a set of feature points.

[0053] (1) SIFT feature point detection: Extreme point detection is performed based on the Gaussian difference pyramid. The first step is to construct a Gaussian pyramid:

[0054] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0055] Among them, G(x,y,σ) is the Gaussian kernel function, σ is the scale parameter, and * represents the convolution operation.

[0056] The next step is to calculate the Gaussian difference:

[0057] D(x,y,σ)=L(x,y,kσ-L(x,y,σ)

[0058] Where k is the scale factor. Detect extreme points in the Gaussian difference space and accurately locate feature points through interpolation. For example, assuming that the original image I(x,y) is 256×256 pixels, use the Gaussian kernel scale parameter σ=1.6 and the scale factor The calculated Gaussian difference image D(x,y,σ) detects feature points at multiple locations in the scale space. These feature points will be used for subsequent image registration.

[0059] (2) Feature point matching: A descriptor-based matching algorithm (such as Euclidean distance or Hamming distance) is used to match feature points of different modalities. The matching results are used to calculate the registration transformation matrix (usually affine transformation or rigid body transformation).

[0060] (3) Image alignment: According to the calculated transformation matrix, the image of one modality is transformed so that it is aligned with the image of the other modality. The affine transformation formula is as follows:

[0061]

[0062] Among them, a 11 ,a 12 ,a 21 ,a 22 are the rotation and scaling parameters, t x , t y is the translation parameter. The aligned image will be used for subsequent feature extraction and fusion, such as: assuming that the affine transformation matrix obtained by feature point matching is:

[0063]

[0064] Transform each pixel (x, y) of the original image to obtain the aligned pixel position (x', y').

[0065] The present invention selects convolutional neural network (CNN) to extract deep features from multimodal image data. CNN is widely used due to its excellent performance in image processing. The present invention adopts an improved ResNet (residual network) structure to extract features, which is characterized by:

[0066] (1) The improved ResNet structure includes multiple residual blocks, each of which consists of two 3×3 convolutional layers with an identity mapping connected in the middle. The formula of the residual block is expressed as:

[0067] y=F(x,{W i})+x

[0068] Where x is the input, {W i} is the convolution kernel weight, F(x,{W i For example, assuming that the input feature vector x is a 64-dimensional vector, the convolution kernel size of the residual block is 3×3, and the output after the convolution operation is F(x,{W i}) is also a 64-dimensional vector. At this time, the output y of the residual block is the input vector x and the convolution output F(x,{W i}) element-wise addition result.

[0069] (2) Feature extraction process: For each modality of data, local features are first extracted through the convolution layer, then downsampled through the pooling layer (such as maximum pooling or global average pooling), and finally mapped to a high-dimensional feature space through the fully connected layer. Assuming that the input image is I, the feature map after the lth convolution layer is expressed as:

[0070] F l (I) = ReLU(W l *F l-1 (I)+b l )

[0071] Among them, W l is the convolution kernel of the lth layer, b l is the bias and ReLU is the activation function. For example, assuming that the size of the input image is 256×256 pixels, the size of the feature map obtained after the first convolution layer is 128×128 pixels. If the size of the convolution kernel is 3×3 and the number of convolution kernels is 64, then the depth (number of channels) of the feature map is 64. After subsequent pooling and convolution operations, a high-dimensional feature vector is finally obtained.

[0072] The present invention selects to use the U-Net structure to perform image segmentation on the thyroid nodule area and extract the region of interest (ROI), which is characterized by:

[0073] (1) The U-Net structure is used for image segmentation. U-Net is a classic convolutional neural network structure, which is particularly suitable for medical image segmentation. The encoder part of U-Net consists of a series of convolution and pooling operations to extract deep features; the decoder part gradually restores the spatial resolution through deconvolution operations and finally outputs a segmentation map of the same size as the input image. The formula of U-Net is expressed as:

[0074] O(x,y)=Softmax(W decoder *F encoder (x,y)+b decoder )

[0075] Among them, O(x,y) is the output segmentation map, W decoder and b decoder are the convolution kernel and bias of the decoder part respectively. For example, assuming that the size of the input image is 512×512 pixels, after passing through the encoder and decoder of U-Net, the size of the output segmentation map is still 512×512 pixels. At this time, each pixel value in the segmentation map represents the probability that the pixel belongs to different categories (such as nodules, background).

[0076] (2) ROI extraction: The nodule region obtained by segmentation is the region of interest (ROI). The ROI will be further used for feature fusion and classification.

[0077] The feature fusion strategy selected by the present invention includes splicing fusion, weighted average fusion and attention mechanism fusion, which is characterized by:

[0078] (1) Concatenation Fusion: Concatenate the feature vectors of multiple modalities directly to form a higher-dimensional feature vector. Assuming that the feature vector from modality 1 is f1 and the feature vector from modality 2 is f2, the fused feature vector is expressed as

[0079] f fusion =[f1;f2]

[0080] Among them, [;] represents the vector concatenation operation. For example, if the dimension of the feature vector f1 of modality 1 is 256, and the dimension of the feature vector f2 of modality 2 is 128, then the concatenated fusion feature vector f fusion The dimension is 384.

[0081] (2) Weighted Average Fusion: The feature vectors of different modes are weighted averaged, and the weights are determined according to the importance of the mode or adaptive learning during model training. The fused features are expressed as:

[0082] f fusion =αf1+(1-α)f2

[0083] Among them, α is a weight parameter, which is determined through training. For example, if the weight α is 0.7, the dimension of the feature vector f1 of modality 1 is 256, and the dimension of the feature vector f2 of modality 2 is 256, then the weighted average fusion feature vector f fusion Each element of f fusion [i]=0.7*f1[i]+0.3*f2[i]

[0084] (3) Attention Fusion: The attention mechanism is introduced to dynamically assign weights according to the feature importance of each modality, thereby achieving more accurate feature fusion. Assuming that the attention weights are w1 and w2, the fused feature is expressed as

[0085] f fusion =w1⊙f1+w2⊙f2

[0086] Here, ⊙ represents element-wise multiplication. For example, the weights w1 and w2 calculated by the attention mechanism are 256-dimensional vectors, and the elements of these vectors float between [0,1]. The fused feature vector f fusion Each element of f fusion [i]=w1[i]⊙f1[i]+w2[i]⊙f2[i]. After the above feature fusion processing, the obtained composite features will be used as the input of the deep learning model for the classification and diagnosis of thyroid nodules.

[0087] The present invention selects a deep neural network based on mixed modality for automatic diagnosis of thyroid nodules. The network includes multiple convolutional layers, pooling layers, fully connected layers and output layers. The specific structure is as follows:

[0088] (1) Convolutional layers and pooling layers: The first few layers of the network are convolutional layers and pooling layers, which are used to further extract and compress features. Each convolutional layer is followed by an activation function (such as ReLU) and a pooling layer (such as max pooling) is used to reduce the size of the feature map. Assume that the output of the lth convolutional layer is f l ,but:

[0089] f l+1 =Pool(ReLU(W l *f l +b l ))

[0090] Pool represents the pooling operation. For example, suppose the input feature map size of the lth convolution layer is 64×64×128 (i.e., 64×64 spatial resolution, 128 channels), the convolution kernel size is 3×3, and the number of convolution kernels is 256. After the convolution operation, the size of the output feature map is 64×64×256. After the maximum pooling operation, the size of the feature map becomes 32×32×256.

[0091] (2) Fully connected layer: The outputs of the convolutional layer and the pooling layer are flattened and then input into the fully connected layer for feature mapping. The output of the fully connected layer is expressed as:

[0092] z=ReLU(W fc f flatten +b fc )

[0093] Among them, W fc and b fc are the weights and biases of the fully connected layer, respectively. For example, the dimension of the flattened input vector is 4096, and the output dimension of the fully connected layer is 1024. At this time, the weight matrix W fc The size is 1024×4096, and the bias vector b fc The dimension is 1024.

[0094] (3) Output layer: The final output layer uses the Softmax function to map the output of the fully connected layer to the probability distribution of the diagnosis result:

[0095]

[0096] in, is the predicted probability distribution vector. For example, assuming that the target category of the diagnostic classification is 3 (such as benign, malignant, and cystic), the dimension of the output layer is 3, and the Softmax function maps the output of the fully connected layer to a probability distribution in the range of [0,1]. Each element It represents the probability that the sample belongs to the i-th category.

[0097] The present invention selects a loss function based on cross entropy. Cross entropy loss can measure the difference between the model prediction result and the true label, and its formula is:

[0098]

[0099] Among them, C is the number of categories, y i is the one-hot encoding of the true label, is the probability distribution predicted by the model. For example, assuming the true label is a benign nodule, represented by y = [1, 0, 0], the probability distribution predicted by the model is At this time, the cross entropy loss is: In addition, in order to prevent the model from overfitting, the present invention also introduces an L2 regularization term, the formula of which is:

[0100]

[0101] Among them, λ is the regularization coefficient, is the L2 norm of the weight of the lth convolutional layer or fully connected layer. For example, the regularization coefficient λ = 0.01, the weight matrix dimension of the lth convolutional layer is 3×3×128×256 (i.e. 3×3 convolution kernel, 128 input channels, 256 output channels), and the L2 regularization term is 0.01 times the sum of the squares of all weight matrix elements. The final loss function is:

[0102]

[0103] The present invention selects the back propagation algorithm and uses a gradient descent optimizer (such as Adam or SGD) to iteratively update the model parameters. Assume that the loss function is The parameter update formula is:

[0104]

[0105] Where η is the learning rate, is the gradient of the loss function with respect to the parameter θ. For example, if the current learning rate η = 0.001, the gradient of the loss function with respect to the parameter θ is Then in one iteration, the parameter update amount is Δθ=0.00005. In order to speed up the training process, the present invention adopts the following optimization measures:

[0106] (1) Learning rate scheduling: The cosine annealing strategy is used to dynamically adjust the learning rate. The formula is:

[0107]

[0108] Among them, η min and η max are the minimum and maximum learning rates respectively, t is the current iteration number, and T is the total iteration number. max =0.01, minimum learning rate η min = 0.0001, total number of iterations T = 10000, current number of iterations t = 5000, then the current learning rate is: η 5000 =0.00505

[0109] (2) Batch Normalization: Add a BatchNormalization layer after each convolutional layer and fully connected layer to accelerate model convergence and prevent overfitting. The formula for Batch Normalization is:

[0110]

[0111] Among them, μ and σ 2 are the mean and variance of the small batch data, respectively, and ε is a constant to prevent division by zero. For example, if the mean of the small batch data is μ=0.5 and the variance is σ 2 =0.04, then after Batch Normalization processing, the standardized data is:

[0112] (3) Dropout: Add a Dropout operation to the fully connected layer to randomly discard some neurons to prevent overfitting. The formula for Dropout is:

[0113] z dropout =z⊙r

[0114] Where r is a binary random vector, and ⊙ represents element multiplication. If z is a 1024-dimensional vector and the Dropout probability is 0.5, each element in r has a 50% probability of being 0 and a 50% probability of being 1. The vector z after Dropout dropout In , each element is the product of the corresponding element of z and r.

[0115] The model inference process diagnoses the newly input multimodal data. The specific process is:

[0116] (1) Data preprocessing: standardize, align and enhance the newly input data.

[0117] (2) Feature extraction and fusion: Use the trained deep neural network to extract features from multimodal data and fuse the extracted features.

[0118] (3) Generation of diagnostic results: The fused features are input into the trained classification model to output the predicted diagnostic results. The final diagnostic results can be directly used for clinical decision-making to assist doctors in diagnosing thyroid nodules.

[0119] The present invention proposes an automatic diagnosis method for thyroid nodules based on multimodal deep learning, which aims to realize automatic diagnosis of thyroid nodules by fusing multimodal data from different imaging sources and combining advanced deep learning models, thereby improving the accuracy and efficiency of diagnosis.

Claims

1. A method for automatic diagnosis of thyroid nodules based on multimodal deep learning, characterized by comprising the following steps: Step 1: Multimodal data collection and preprocessing: 1.1 Multimodal data acquisition includes the collection of ultrasound image data, CT image data and patient clinical data, and is stored in the DICOM standard format; 1.2 Data standardization includes grayscale standardization and spatial resolution standardization to ensure the consistency of the format and scale of multimodal data; 1.3 Realize spatial alignment of multimodal data through image registration method based on feature point matching; Step 2: Feature extraction and fusion: 2.1 Convolutional neural network is used to extract deep features from multimodal imaging data, and an improved ResNet structure is adopted; 2.2 Use the U-Net structure to segment the thyroid nodule area and extract the region of interest; 2.3 Feature fusion strategies include splicing fusion, weighted average fusion and attention mechanism fusion; Step 3: Training and optimization of deep learning models: 3.1 Construction of a mixed-modality deep neural network, including multiple convolutional layers, pooling layers, fully connected layers, and output layers; 3.2 The training process of the model uses a gradient descent optimizer to iteratively update the parameters, and the loss function includes cross entropy loss and L2 regularization term; 3.3 Model training and optimization; 3.4 Model reasoning includes the preprocessing of new input data, feature extraction and fusion, and the generation of final diagnostic results.

2. The method for automatic diagnosis of thyroid nodules based on multimodal deep learning according to claim 1, characterized in that: A multimodal diagnostic system is constructed by collecting multimodal data, including but not limited to the following modes of data: Ultrasound imaging data: As the main data source for thyroid nodule diagnosis, ultrasound imaging is non-invasive and highly real-time; CT imaging data: CT images have high spatial resolution and can provide detailed structural information of thyroid nodules; Patient clinical data: including the patient's medical history, blood test results, etc., which can provide auxiliary diagnosis information; During the data collection process, it is necessary to ensure the diversity and representativeness of the data to cover different types of thyroid nodules and come from different medical institutions and equipment; the collected data should comply with the DICOM standard and be properly stored and managed for data standardization; The specific steps include: (1) Grayscale standardization: Convert image data of different modalities into a unified grayscale range. Usually, a linear normalization method is used to map pixel values ​​to the range of [0, 1]. The formula is as follows: Among them, I(x,y) is the original pixel value, I max and I min are the minimum and maximum pixel values ​​of the image, respectively. For example, the pixel value range of an ultrasound image is [20,180], then I min =20,I max =180, for a pixel value I(x,y)=100, the value after normalization is: (2) Spatial resolution standardization: By using bilinear interpolation, all modal image data are adjusted to the same spatial resolution for subsequent image registration and fusion. Assume that the target resolution is R t =256*256 pixels, the original resolution is R o =128*128 pixels, the interpolation formula is: I′(x,y)=(1-α)(1-β)I(x1,y1)+α(1-β)I(x2,y1)+(1-α)βI(x1,y2)+αβI(x2,y2) Among them, α and β are interpolation weights, and x1, y1 and x2, y2 are neighboring pixels of the pixel to be interpolated. For example, assuming that in the original image, the pixel value of (x1, y1) = (50, 50) is 120, the pixel value of (x2, y1) = (51, 50) is 130, the pixel value of (x1, y2) = (50, 51) is 110, and the pixel value of (x2, y2) = (51, 51) is 140. For the interpolation position (x, y) = (50.5, 50.5), that is, α = β = 0.5, the pixel value after interpolation is: I'(50.5,50.5)＝0.25*120+0.25*130+0.25*110+0.25*140＝125。 3. The method for automatic diagnosis of thyroid nodules based on multimodal deep learning according to claim 2, characterized in that: To ensure the spatial alignment of multimodal data, an image registration method based on feature point matching is adopted. First, feature point detection is performed on images of different modalities. Commonly used detection algorithms include scale-invariant feature transformation and fast robust feature. For each image to be registered, the detector finds a set of feature points; (1) Scale-invariant feature transform (SIFT) feature point detection: Extreme point detection is performed based on the Gaussian difference pyramid. The first step is to construct a Gaussian pyramid: L(x,y,σ)=G(x,y,σ)*I(x,y) Among them, G(x,y,σ) is the Gaussian kernel function, σ is the scale parameter, and * represents the convolution operation; Compute the difference of Gaussians: D(x,y,σ)=L(x,y,kσ)-L(x,y,σ) Where k is the scale factor; detect extreme points in the Gaussian difference space and accurately locate feature points by interpolation; assume that the original image I(x, y) is 256×256 pixels, use the Gaussian kernel scale parameter σ=1.6 and the scale factor The calculated Gaussian difference image D(x, y, σ) detects feature points at multiple locations in the scale space; these feature points will be used for subsequent image registration; (2) Feature point matching: A descriptor-based matching algorithm is used to match feature points of different modalities. The matching results are used to calculate the registration transformation matrix. (3) Image alignment: According to the calculated transformation matrix, the image of one modality is transformed so that it is aligned with the image of the other modality. The affine transformation formula is as follows: Among them, a 11 ,a 12 ,a 21 ,a 22 are the rotation and scaling parameters, t x , t y is the translation parameter; the aligned image will be used for subsequent feature extraction and fusion, and each pixel (x, y) of the original image is transformed to obtain the aligned pixel position (x', y').

4. The method for automatic diagnosis of thyroid nodules based on multimodal deep learning according to claim 3, characterized in that: Convolutional neural network is selected to extract deep features from multimodal image data, and an improved residual network structure is used to extract features. (1) The improved residual network structure includes multiple residual blocks. Each residual block consists of two 3×3 convolutional layers with an identity mapping connected in the middle. The formula of the residual block is expressed as: y=F(x,{W i })+x Where x is the input, {W i } is the convolution kernel weight, F(x,{W i }) is the output of the convolutional layer, and y is the output of the residual block; (2) Feature extraction process: For each modality of data, local features are first extracted through the convolution layer, then downsampled through the pooling layer, and finally mapped to a high-dimensional feature space through the fully connected layer; the input image is I, and the feature map after the lth convolution layer is expressed as: F l (I)=ReLU(W l *F l-1 (I)+b l ) Among them, W l is the convolution kernel of the lth layer, b l is the bias and ReLU is the activation function. After subsequent pooling and convolution operations, a high-dimensional feature vector is finally obtained.

5. The method for automatic diagnosis of thyroid nodules based on multimodal deep learning according to claim 4, characterized in that: Use the U-Net structure to segment the thyroid nodule area and extract the region of interest. (1) The U-Net structure is used for image segmentation. The formula of U-Net is expressed as: O(x,y)=Softmax(W decoder *F encoder (x,y)+b decoder ) Among them, O(x,y) is the output segmentation map, W decoder and b decoder They are the convolution kernel and bias of the decoder part respectively; (2) Region of interest extraction: The nodule region obtained by segmentation is the region of interest, which is further used for feature fusion and classification.

6. The method for automatic diagnosis of thyroid nodules based on multimodal deep learning according to claim 5, characterized in that: The feature fusion strategies include concatenation fusion, weighted average fusion and attention mechanism fusion. (1) Concatenation and fusion: The feature vectors of multiple modalities are directly concatenated to form a higher-dimensional feature vector. Let the feature vector from modality 1 be f1 and the feature vector from modality 2 be f2. The fused feature vector is expressed as f fusion =[f1;f2] Among them, [;] represents the vector concatenation operation; (2) Weighted average fusion: The feature vectors of different modalities are weighted averaged, and the weights are determined according to the importance of the modality or adaptive learning during model training. The fused features are expressed as: f fusion =αf1+(1-α)f2 Among them, α is the weight parameter, which is determined through training; (3) Attention mechanism fusion: The attention mechanism is introduced to dynamically assign weights according to the feature importance of each modality, thereby achieving more accurate feature fusion; let the attention weights be w1 and w2, then the fused feature is expressed as f fusion =w1⊙f1+w2⊙f2 Among them, ⊙ represents element multiplication. After the above feature fusion processing, the obtained composite features will be used as the input of the deep learning model for the classification and diagnosis of thyroid nodules.

7. The method for automatic diagnosis of thyroid nodules based on multimodal deep learning according to claim 6, characterized in that A mixed-modality deep neural network is selected for automatic diagnosis of thyroid nodules; it includes multiple convolutional layers, pooling layers, fully connected layers, and output layers. The specific structure is as follows: (1) Convolutional layers and pooling layers: The first few layers of the network are convolutional layers and pooling layers, which are used to further extract and compress features. Each convolutional layer is followed by an activation function, and the size of the feature map is reduced through the pooling layer. Let the output of the lth convolutional layer be f l ,but: f l+1 =Pool(ReLU(W l *f l +b l )) Among them, Pool represents the pooling operation; (2) Fully connected layer: The outputs of the convolutional layer and the pooling layer are flattened and then input into the fully connected layer for feature mapping. The output of the fully connected layer is expressed as: z=ReLU(W fc f flatten +b fc ) Among them, W fc and b fc are the weights and biases of the fully connected layer respectively; (3) Output layer: The final output layer uses the Softmax function to map the output of the fully connected layer to the probability distribution of the diagnosis result: in, is the predicted probability distribution vector.

8. The method for automatic diagnosis of thyroid nodules based on multimodal deep learning according to claim 7, characterized in that Select a loss function based on cross entropy. The cross entropy loss can measure the difference between the model prediction results and the true label. The formula is: Among them, C is the number of categories, y i is the one-hot encoding of the true label, is the probability distribution predicted by the model; In order to prevent the model from overfitting, the L2 regularization term is introduced, and its formula is: Among them, λ is the regularization coefficient, is the L2 norm of the weights of the lth convolutional layer or fully connected layer.

9. The method for automatic diagnosis of thyroid nodules based on multimodal deep learning according to claim 8, characterized in that Select the back propagation algorithm and use the gradient descent optimizer to iteratively update the model parameters. Let the loss function be The parameter update formula is: Where η is the learning rate, is the gradient of the loss function with respect to the parameter θ; In order to speed up the training process, the following optimization measures are taken: (1) Learning rate scheduling: The cosine annealing strategy is used to dynamically adjust the learning rate. The formula is: Among them, η min and η max are the minimum and maximum learning rates respectively, t is the current iteration number, and T is the total iteration number; (2) Batch Normalization: Add a Batch Normalization layer after each convolutional layer and fully connected layer to accelerate model convergence and prevent overfitting; the formula for Batch Normalization is: Among them, μ and σ 2 are the mean and variance of the small batch data respectively, and ε is a constant to prevent division by zero; (3) Dropout: Add a Dropout operation to the fully connected layer to randomly discard some neurons to prevent overfitting; the formula for Dropout is: With dropout =z⊙r Where r is a binary random vector and ⊙ represents element-wise multiplication.

10. The method for automatic diagnosis of thyroid nodules based on multimodal deep learning according to claim 9, characterized in that: The model inference process diagnoses the newly input multimodal data. (1) Data preprocessing: standardize, align and enhance the newly input data; (2) Feature extraction and fusion: Use the trained deep neural network to extract features from multimodal data and fuse the extracted features; (3) Generation of diagnostic results: The fused features are input into the trained classification model and the predicted diagnostic results are output.

Images