Thyroid nodule image classification method based on ultrasound and infrared multi-modal images

Abstract

The application discloses a thyroid nodule image classification method based on ultrasound and infrared thermal image multimodal images. The application is based on ultrasound and infrared thermal images, combined with a multimodal learning method, and provides a self-adaptive multimodal hybrid model, which is composed of an intra-modal mixed encoder, a self-adaptive cross-modal encoder and a classification head. The intra-modal mixed encoder can model global features while extracting local features; the self-adaptive cross-modal encoder can formulate personalized modal weights according to different cases, and simultaneously perform information interaction and fusion of inter-modal features; and the classification head classifies the obtained fusion features. The method automatically classifies the thyroid nodule images of a subject based on ultrasound and infrared thermal images of the subject by using an AmmH model, and provides objective and accurate classification results for doctors, so that auxiliary diagnosis is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the automatic screening, analysis and classification of thyroid nodules in the field of deep learning and medicine, in particular to a thyroid nodule classification method based on ultrasound (US) and infrared thermal (IRT) multi-modal images. BACKGROUND

[0002] The thyroid is located below the front of the neck in the human body, and is one of the important glands in the human body. The thyroid hormone produced by the thyroid gland controls the metabolism of the human body. The thyroid hormone not only affects the cholesterol level and heart rate of the human body, but also affects the mental state of the human body. Normal thyroid function is the premise for the normal operation of every organ in the body. In recent years, the incidence of thyroid cancer has been increasing year by year. The initial symptoms of thyroid cancer are thyroid nodules in the neck, which can usually be divided into benign nodules (non-cancerous nodules) and malignant nodules (which can lead to thyroid cancer) according to their characteristics. According to epidemiological data, the incidence of thyroid nodules in the population is 19%-68%, but only 5%-15% are malignant. For benign cases, fine needle aspiration biopsy or surgery does not need to be performed, and accurate benign and malignant analysis of the nodules can not only reduce the risk to the patient, but also significantly reduce the medical cost.

[0003] The incidence of thyroid nodules in adults reaches 67%, and for most of them, the detection technology of thyroid nodules is best non-invasive. In the current diagnosis of thyroid nodules, US detection is the most widely used diagnostic method for clinical diagnosis of thyroid disease due to its non-radiation, non-invasive, good repeatability, low cost and other advantages. At present, many researchers have used US images to classify thyroid nodules based on deep learning methods. Soon et al. adopted the method of transfer learning using a pre-trained deep learning model VGG16 to classify thyroid nodules in US images, and Qing et al. applied the deep learning algorithm Inception-v3 to distinguish the benign and malignant thyroid nodules. These methods have achieved certain results in the classification of thyroid nodules. However, these classification methods of thyroid nodules are only based on US imaging modalities. In actual clinical diagnosis, doctors often use multiple modalities of medical data, such as clinical text data, image waveform data, and biological data, to make more comprehensive and accurate diagnoses. In recent years, another IRT imaging technology with the same characteristics of non-radiation, non-invasive, and good repeatability has also been used by researchers for the detection of thyroid diseases. Helmy et al. used a cube as a neck model and used a surface numerical heat map to build a prototype system, verifying the potential of thermal imaging in thyroid tumor detection. Bahramian et al. of KN Toosi University of Technology in Tehran used an artificial neural network (ANN) to analyze the neck heat map to distinguish between healthy and cancerous thyroid tissue. Fan Xuefeng et al. of Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, conducted a study on the multifractal characteristics of dynamic infrared images of healthy thyroid glands under constant temperature and humidity conditions and statistically analyzed and tested the differences in multifractal characteristic parameters of different individuals' thyroid glands. The results showed that the temperature fluctuations of healthy individuals' thyroid glands had certain differences but the amplitude was small, fluctuating within 0.15℃. These studies prove that IRT imaging technology has certain potential in the detection of thyroid diseases. US images and IRT images can reflect the information of the lesion area in the thyroid from different angles. US images can reflect the size, boundary, shape, aspect ratio, echo, calcification, and internal structure of thyroid nodules, while IRT images can reflect the temperature changes caused by early metabolic changes and blood circulation changes in the lesion area. In multi-modal learning tasks, when one modality's data is disturbed by noise, the information provided by other modalities can help correct the noise and synchronize the consistent information in different modalities, improving the accuracy and robustness of decision-making. In addition, there are certain differences between data collected by different instruments in different modalities, that is, each modality can provide certain information for the remaining modalities while also containing information that other modalities do not have.The observation advantage of integrating different modal data can achieve more comprehensive semantic expression and improve the understanding depth of the model to the target task. Therefore, it is necessary to comprehensively utilize the information from US images and IRT images based on multi-modal learning method to achieve more accurate classification of thyroid nodules. In this task, we face the following challenges: first, how to make the model have more sufficient feature expression ability to obtain high-level abstract feature expression within the modal; second, how to promote the information interaction and reasonable feature fusion of abstract features between modalities.

[0004] Deep learning method promotes the development of multi-modal learning in medical field. Deep learning method obtains higher abstract feature expression ability through layer-by-layer screening and recombination of low-dimensional features. By combining high-level abstract features of multiple different modalities, it greatly promotes the learning and fusion between modalities. Given the powerful feature expression and fusion performance of deep learning method, it has become an important means to solve the increasingly complex multi-modal learning of heterogeneous data. Since the deep learning revival, Convolutional Neural Networks (CNN) has always dominated the field of computer vision. CNN can preserve the neighborhood relationship and spatial local characteristics in images, and through three strategies of local receptive field, weight sharing and down-sampling, it reduces the complexity of network model, and has invariance to translation, rotation, scale and other forms of variation. However, because of this, CNN lacks the ability to model global information in images. In the past two years, deep learning models based on Transformer have also been applied to the field of computer vision and achieved good results. The use of self-attention (SA) mechanism in Transformer enables the model to learn long-range dependencies and better model global information in images. A key factor affecting the ability to learn such dependencies is the path length that the forward and backward signals must pass through in the network. The shorter these paths between any combination of positions in the input and output sequences, the easier it is to learn long-range dependencies. In CNN-based models, the number of operations required to calculate the association between two positions through convolution increases with distance, while the number of operations required to calculate the association between two positions through SA is independent of distance. Transformer-based models lack the inductive bias of CNN, such as translation invariance and locality, which makes the model lose the ability to capture local features. If we can combine the advantages of local feature extraction in CNN and global modeling in Transformer, the model will have more powerful feature representation ability and obtain more rich high-level feature expression within the modal.

[0005] In the multi-modal learning method, how to fuse the high-level semantic features from different modalities has always been the focus of researchers. The fusion method can be mainly divided into three categories: input-level fusion, feature-level fusion and decision-level fusion. The most common fusion strategy is to directly concatenate the features from different modalities. Recently, researchers have also proposed other strategies for feature fusion, such as using Kronecker product and orthogonal loss strategies. However, these fusion strategies lack information interaction between features from different modalities.

[0006] In addition, in the recognition and classification of thyroid nodule images, an ideal multi-modal model should consider the individual differences between different cases and develop a personalized modal weight strategy according to the specific patient case. Currently, there is no similar method disclosed. SUMMARY

[0007] The present application aims to solve the problem of using only a single modality in the current deep learning-based thyroid nodule recognition and classification method, and proposes a thyroid nodule classification method based on US and IRT multi-modal images. In this method, a hybrid encoder is constructed to fully extract intra-modal features by combining the ability of CNN in local feature expression and the advantage of Transformer in global information modeling. In order to achieve reasonable and efficient feature fusion, an adaptive cross-modal encoder is designed, which can interact information between high-level semantic features from different modalities and develop a personalized modal weight according to different cases. In general, this method can effectively extract features related to thyroid nodules from US images and IRT images, and reasonably fuse features from the two different modalities, so that the final features can fully represent the information related to thyroid nodule diagnosis, realize more accurate classification of thyroid nodules, and provide more objective and accurate image classification reference results for doctors to assist in diagnosis and improve the efficiency and accuracy of diagnosis.

[0008] The technical solutions provided by the present application are as follows:

[0009] A thyroid nodule image classification method based on ultrasound and infrared thermal image multi-modal images, characterized in that it comprises the following steps:

[0010] S1: Prepare a thyroid multi-modal dataset;

[0011] S2: Construct two hybrid intra-modal encoders (HIME) to extract features from IRT images and US images, respectively. The internal structures of the two hybrid encoders are consistent.

[0012] S3: Construct an adaptive cross-modal encoder (ACME) to adaptively fuse the features of the IRT images and the features of the US images obtained in S2;

[0013] S4: Construct a classification head based on a multilayer perceptron (MLP) to classify the fused features related to thyroid nodule diagnosis obtained in S3;

[0014] S5: Combine the two HIMEs constructed in S2, the ACME constructed in S3, and the MLP Head constructed in S4 into an adaptive multi-modal hybrid (AmmH) model;

[0015] S6: Divide the thyroid multi-modal dataset into a training set, a validation set, and a test set according to a certain proportion, define a loss function and an optimizer, and define a learning rate reduction method;

[0016] S7: Based on the backpropagation algorithm, perform multiple iterations of end-to-end training on the AmmH model, take the model parameter value with the highest classification accuracy on the test set as the final AmmH model parameter value for thyroid nodule image classification;

[0017] S8: Preprocess the thyroid multi-modal data (i.e., paired IRT images and US images of the thyroid) as input to the AmmH model, and output the classification results of the thyroid nodule images.

[0018] Further, in step S1, the thyroid multi-modal dataset is constructed by the following steps:

[0019] S1.1: Collect thyroid IRT images and US images of the subject, and obtain thyroid nodule diagnosis results;

[0020] S1.2: Preprocess the collected thyroid IRT images and US images, and resize them to a consistent size;

[0021] S1.3: Pair the preprocessed IRT images and US images in S1.2 as samples, and use the thyroid nodule diagnosis results in S1.1 as sample labels to obtain the thyroid multi-modal dataset.

[0022] Further, in step S2, the mixed-modal intra-encoder HIME is constructed by the following steps:

[0023] S2.1: Construct an Intra-modal Convolutional Encoder (Intra-modal CE) to extract features of a single modality, and the obtained intra-modal local feature representation is as follows:

[0024]

[0025] where m represents the imaging modality, m ∈ {IRT, US}; represents the intra-modal local feature of modality m; represents the intra-modal convolutional encoder corresponding to modality m; m represents the modality m image; m ∈RC*H*W; C, H, W represent the channel number, height, and width of the modality m image; C', H', W' represent the channel number, height, and width of the feature map of the modality m local feature;

[0026] S2.2: Construct a Feature Embedding Layer (FEL), and the purpose of the FEL is to bridge the convolutional encoder and the Transformer encoder. First, the feature map obtained in S2.1 is deconvolved to obtain the required number and size; then the two-dimensional feature map is flattened to obtain a one-dimensional Token sequence required by the Transformer encoder input; the flattening operation loses spatial information, which is very important for images, so position encoding (PE) is added to the obtained Token sequence to obtain the final one-dimensional Token sequence representation as follows:

[0027]

[0028] where, represents the one-dimensional Token sequence processed by the FEL as the input of the Transformer encoder; Flatten represents the flattening operation; Deconv m represents the deconvolution operation of modality m; PE m represents the position encoding of modality m; C", H", W" represent the number and dimension of the one-dimensional Token sequence corresponding to modality m, respectively;

[0029] S2.3: Construct an Intra-modal Transformer Encoder (Intra-modal TE) to model the global context of the local features processed by the FEL in S2.2 and establish long-distance dependencies. The Intra-modal TE is composed of L1 Intra-modal Transformer blocks, each of which is composed of a Layer Normalization (LN) layer, an SA layer, an LN layer, and an MLP layer. The MLP layer is composed of two fully connected layers with Gaussian Error Linear Units (GELU) activation functions.

[0030] The high-level abstract semantic feature representation with global context information and local information obtained by the IMTE is as follows:

[0031]

[0032] where F m represents the high-level abstract semantic feature of the modality m obtained by the hybrid intra-modal encoder HIME; MLP m represents a multi-layer perceptron layer; Z m is an intermediate variable; SA m (·) is self-attention calculation; F m ∈R C″*H″w″ .

[0033] Further, in step S2.3, the self-attention calculation formula in the SA layer is as follows:

[0034]

[0035]

[0036]

[0037]

[0038] where Q m , K m , and V m represent the query vector, key vector, and value vector required for self-attention calculation, respectively; are all learnable parameters; LN(·) represents layer normalization; T represents the transpose of a matrix; d k represents the dimension of the query vector K m ; and the calculation formula of softmax(·) is where e is a natural constant and k represents the dimension of vector x.

[0039] Further, the constructing the adaptive cross-modal encoder ACME in step S3 specifically comprises the following steps:

[0040] S3.1: An adaptive modality-weight generation (AMWG) network is constructed for customizing different modality weights for each case according to the actual case situation; the AMWG module is a multi-layer perceptron (MLP) composed of three fully connected layers with rectified linear unit (ReLU) activation functions. In order to reduce repeated calculation and accelerate convergence, the input is the high-level abstract semantic features F m of each modality m obtained in step S2.3, and the output is the customized optimal weight w m of each modality;

[0041] The AMWG operation process is as follows:

[0042] (w US , w IRT ) = softmax(MLP AMWG ([F Us , F IRT ])

[0043] where w US , w IRT represent the optimal modality weights corresponding to the modalities US and IRT, F US、 F IRT represent the high-level abstract semantic features of the modalities US and IRT, respectively, [·, ·] represents the splicing operation, softmax(·) operation makes w US + w IRr = 1, and MLP AWMG表 is a multi-layer perceptron composed of three fully connected layers with rectified linear unit (ReLU) activation functions;

[0044] S3.2: An inter-modal transformer encoder (Inter-modal TE) is constructed for information interaction and adaptive feature fusion of the customized modality weights obtained in step S3.1 and the features from different modalities obtained in step S2, and the steps are as follows:

[0045] First, the learnable parameter vector cls Token is introduced as the last classification feature, and the position encoding PE is added as the input of the Inter-modal TE, which is represented as follows:

[0046] F token = [cls token, w US * F Us + w IRT * F IRT] + PE

[0047] where F token ∈ R (c″+1)*H″W″ ;

[0048] The Inter-modal TE is composed of L2 Inter-modal blocks, and each Inter-modal block is composed of LN layer, multi-head self-attention (MSA) layer, LN layer and MLP layer, and the MLP layer is composed of two fully connected layers with Gaussian error linear unit (GELU) activation function.

[0049] Further, in step S3.2, the self-attention calculation formula in the MSA layer is as follows:

[0050]

[0051]

[0052]

[0053]

[0054] MSA = W o [head 1 , head 2 ... head N ]

[0055] F = MLP (LN (Z)) + Z, Z = MSA (LN (F token ) ) + F token

[0056] where Q i , K i , V i represent the query vector, key vector and value vector required for self-attention calculation of the i-th head, respectively; W o are all learnable parameters; LN(·) represents layer normalization; head i , SAi represents the self-attention calculation result of the i-th head; represents the query vector K i dimension; F is in R (c″+1)*H″w″ .

[0057] Further, the constructing the MLP-based classification head (MLP Head) in step S4 comprises the following steps:

[0058] S4.1: taking the value F[0] of the first position in the feature vector F obtained in step S3 as the input of the classification head MLPHead;

[0059] S4.2: constructing the MLP Head, which is a fully connected layer with a ReLU activation function; inputting F[0], and outputting a classification result, represented as follows:

[0060] cls=MLP Head(F[0])

[0061] wherein cls is the final obtained classification category, and cls is in {benign, malignant}.

[0062] Further, the loss function in step S6 is a cross entropy loss function (CE Loss), which is defined as follows:

[0063] CE Loss=-[y log(y′)+(1-y)log(1-y′)]

[0064] wherein y represents the real label of the sample, and y' is the predicted label.

[0065] Further, in step S6, the optimizer is used to update and calculate the network parameters that affect the model training and the model output, so as to approach or reach the optimal value, thereby minimizing or maximizing the loss function; and the learning rate decreasing mode refers to that the learning rate decreases with the increase of the training iteration number, so as to help the model converge.

[0066] Further, the preprocessing mode in step S8 is the same as the preprocessing mode mentioned in S1.2.

[0067] The beneficial effects of the present application are as follows:

[0068] (1) The present application proposes a thyroid nodule classification method based on US and IRT images, which uses two imaging technologies, US imaging technology and IRT imaging technology, to obtain US images and IRT images of the thyroid part of the patient. A multi-modal learning model is built to jointly learn the features related to thyroid nodule diagnosis in the US images and IRT images, which has a higher accuracy in thyroid nodule classification compared with single modal methods.

[0069] (2) The application uses a hybrid intra-modal encoder HIME when extracting features of a single modal image, which bridges Intra-modal CE and Intra-modal TE through FEL. In this way, HIME combines the advantages of CNN in extracting local features of images due to translational invariance and other inductive biases, and the global modeling ability of Transformer based on SA to establish long-distance dependencies, so that HIME has stronger feature extraction capability.

[0070] (3) The application designs an adaptive cross-modal encoder ACME when fusing features from two different modalities. ACME is composed of an AMWG network and an Inter-modal TE. The AMWG network fully considers the differences between individuals and gives customized personalized modal weights according to the actual situation of different cases. The introduction of Inter-modal TE further realizes the information interaction between the two modal features, which makes ACME more reasonable and efficient when fusing the two modal features. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 is the workflow diagram of the thyroid nodule image classification based on multi-modal images in the application;

[0072] Figure 2 is the flowchart of data acquisition in the application;

[0073] Figure 3 is an example of benign and malignant case data in the thyroid multi-modal data set made in the application;

[0074] Figure 4 is a model overview diagram of the adaptive multi-modal hybrid model AmmH in the application;

[0075] Figure 5 is a comparison diagram of the Intra-modal Transformer block in the hybrid intra-modal encoder HIME and the Inter-modal Transformer block in the adaptive cross-modal encoder ACME in the application. DETAILED DESCRIPTION

[0076] The technical solutions in the embodiments of the application will be described clearly and completely below in combination with the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments of the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0077] Embodiment

[0078] The application provides a thyroid nodule image classification method based on US and IRT images, and a workflow diagram is as shown in Figure 1 The method comprises the following steps:

[0079] S1: preparing a thyroid multi-modal dataset. A thyroid data acquisition flowchart is as shown in Figure 2 The benign and malignant case data in the thyroid multi-modal dataset is as shown in Figure 3

[0080] S2: constructing two hybrid modal intra-encoders HIME, respectively extracting features of IRT images and US images, and the internal structures of the two hybrid encoders are consistent.

[0081] S3: constructing an adaptive cross-modal encoder ACME to adaptively fuse the features of the IRT images and the features of the US images obtained in S2.

[0082] S4: constructing a classification head MLP Head based on a multi-layer perception MLP to classify the fused features related to thyroid nodule diagnosis obtained in S3.

[0083] S5: combining the two HIME constructed in S2, the ACME constructed in S3 and the MLP Head constructed in S4 into an adaptive multi-modal hybrid AmmH model. An overview diagram of the AmmH model is as shown in Figure 4

[0084] S6: dividing the thyroid multi-modal dataset into a training set, a validation set and a test set according to a certain proportion, defining a loss function and an optimizer, and defining a learning rate reduction method. Specifically, in the embodiment, the division ratio of the multi-modal dataset is training set: validation set: test set = 6:2:2.

[0085] S7: based on a back propagation algorithm, performing end-to-end multiple iteration training on the AmmH model, taking the model parameter value with the highest classification accuracy on the test set as the final AmmH model parameter value used for thyroid nodule image classification. Specifically, in the embodiment, the total number of training rounds is set to 200, and the training batch size is set to 32.

[0086] S8: taking the preprocessed thyroid multi-modal data (i.e., paired data of IRT images and US images of the thyroid) as the input of the AmmH model, and outputting the thyroid nodule image classification result. Specifically, in the embodiment, the classification result of the thyroid nodule is divided into benign and malignant two categories.

[0087] Further, the step S1 of preparing the thyroid multi-modal dataset specifically comprises the following steps:​​

[0088] S1.1: Collecting the IRT image and the US image of the thyroid of the subject, and obtaining the diagnosis result of the thyroid nodule.

[0089] S1.2: Preprocessing the collected IRT image and US image of the thyroid, and resizing to consistent size. Specifically, in the embodiment, the size of the IRT image and the US image is uniformly resized to 224*224.

[0090] S1.3: Pairing the IRT image and the US image preprocessed in S1.2 as a sample, and taking the diagnosis result of the thyroid nodule in S1.1 as a sample label, to obtain a thyroid multi-modal dataset. Specifically, in the embodiment, the sample label includes two kinds: benign and malignant.

[0091] Further, the step S2 of constructing the hybrid intra-modal encoder HIME specifically includes the following steps:

[0092] S2.1: Constructing an intra-modal convolutional encoder Intra-modal CE to extract features of a single modality, and obtaining the intra-modal local feature representation as follows:

[0093]

[0094] Wherein m represents the imaging modality, m∈{IRT, US}; represents the intra-modal local feature of modality m; represents the intra-modal convolutional encoder corresponding to modality m; I m represents the modality m image; I m ∈R C*H*W ; C, H, W represent the channel number, height, and width of the modality m image; C', H', W' represent the channel number, height, and width of the feature map of the modality m local feature. Specifically, in the embodiment, C=3, H=224, W=224; C'=512, H'=8, W'=8.

[0095] Specifically, the convolutional encoder in the embodiment adopts a network structure basically the same as the residual network model ResNet18, and the only difference is that the global average pooling layer and the fully connected layer in ResNet18 are removed in the embodiment.

[0096] S2.2: Construct a feature embedding layer FEL, the purpose of FEL is to bridge the convolutional encoder and the Transformer encoder. First, the feature map obtained in S2.1 is deconvolved to obtain the required number and size; then the two-dimensional feature map is flattened to obtain the one-dimensional Token sequence required by the Transformer encoder input; the flattening operation loses spatial information, which is very important for images, so position encoding PE is added to the obtained Token sequence. The final one-dimensional Token sequence is represented as follows:

[0097]

[0098] wherein, represents the one-dimensional Token sequence processed by FEL as the input of the Transformer encoder; Flatten represents the flattening operation; Deconv m represents the deconvolution operation of the modal m; PE m represents the position encoding of the modal m. PE m ∈R c″*H″w″ ; C", H" and W" represent the number and dimension of the one-dimensional Token sequence corresponding to the modal m, respectively. Specifically, in this embodiment, C" = 196, H" W" = 576.

[0099] S2.3: Construct an intra-modal Transformer encoder Intra-modal TE to globally model the local features processed by FEL in S2.2 and establish long-distance dependency relationships. Intra-modal TE is composed of L1 Intra-modal Transformer blocks, each of which is composed of a layer normalization LN layer, an SA layer, an LN layer, and an MLP layer. The MLP layer is composed of two fully connected layers with a Gaussian error linear unit GELU activation function. Specifically, in this embodiment, L1 = 4. The self-attention calculation formula in the SA layer is as follows:

[0100]

[0101]

[0102]

[0103]

[0104] wherein, Q m , K m , V mrespectively represent the query vector, key vector, value vector required for self-attention calculation; are all learnable parameters; LN(·) represents layer normalization; T represents the transpose of a matrix; dk represents the dimension of the query vector K m ; the calculation formula of softmax(·) is where e is a natural constant, and k represents the dimension of the vector x. Specifically, in the present embodiment, k = 576.

[0105] The high-level abstract semantic feature representation with global context information and local information obtained through IMTE is as follows:

[0106]

[0107] where F m represents the high-level abstract semantic feature of the modality m finally obtained by the hybrid modality intra-encoder HIME; MLP m represents a multi-layer perceptron layer; Z m is a calculation intermediate variable; SA m (·) is a self-attention calculation; F m ∈ R C″*H″W″ .

[0108] Further, the construction of the adaptive cross-modality encoder ACME in step S3 specifically includes the following steps:

[0109] S3.1: An adaptive modality weight generation AMWG network is constructed, which is used to adaptively customize different modality weights for each case. The AMWG module is actually an MLP composed of three fully connected layers with a rectified linear unit ReLU activation function. In order to reduce repeated calculations and accelerate convergence, the input thereof is the high-level abstract semantic feature F m of each modality m obtained in step S2.3, and the output thereof is the customized optimal weight w m of each modality. The AMWG operation process is represented as follows:

[0110] (w US , w IRT ) = softmax(MLP AMWG ([F US , F IRT ])

[0111] where w US , w IRT represent the optimal modality weights corresponding to the modalities US and IRT, F US , F IRT represent the high-level abstract semantic features of the modalities US and IRT, [·, ·] represents a splicing operation, and the softmax(·) operation makes wUS +w IRT =1;MLP AMWG represents a multi-layer perceptron composed of three fully connected layers with rectified linear unit (RELU) activation function.

[0112] S3.2: Construct an inter-modal transformer encoder (Inter-modal TE) to interact information and adaptively fuse the features from different modalities obtained in S2 and the customized modal weights obtained in S3.1. First, we introduce a learnable parameter vector cls Token as the last classification feature, and add a position encoding PE as the input of the Inter-modal TE, which is represented as follows:

[0113] F token =[cls token,w US *F US +w IRT *F IRT ]+PE

[0114] where F token ∈R (c″+1)*H″W″ .

[0115] The Inter-modal TE is composed of L2 Inter-modal blocks, each of which is composed of LN layers, multi-headed self-attention (MSA) layers, LN layers, and MLP layers, and the MLP layers are composed of two fully connected layers with Gaussian error linear unit (GELU) activation function. The comparison between the Intra-modal Transformer block in the hybrid modal intra-encoder (HIME) and the Inter-modal Transformer block in the adaptive cross-modal encoder (ACME) is shown in FIG. 8. Specifically, in this embodiment, L2=4. The self-attention calculation formula in the MSA layer is as follows: Figure 5

[0116]

[0117]

[0118]

[0119]

[0120] MSA=W o ​[head 1 , head 2 ...head N ]

[0121] F = MLP(LN(Z)) + Z, Z = MSA(LN(F token )) + F token

[0122] where Q i , K i , V i represent the query vector, key vector, value vector required for self-attention calculation of the i-th head, respectively; W o are all learnable parameters; head i , SA i represent the self-attention calculation result of the i-th head; represents the dimension of the query vector K i ; F ∈ R (C″+1)*H″W″ . Specifically, in the present embodiment, the number of heads is 8.

[0123] Further, the step S4 of constructing the classification head based on MLP (MLP Head) includes the following steps:

[0124] S4.1: Take the value F[0] at the first position in the feature vector F obtained in step S3 as the input of the classification head MLPHead.

[0125] S4.2: Construct the MLP Head, which is a fully connected layer with a ReLU activation function. The input is F[0] and the output is the classification result. It is represented as follows:

[0126] cls = MLPHead(F[0])

[0127] where cls is the final classification category, cls ∈ {benign, malignant}; F[0] is the value F[0] at the first position in the feature vector F obtained in step S3.

[0128] Further, the loss function in step S6 is the cross entropy loss function (CrossEntropy Loss, CELoss), which is defined as follows:

[0129] CE Loss = -[y log(y') + (1-y) log(1-y')]

[0130] where y represents the true label of the sample, and y' is the predicted label.

[0131] ​Further, the optimizer in step S6 is used to update and calculate the network parameters that affect the model training and model output, so as to approach or reach the optimal value, thereby minimizing (or maximizing) the loss function. Specifically, in the embodiment, the Adam optimizer is used.

[0132] Further, the learning rate decreasing method in step S6 refers to the learning rate decreasing with the increase of the number of training iterations, which helps the model converge. Specifically, in the embodiment, the cosine annealing learning rate (CosineAnnealingLearning Rate) decreasing method is used, and the initial learning rate is set to 1e-3 and the minimum learning rate is set to 1e-4.

[0133] Further, the preprocessing method in step S8 is the same as the preprocessing method mentioned in S1.2. Specifically, both the US image and the IRT image are resized to 224*224.

[0134] At present, most of the deep learning-based thyroid nodule image classification methods are based on only a single modality image US image, because US imaging is non-invasive, low-cost, easy to obtain and has other excellent characteristics. The US image and the IRT image can provide information for the diagnosis of thyroid nodules from different angles, for example, the US image can provide morphological information such as the size, shape, aspect ratio, edge and calcification of the lesion area, and the IRT image can provide metabolic information of the lesion area represented by temperature changes. These information complement each other, and based on the multi-modal learning method, joint learning of the features related to thyroid nodules in the two modal images can provide more accurate thyroid nodule classification results to achieve auxiliary diagnosis. The multi-modal model provided by the present application combines the ability of CNN in local feature expression and the advantage of Transformer in global information modeling, and constructs a hybrid encoder to fully extract the features within the modal. In addition, in order to realize reasonable and efficient feature fusion, the present application designs a self-adaptive cross-modal encoder, which can interact the high-level semantic features between the modal, and can formulate personalized modal weights according to different cases. In general, this method can effectively extract features related to thyroid nodules from US images and IRT images, and reasonably fuse features from the two different modalities, so that the final obtained features can fully represent the information related to the diagnosis of thyroid nodules, and realize more accurate classification of thyroid nodules, and provide objective and accurate reference for doctors to assist in diagnosis.

[0135] The above description is only a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto, and any modification, equivalent replacement and improvement made by any person skilled in the art within the technical scope disclosed by the present application shall be included in the protection scope of the present application.

Claims

1. A method for classifying thyroid nodule images based on ultrasound and infrared multimodal imaging, characterized in that, Includes the following steps: S1: Create a multimodal dataset of thyroid glands; S2: Construct two hybrid modal intra-encoders (HIMEs) to extract features from IRT and US images respectively; S3: Construct an adaptive cross-modal encoder ACME to adaptively fuse the features of the IRT image and the US image; The construction of the adaptive cross-modal encoder ACME specifically includes the following steps: S3.1: Construct an Adaptive Modality Weight Generation (AMWG) network to adaptively customize different modality weights for each case based on the actual case situation; the AMWG module is a multilayer perceptron (MLP) consisting of three fully connected layers with modified linear units (ReLU) activation functions, whose input is each modality obtained in step S2.

3. m High-level abstract semantic features The output is the optimal weights for each modality, customized according to the settings. ; The AMWG calculation process is as follows: in, This represents the optimal modal weights corresponding to modes US and IRT. , These represent the high-level abstract semantic features of modal US and IRT, respectively. This indicates a splicing operation. Operation makes , MLP AWMG This represents a multilayer perceptron consisting of three fully connected layers with modified linear unit ReLU activation functions; S3.2: Construct an inter-modal Transformer encoder (TE) to perform information exchange and adaptive feature fusion on the features from different modalities obtained in S2 and the customized modal weights obtained in S3.

1. The steps are as follows: First, a learnable parameter vector cls Token is introduced as the final classification feature, and a positional encoding PE is added as the input to the Inter-modal TE, as shown below: in, ; Inter-modal TE consists of L2 inter-modal blocks. Each inter-modal block consists of an LN layer, a multi-head self-attention MSA layer, an LN layer, and an MLP layer. The MLP layer consists of two fully connected layers with Gaussian error linear units (GELU) activation functions. S4: Construct a classification head based on multilayer perceptron (MLP) to classify the fusion features related to thyroid nodule diagnosis obtained in S3; S5: Combine HIME, ACME, and MLP Head into an adaptive multimodal hybrid AmmH model; S6: Divide the thyroid multimodal dataset into training, validation and test sets according to a certain ratio, define the loss function and optimizer, and define the learning rate decrement method; S7: Based on the backpropagation algorithm, the AmmH model is trained end-to-end multiple times. The model parameter values ​​with the highest classification accuracy on the test set are taken as the final AmmH model parameter values ​​used for thyroid nodule classification. S8: The preprocessed multimodal data of the thyroid gland is used as the input of the AmmH model to output the classification results of the thyroid nodule images.

2. The method according to claim 1, characterized in that: Step S1, creating the thyroid multimodal dataset, specifically includes the following steps: S1.1: Acquire thyroid IRT and US images of the subject to obtain diagnostic results for thyroid nodules; S1.2: Preprocess the acquired thyroid IRT and US images and resize them to a uniform size; S1.3: Pair the preprocessed IRT images and US images from S1.2 as samples, and use the diagnostic results of thyroid nodules from S1.1 as sample labels to obtain a thyroid multimodal dataset.

3. The method according to claim 1, characterized in that; The construction of the hybrid modal internal encoder (HIME) in step S2 specifically includes the following steps: S2.1: Construct an Intra-modal CE to extract features from a single modality. The resulting intra-modal local feature representation is as follows: in m Indicates imaging mode, ; Representing modes m Local features within the modality; Representing modes m The corresponding modal in-convolution encoder; Representing modes m image; ; ; This represents the number of channels, height, and width of the modal m image; The number of channels, height, and width of the feature map representing the local features of mode m; S2.2: Construct the Feature Embedding Layer (FEL). First, perform deconvolution on the feature maps obtained in S2.1 to obtain the required number and size. Then, flatten these two-dimensional feature maps to obtain the one-dimensional token sequence required for the Transformer encoder input. Finally, add positional encoding (PE) to the obtained token sequence to obtain the final one-dimensional token sequence representation as follows: in, This represents the one-dimensional token sequence that serves as the input to the Transformer encoder after processing by FEL. This indicates that the flattening operation will be performed; Representing modes m The deconvolution operation; Representing modes m Location encoding; ; Representing modes m The number and dimension of the corresponding one-dimensional token sequence; S2.3: Construct an Intra-modal Transformer encoder (TE) to globally model the local features after FEL processing in S2.2 and establish long-distance dependencies. The Intra-modal TE consists of L1 Intra-modal Transformer blocks. Each Intra-modal Transformer block consists of four parts: a layer normalized LN layer, an SA layer, an LN layer, and an MLP layer. The MLP layer consists of two fully connected layers with Gaussian error linear units (GELU) activation functions. The high-level abstract semantic feature representation with global and local context information obtained by the intramodal Transformer encoder is as follows: in, This represents the high-level abstract semantic features of mode m ultimately obtained by the HIME encoder in mixed modalities; Indicates a multi-layer perceptron; To calculate intermediate variables; For self-attention calculation; .

4. The method according to claim 3, characterized in that: In step S2.3, the self-attention calculation formula in the SA layer is as follows: in, These represent the query vector, key vector, and value vector required for self-attention computation, respectively. These are all learnable parameters; Representation layer normalization; Represents the transpose of a matrix; Represents the query vector The dimension; The calculation formula is: ,in Let k be a natural constant, and k denote a vector. Dimensions.

5. The method according to claim 1, characterized in that: In step S3.2, the self-attention calculation formula in the MSA layer is as follows: in, These represent the query vector, key vector, and value vector required for the self-attention calculation of the i-th head, respectively. These are all learnable parameters; Representation layer normalization; , This represents the self-attention calculation result for the i-th head; Represents the query vector The dimension; .

6. The method according to claim 1, characterized in that: Step S4, constructing the MLP-based classification head, includes the following steps: S4.1: Obtain the value of the first position in the feature vector F obtained in step S3. As input to the classification head MLP Head; S4.2: Construct the MLP Head, which is a fully connected layer with a ReLU activation function; input The classification results are output as follows: Where cls represents the final classification category. .

7. The method according to claim 1, characterized in that: The loss function mentioned in step S6 is the cross-entropy loss function CE Loss, defined as follows: Where y represents the true label of the sample. For predicted labels.

8. The method according to claim 1, characterized in that: In step S6, the optimizer is used to update and calculate the network parameters that affect model training and model output, so that they approach or reach the optimal value, thereby minimizing or maximizing the loss function; the learning rate decreasing method refers to the learning rate decreasing as the number of training iterations increases, in order to help the model converge.

9. The method according to claim 1, characterized in that: The preprocessing method described in step S8 is the same as the preprocessing method mentioned in S1.2.

Images