A multimodal thyroid tumor image classification method and terminal device

Abstract

The present invention discloses a multimodal thyroid pathology image classification method and terminal device, wherein the method comprises the steps of: using three ResNet18 networks to extract information features of three modal thyroid pathology images respectively to obtain trimodal information features, wherein the three modal thyroid pathology images include thyroid ultrasound images, thyroid elasticity images, and thyroid blood flow images; using a multimodal multi-head attention module to extract the common information features of the three modal thyroid pathology images; fusing the trimodal information features and the common information features, and using a residual network to classify the thyroid pathology images, and outputting the classification results. The present invention verifies the designed multimodal thyroid pathology image classification method on a multimodal thyroid ultrasound dataset provided by a cooperative research unit, and the results prove that this method can accurately classify thyroid pathology images, providing ultrasound doctors with fast and accurate assistance in diagnosing thyroid cancer.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a multimodal thyroid tumor image classification method and terminal equipment. Background Art

[0002] Thyroid cancer is the most common thyroid malignancy, accounting for about 1% of systemic malignant tumors, and can occur in multiple age groups. The thyroid tissue structure is complex, and there are many interference factors in the ultrasound image. Therefore, it is difficult for doctors to directly classify the lesion characteristics in the ultrasound image of thyroid cancer. In order to alleviate this situation, in addition to thyroid ultrasound imaging, doctors usually combine ultrasound elasticity imaging and ultrasound blood flow imaging to more accurately identify thyroid cancer images.

[0003] At present, deep learning has achieved promising results in computer-aided diagnosis due to the advantages of obtaining high-dimensional features. However, most of the existing studies are based on classification diagnosis based on a single thyroid ultrasound image, and fail to utilize the soft and hard information of the tissue in the elasticity image and the blood flow information in the blood flow image, resulting in a low recognition accuracy of thyroid cancer images.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, an object of the present invention is to provide a multimodal thyroid tumor image classification method and terminal device, aiming to solve the problem of low classification accuracy of thyroid cancer images in the prior art.

[0006] The technical solution of the present invention is as follows:

[0007] A multimodal thyroid pathology image classification method, comprising the steps of:

[0008] Acquiring three-modality thyroid pathology images, wherein the three-modality thyroid pathology images include a thyroid ultrasound image, a thyroid elasticity image, and a thyroid blood flow image;

[0009] Three ResNet18 networks are used to extract information features of the three modal thyroid pathology images respectively to obtain trimodal information features;

[0010] A multimodal multi-head attention module is used to extract the common information features of thyroid pathology images of three modalities;

[0011] After the three-modal information features and the common information features are fused, the residual network is used to classify the thyroid pathology images and the classification results are output.

[0012] The multimodal thyroid pathology image classification method, wherein three ResNet18 networks are used to extract information features of the three modal thyroid pathology images respectively, and the step of obtaining the trimodal information features includes:

[0013] In the ResNet18 network, the dual complex wavelet transform is used to replace the convolution operation to obtain the improved ResNet18 network;

[0014] Three improved ResNet18 networks are used to extract information features from the three modal thyroid pathology images respectively to obtain the three modal features themselves;

[0015] Adding a multimodal feature fusion module to the ResNet18 network, and extracting common features of the thyroid pathology images of the three modalities through the multimodal feature fusion module;

[0016] The three modal features themselves are fused with the common features through the multimodal feature fusion module to obtain the three modal information features.

[0017] The multimodal thyroid pathology image classification method, wherein the step of extracting the common features of the three modal thyroid pathology images through the multimodal feature fusion module comprises:

[0018] The three modal features are concatenated together, and 1x1 convolution is used to extract the common features of the three modal features.

[0019] The channel information of the three modal features is extracted through global average pooling, and then the extracted channel information vector is used to construct a channel information feature F containing the relevant information between the channels of the three modalities. C (i, j, k);

[0020] Then use 1x1 convolution and global average pooling to transform the channel information feature F C C×C×C Further extraction is performed to readjust the channel weights of the channel information extracted from the three modal features themselves;

[0021] A self-attention module is used to extract useful features from the three modal features, and the useful features are fused with the common features extracted from the three modal features to obtain fused features;

[0022] The fused features are sent back to each improved ResNet18 network to guide the feature extraction direction of the improved ResNet18 network, so that the feature dimensions extracted by the three improved ResNet18 networks are close, and the trimodal information features are obtained.

[0023] The multimodal thyroid pathology image classification method, wherein the channel information feature F C The value of (i, j, k) is obtained by multiplying the channel information of the corresponding coordinate positions of the three modes. C (i, j, k) = V1(i)*V2(j)*V3(k)i, j, k∈[0, C), where V1, V2, V3 represent channel information vectors extracted from the three modal features themselves; i, j, k represent the corresponding coordinate positions.

[0024] The multimodal thyroid pathology image classification method, wherein the fused feature expression is:

[0025] F′ i = Bottleneck({Attention, Conv}(F1, F2, F3)), i=1, 2, 3, where,

[0026] Bottleneck() represents the bottleneck layer, which is used to extract features. {Attention, Conv} represents the attention module and 1x1 convolution module respectively.

[0027] The multimodal-based thyroid pathology image classification method, wherein the parameter acquisition step of the multimodal feature fusion module is:

[0028] Setting a self-supervised task to pre-train the ResNet18 network, wherein the self-supervised task is a classification task of thyroid ultrasound images, thyroid elasticity images, and thyroid blood flow images;

[0029] The loss function used in the self-supervised classification task is the cross entropy function, and the classification process is expressed as: self =-∑ classesf (y true )log y pred , where fO represents label smoothing;

[0030] After completing the self-supervised pre-training, the parameters in the ResNet18 network are migrated to the multimodal feature fusion module.

[0031] The multimodal thyroid pathology image classification method, wherein the step of using a multimodal multi-head attention module to extract common information features of three modal thyroid pathology images comprises:

[0032] The input three-modality thyroid pathology images are divided into multiple sub-images using a dynamic window;

[0033] The vectors Q, K, and V in an attention calculation mechanism are respectively encoded by sub-images of the same position and size in the three modal images, that is, the common information of the same position of the three modal thyroid pathology images is extracted through one attention;

[0034] The attention calculations corresponding to all sub-images constitute a multi-head attention mechanism, and the output is a feature vector containing the common information of the three modalities. The feature vector is input into the encoder to extract the common information features.

[0035] A storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the multi-modality-based thyroid pathology image classification method of the present invention.

[0036] A terminal device, comprising: a processor, a memory and a communication bus; the memory stores a computer-readable program executable by the processor;

[0037] The communication bus realizes the connection and communication between the processor and the memory;

[0038] When the processor executes the computer-readable program, the steps in the multi-modality-based thyroid pathology image classification method of the present invention are implemented.

[0039] Beneficial effects: The present invention proposes to use three ResNet18s improved by using dual-tree complex wavelet transform as branches to extract the trimodal information features of the three modal thyroid pathology images respectively, and then use a multimodal multi-head attention module branch to extract the common information features of the three modal thyroid pathology images, fuse the trimodal information features and the common information features, and use the residual network to classify the thyroid pathology images and output the classification results. At the same time, in order to better fuse the information features between the modalities, a multimodal feature fusion module (MMFG) is also proposed to guide the feature extraction direction of each branch and narrow the differences between the information features of each modality. The present invention verifies the designed multimodal thyroid pathology image classification method on the multimodal thyroid ultrasound dataset provided by the cooperative research unit. The results show that this method can accurately classify thyroid pathology images and provide ultrasound doctors with fast and accurate assistance in diagnosing thyroid cancer. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The present invention is a flowchart of a multimodal thyroid pathology image classification method.

[0041] Figure 2 This is a schematic diagram of the principle of thyroid pathology image classification based on multimodality in the present invention.

[0042] Figure 3 This is the schematic diagram of the multimodal feature fusion module.

[0043] Figure 4 This is a schematic diagram of the multi-modal multi-head attention mechanism.

[0044] Figure 5 A schematic diagram of a terminal device. DETAILED DESCRIPTION

[0045] The present invention provides a thyroid pathology image classification method and terminal device based on multimodality. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0046] The thyroid gland is the main hormone gland that controls the growth, metabolism, and maturation of the body. However, thyroid cell tissue may grow abnormally, leading to the formation of benign or malignant thyroid lesions. Ultrasound is a typical non-invasive diagnostic method and is often used to examine thyroid cancer lesions. However, due to the limitations of the information provided by ultrasound images, ultrasound elastography and thyroid blood flow imaging are also used clinically to assist in diagnosis. However, the existing diagnosis is made visually by doctors, which is a time-consuming, labor-intensive, and highly subjective process. Therefore, the automatic differentiation of benign and malignant thyroid nodules is very useful for thyroid imaging reports.

[0047] Based on this, the present invention provides a flowchart of a multimodal thyroid pathology image classification method, such as Figure 1 As shown, it includes the steps of:

[0048] S10, acquiring three-modal thyroid pathology images, wherein the three-modal thyroid pathology images include a thyroid ultrasound image, a thyroid elasticity image, and a thyroid blood flow image;

[0049] S20, using three ResNet18 networks to extract information features from the three modal thyroid pathology images respectively to obtain trimodal information features;

[0050] S30, using a multi-modal multi-head attention module to extract common information features of thyroid pathology images of three modalities;

[0051] S40, after fusing the three-modal information features and the common information features, classify the thyroid pathology images using a residual network and output the classification results.

[0052] Specifically, since the input is thyroid pathology images of three different modalities, the image information thereof is quite different, so the present embodiment adopts three ResNet18 networks as feature extraction branches to extract information features of the three modal thyroid pathology images respectively; and due to the differences between the modalities, the features extracted by the three ResNet18 network branches are quite different, and the fusion effect is not good, so the present embodiment proposes a multimodal feature fusion module to guide the direction of each modality information extraction branch, and to reduce the difference between the feature information extracted by the three ResNet18 network branches as much as possible, which is more conducive to the final multimodal information fusion; and in addition to extracting information from each modality, the present embodiment also uses multimodal multi-head attention to extract the common information features of the three modal thyroid pathology images, and after fusing the three modal information features and the common information features, and using the residual network to classify the thyroid pathology images, it can output accurate classification results, and better improve the network's diagnostic accuracy for thyroid cancer.

[0053] In this embodiment, the three modal thyroid pathology images include thyroid ultrasound images, thyroid elasticity images and thyroid blood flow images. This embodiment uses multimodal thyroid pathology images collected by the People's Hospital of Nanshan District, Shenzhen. The data set contains a total of 234 trimodal cases, including 126 benign cases and 108 malignant cases, each of which has been diagnosed by hospital pathology. This embodiment data set has only 234 available cases, and the data sample size is small. Therefore, this embodiment performs data enhancement on the original data: mirroring, random angle rotation, adding Gaussian noise, adding impulse noise, and brightness adjustment.

[0054] In some embodiments, three ResNet18 networks are used to extract information features of the three modal thyroid pathology images respectively, and the steps of obtaining trimodal information features include: replacing the convolution operation with dual complex wavelet transform in the ResNet18 network to obtain an improved ResNet18 network; using three improved ResNet18 networks to extract information features of the three modal thyroid pathology images respectively to obtain trimodal self-features; adding a multimodal feature fusion module to the ResNet18 network, and extracting the common features of the three modal thyroid pathology images through the multimodal feature fusion module; fusing the trimodal self-features with the common features through the multimodal feature fusion module to obtain trimodal information features.

[0055] Specifically, if Figure 2As shown, the network used for thyroid pathology image classification of the present invention is composed of two branches. Branch one is composed of three ResNet18 networks. In image classification, the residual network has always performed well in image classification tasks with its effective residual connection structure. Compared with the traditional fully connected network, the residual connection can avoid gradient explosion and gradient vanishing, and can also accelerate convergence. At the same time, since ultrasound images usually have problems such as artifacts and speckle noise, this embodiment uses a dual-tree complex wavelet transform (DTCWT) to replace the convolution operation therein to obtain an improved ResNet18 network. The dual-tree complex wavelet transform uses a binary tree structure for a two-way wavelet transform, which overcomes the problem of the lack of translation invariance of the wavelet transform. In this embodiment, the low-frequency information L and high-frequency information H of the image can be obtained by performing a dual-tree complex wavelet transform on the input feature F, that is, H, L = DTCWT (F), which can eliminate the high-frequency part with more noise and retain the low-frequency part with more information. This embodiment uses three improved ResNet18 as network branches for feature extraction, such as Figure 2 As shown, the images of the three modalities are each sent to the corresponding ResNet18 branch to extract features. At the same time, in order to better combine the information between the modalities, this embodiment uses a multimodal feature fusion module branch to extract the common features of the three modalities, and then fuse them with the features of each modality itself to guide the feature extraction of the subsequent network layer. Branch 2 uses a multi-head attention and autoencoder to extract the common information features of the three modal images, and finally fuses the three-modal information features extracted by branch 1 for network classification.

[0056] In some implementations, since differences between data may lead to differences between the extracted features, in order to better utilize the trimodal information features, this embodiment adds the following to the ResNet18 network: Figure 3 The multimodal feature fusion module shown in the figure, as shown in the figure, F1, F2, and F3 are the three modal self-features extracted by the three improved ResNet18 branches. In this embodiment, the three modal self-features are first cascaded together, and the common features of the three modal self-features are extracted by 1x1 convolution; at the same time, for the cascaded three modal self-features, the channel information of the three modal self-features is extracted by global average pooling, and then the extracted channel information vector is used to construct a channel information feature F containing the relevant information between the channels of the three modalities. C (i, j, k), where the channel information feature F C The value of (i, j, k) is obtained by multiplying the channel information of the corresponding coordinate positions of the three modes. C (i, j, k) = V1(i)*V2(j)*V3(k) i, j, k∈[0, C), where V1, V2, V3 represent the channel information vectors extracted from the three modal features; i, j, k represent the corresponding coordinate positions, FC (i, j, k) contains the relevant information between each channel of the three modalities.

[0057] Then, this embodiment uses 1x1 convolution and global average pooling to process the channel information feature F C (i, j, k) is further extracted to readjust the channel weights of the channel information extracted from the three-modal self-features; then this embodiment uses a self-attention module to extract useful features from the three-modal self-features F1, F2, and F3, and fuses the useful features with the common features extracted from the three-modal self-features to obtain fused features F′1, F′2, and F′3. The fused feature expression is: F′i=Bottleneck({Attention, Conv}(F1, F2, F3)), i=1, 2, 3, where Bottleneck() represents the bottleneck layer for feature extraction, and {Attention, Conv} represent the attention module and the 1x1 convolution module respectively. The fused features are sent back to each improved ResNet18 network to guide the feature extraction direction of the improved ResNet18 network, so that the feature dimensions extracted by the three improved ResNet18 networks are close, reducing the impact caused by modality differences, thereby obtaining trimodal information features.

[0058] In some embodiments, due to the small number of cases with three modalities at the same time, there are a large number of data with only a single modality or two modalities. In order to make better use of the data, the impact of the small number of samples in the multimodal data set is also reduced as much as possible. In this embodiment, a self-supervised task is set to pre-train the ResNet18 network, and the self-supervised task is a classification task for thyroid ultrasound images, thyroid elasticity images, and thyroid blood flow images. Since the thyroid elasticity image and thyroid blood flow image are selected by doctors on the basis of thyroid ultrasound images, the area around the nodule is obtained to obtain the soft and hard information and blood flow information of the tissue in the area. Therefore, the background information outside the elastic information and blood flow information area is the same as the ultrasound image. The self-supervised task of classifying three modal images can prompt ResNet18 to classify images according to the elastic information area and the blood flow information area. Moreover, in this embodiment, the images between the three modalities are synthesized, and the information of other modalities is used to interfere with the current modality, so that the network needs to extract and judge the modal information area in the image more accurately, so that the pre-trained network can better focus on the useful information area of ​​each modality and shield the useless background information outside the area. During the training process, the network was trained and optimized by the stochastic gradient descent algorithm, with the optimizer momentum set to 0.99 and the initial learning rate set to 10 -3 , and dynamically adjusts with the number of learning steps.

[0059] In this embodiment, the loss function used in the self-supervised classification task is the cross entropy function. At the same time, in order to obtain better classification results, a label smoothing method is used: self =-∑ classesf (y true )logy pred , where f() represents label smoothing, which can better improve the network classification performance; after completing the self-supervised pre-training, the parameters in the ResNet18 network are migrated to the multimodal feature fusion module for subsequent classification and diagnosis of multimodal thyroid images.

[0060] In some embodiments, the multimodal feature fusion module can effectively guide the ResNet18 branch to extract the modal effective information from each modality, while reducing the gap between the features of each modality. At the same time, we also propose a multimodal multi-head attention mechanism to effectively extract the common features of the three modal images. Its structure is as follows Figure 4 As shown, this embodiment uses a dynamic window to divide the input three-modal thyroid pathology images into multiple sub-images. In order to extract the common information of the three modalities, the vectors Q, K, and V in an attention calculation mechanism are respectively encoded by sub-images of the same position and size of the three modal images, that is, one attention extracts the common information of the three modalities at the same position. The attention calculations corresponding to all sub-images constitute a multi-head attention mechanism, and the output is a feature vector containing the common information of the three modalities, which is sent to the subsequent Transform encoder to extract further common information features, and finally used together with the information features of the three modalities extracted by the ResNet18 branch for the diagnosis and classification of the thyroid gland.

[0061] The vectors Q, K, and V in an attention calculation mechanism are respectively encoded by sub-images of the same position and size in the three modal images, that is, the common information of the same position of the three modal thyroid pathology images is extracted through one attention;

[0062] The attention calculations corresponding to all sub-images constitute a multi-head attention mechanism, and the output is a feature vector containing the common information of the three modalities. The feature vector is input into the encoder to extract the common information features.

[0063] A storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the multi-modality-based thyroid pathology image classification method of the present invention.

[0064] A terminal device, comprising: a processor, a memory and a communication bus; the memory stores a computer-readable program executable by the processor;

[0065] The communication bus realizes the connection and communication between the processor and the memory;

[0066] When the processor executes the computer-readable program, the steps in the multi-modality-based thyroid pathology image classification method of the present invention are implemented.

[0067] In some embodiments, a storage medium is also provided, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the multi-modality-based thyroid pathology image classification method of the present invention.

[0068] In some implementations, the present application also provides a terminal device, such as Figure 5 As shown, it includes at least one processor (processor) 20; display screen 21; and memory (memory) 22, and may also include a communications interface (Communications Interface) 23 and a bus 24. Among them, the processor 20, the display screen 21, the memory 22 and the communication interface 23 can communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communication interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0069] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0070] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.

[0071] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0072] In addition, the specific process of loading and executing multiple instruction processors in the storage medium and the terminal device has been described in detail in the above method and will not be described here one by one.

[0073] In summary, the present invention proposes to use three ResNet18s improved by using dual-tree complex wavelet transform as branches to extract the trimodal information features of the three modal thyroid pathology images respectively, and then use a multimodal multi-head attention module branch to extract the common information features of the three modal thyroid pathology images, fuse the trimodal information features and the common information features, and use the residual network to classify the thyroid pathology images and output the classification results. At the same time, in order to better fuse the information features between the modalities, a multimodal feature fusion module (MMFG) is also proposed to guide the feature extraction direction of each branch and narrow the differences between the information features of each modality. The present invention verifies the designed multimodal thyroid pathology image classification method on the multimodal thyroid ultrasound data set provided by the cooperative research unit. The results show that this method can accurately classify thyroid pathology images and provide ultrasound doctors with fast and accurate assistance in diagnosing thyroid cancer.

[0074] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multimodal thyroid pathology image classification method, characterized in that: Includes steps: Acquiring three-modality thyroid pathology images, wherein the three-modality thyroid pathology images include a thyroid ultrasound image, a thyroid elasticity image, and a thyroid blood flow image; In the ResNet18 network, the dual complex wavelet transform is used to replace the convolution operation to obtain the improved ResNet18 network; Three improved ResNet18 networks are used to extract information features from the three modal thyroid pathological images respectively to obtain the three modal features themselves; Adding a multimodal feature fusion module to the ResNet18 network, and extracting common features of the thyroid pathology images of the three modalities through the multimodal feature fusion module; The three-modal features are fused with the common features through the multimodal feature fusion module to obtain a three-modal information feature; A multimodal multi-head attention module is used to extract the common information features of thyroid pathology images of three modalities; After fusing the three-modal information features and the common information features, a residual network is used to classify the thyroid pathology images and output the classification results; The parameter acquisition steps of the multimodal feature fusion module are as follows: A self-supervised task is set to pre-train the ResNet18 network, wherein the self-supervised task is a classification task of thyroid ultrasound images, thyroid elasticity images, and thyroid blood flow images; and the thyroid pathology images of the three modalities are synthesized in pairs, and the information of other modalities is used to interfere with the current modality; The loss function used in the self-supervised task is the cross entropy function, and the classification process is expressed as: self =-∑ classes f(y true )logy pred , where f() represents label smoothing; After completing the self-supervised pre-training, the parameters in the ResNet18 network are migrated to the multimodal feature fusion module; The steps of using the multimodal multi-head attention module to extract the common information features of the three modal thyroid pathology images include: The input three-modality thyroid pathology images are divided into multiple sub-images using a dynamic window; The vectors Q, K, and V in an attention calculation mechanism are respectively encoded by sub-images of the same position and size in the three modal images, that is, the common information of the same position of the three modal thyroid pathology images is extracted through one attention; The attention calculations corresponding to all sub-images constitute a multi-head attention mechanism, and the output is a feature vector containing the common information of the three modalities. The feature vector is input into the encoder to extract the common information features.

2. The multimodal thyroid pathology image classification method according to claim 1, characterized in that: The step of extracting common features of the three modal thyroid pathology images by the multimodal feature fusion module comprises: The three modal features are concatenated together, and 1x1 convolution is used to extract the common features of the three modal features. The channel information of the three modal features is extracted through global average pooling, and then the extracted channel information vector is used to construct a channel information feature F containing the relevant information between the channels of the three modalities. C (i, j, k); Then use 1x1 convolution and global average pooling to transform the channel information feature F C C×C×C Further extraction is performed to readjust the channel weights of the channel information extracted from the three modal features themselves; A self-attention module is used to extract useful features from the three modal features, and the useful features are fused with the common features extracted from the three modal features to obtain fused features; The fused features are sent back to each improved ResNet18 network to guide the feature extraction direction of the improved ResNet18 network, so that the feature dimensions extracted by the three improved ResNet18 networks are close, and the trimodal information features are obtained.

3. The multimodal thyroid pathology image classification method according to claim 2, characterized in that: The channel information feature F C The value of (i, j, k) is obtained by multiplying the channel information of the corresponding coordinate positions of the three modes. C (i, j, k) = V1(i)*V2(j)*V3(k)i, j, k∈[0, C), where V1, V2, V3 represent channel information vectors extracted from the three modal features themselves; i, j, k represent the corresponding coordinate positions.

4. The multimodal thyroid pathology image classification method according to claim 2, characterized in that: The fused feature expression is: F′ i =Bottleneck({Attention,Conv}(F1, F2, F3)), i=1, 2, 3, where Bottleneck() represents the bottleneck layer for extracting features, and {Attention, Conv} represent the attention module and 1x1 convolution module respectively.

5. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the multi-modality-based thyroid pathology image classification method as described in any one of claims 1-4.

6. A terminal device, characterized in that: include: Processor, memory and communication bus; The memory stores a computer-readable program executable by the processor; The communication bus realizes the connection and communication between the processor and the memory; When the processor executes the computer-readable program, the processor implements the steps in the multi-modality-based thyroid pathology image classification method as described in any one of claims 1-4.

Images