A tensor chain decomposition-based frequency domain noise reduction network multi-modal medical image classification method

Abstract

The application belongs to the field of medical image classification, and discloses a kind of multi-modal medical image classification method of frequency domain noise reduction network based on tensor chain decomposition, comprising the following steps: step 1: the training set images of two modalities are input into two backbone networks to extract image features;Step 2: the image features are input into the tensor chain decomposition module to obtain the core feature tensor;Step 3: the core feature tensors of the images of the two modalities are weighted and summed to obtain the core feature fusion tensor, which is then input into the frequency domain noise reduction module to reduce the noise generated by fusion and improve the cross-domain generalization ability;Step 4: the core feature fusion tensor is multiplied according to the tensor chain decomposition formula to obtain the fusion feature;Step 5: the fusion feature is input into the classifier to obtain the classification result, and the classification label is used for supervised learning.The medical image classification network of the application can effectively prevent overfitting and dimension disaster, and has good generalization ability.

Description

Technical Field

[0001] This invention belongs to the field of medical image classification, specifically relating to a multimodal medical image classification method based on a frequency domain denoising network using tensor chain decomposition. Background Technology

[0002] Multimodal fusion: Each modality carries different information, and different modalities usually carry complementary key information, which is beneficial for the network to learn and make better decisions. Deep learning-based fusion methods are generally divided into three types: fusing raw data, fusing decision results, and fusing intermediate features. Papers such as "Brain tumor segmentation using convolutional neural networks in MRI images" employ a method of fusing raw data. This method fuses raw images at the smallest granularity, preserving the original image information to the greatest extent. However, this method is computationally intensive and cannot address the heterogeneity issues inherent in multimodal medical images. Papers such as "Fully convolutional networks for multi-modality isointense infantbrain image segmentation" use a method of fusing decision results. This method inputs each modality's image into an independent network to learn and obtain its own prediction results. Finally, the decisions from the last layer of the network are fused. This method can preserve the complete information of each modality, but it lacks the underlying interaction of multimodal data, limiting the network's performance. Most multimodal fusion networks use a method of fusing intermediate features. This method is more diverse and flexible, and can overcome the problem of the first two fusion methods suppressing intramodal or intermodal interactions.

[0003] While fusing intermediate features is suitable for multimodal medical image problems of varying quality, these images typically exhibit different statistical characteristics across different modalities. Simple feature concatenation cannot effectively integrate complementary information from two modalities, resulting in significant redundancy and potentially leading to the curse of dimensionality in data processing. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multimodal medical image classification method based on a frequency domain denoising network using tensor chain decomposition. This method primarily achieves effective fusion of multimodal medical images by improving tensor decomposition techniques and combining CNN with traditional signal processing techniques. This reduces redundant noise generated during the fusion of images from different modalities, thereby improving the accuracy of multimodal medical image classification.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0006] This invention is a multimodal medical image classification method based on a frequency domain denoising network using tensor chain decomposition, comprising the following steps:

[0007] Step S1: Input the training set images I1 and I2 of the two modalities into two backbone networks to extract the deep classification features F1 and F2 of the medical images of the two modalities; the backbone networks use layer1, layer2 and layer3 of ResNet50 to extract the deep classification features of the medical images of the two modalities, and the backbone networks for extracting the deep classification features of the medical images of the two modalities are completely identical.

[0008] Step S2: Input the depth classification features F1 and F2 of the two modal medical images into two tensor chain decomposition modules to obtain the core feature tensors, i.e., two three-dimensional tensors. and four two-dimensional matrices The tensor chain decomposition is a method of tensor decomposition, and it is expressed as follows:

[0009]

[0010] Where X(i1,i2,…,i) N ) represents an N-dimensional tensor. Let represent the nth 3D core matrix. The first dimension of the first core matrix and the third dimension of the Nth core matrix in tensor chain decomposition are generally considered to be 1, meaning the first and Nth core matrices can be viewed as two-dimensional matrices. When the deep classification features are three-dimensional, the tensor chain decomposition is expressed as:

[0011]

[0012] Where X(c,h,w) represents the deep classification feature, and Rc and Rw represent the rank of the tensor chain decomposition in the channel and width dimensions, respectively. This represents the nth three-dimensional core feature tensor obtained from the decomposition. The three-dimensional feature tensor in deep learning can be viewed as the product of a three-dimensional core feature tensor and two two-dimensional core feature tensors. By controlling the rank value, the number of network parameters can be reduced, preventing overfitting and the curse of dimensionality.

[0013] The core feature tensor and The generation method is represented as:

[0014]

[0015]

[0016] Where GAP_c and GAP_w represent global average pooling, generating a vector that retains only one dimension of X(c,h,w). Rc and Rw vectors are concatenated to obtain... and

[0017] Step S3: The core feature tensors of the two modalities are fused using a feature-level fusion method, which involves weighted summation of the core feature tensors of the two modalities to obtain the core feature fusion tensor. The result is then input into a frequency domain noise reduction module to reduce noise generated by fusion and improve cross-domain generalization ability; the fusion of the two modal features can be expressed as:

[0018]

[0019]

[0020]

[0021] Wherein, λ1, λ2, and λ3 are learnable parameters that can be dynamically adjusted during network training to optimize the weighting of the two modalities in the classification task. The frequency domain noise reduction module specifically comprises:

[0022] Step S3-1: ... Perform a Fast Fourier Transform on each channel individually to obtain the corresponding frequency domain representation. It can be represented as:

[0023]

[0024] in, Let represent the core feature fusion tensor in the spatial domain, h and w represent the two dimensions of the core feature fusion tensor in the spatial domain, x and y represent the two dimensions of the core feature fusion tensor in the frequency domain, n and m increment from 0 to h and w respectively, j represents an imaginary number, and e represents 2.1718… an infinite non-repeating decimal.

[0025] Step S3-2: ... Three consecutive frequency-domain spatial attention modules are input, and different frequency components are adaptively modulated to filter out noise and frequency components that are not conducive to generalization. The spatial attention module specifically includes the following steps:

[0026] Step S3-2-1: Spatial mapping is performed using a 1×1 convolutional layer, a BN layer, and a ReLU activation function;

[0027] Step S3-2-2: Feature aggregation is performed, consisting of average pooling and max pooling operations;

[0028] Step S3-2-3: Spatial mask is learned using a 7×7 convolutional layer, a BN layer, and a Sigmoid activation function;

[0029] Step S3-2-4: Multiply the outputs of steps S3-2-1 and S3-2-3 to complete the modulation of different frequency components.

[0030] Step S3-3: Perform an inverse fast Fourier transform on each channel of the output from step S3-2 individually.

[0031] Step S4: Multiply the core feature fusion tensor output by the frequency noise reduction module in S3 according to the tensor chain decomposition formula to obtain the fused feature;

[0032] Step S5: Input the fused features into a classifier consisting of a series of average pooling, fully connected layers, and random deactivation operations to obtain the classification result;

[0033] Step S6: Based on the loss function, train the entire classification network, including the tensor chain decomposition module and the frequency domain denoising module, using the classification labels to achieve classification of multimodal medical images; the loss function is the cross-entropy loss function. In the Seven-Point Checklist multimodal skin cancer dataset, the loss function is the mean of the losses of the eight labels, which can be specifically expressed as:

[0034]

[0035] Where L represents the network loss, L diag ,L pn ,L str ,L pig ,L rs ,L dag ,L bwv L_vs is the cross-entropy loss for 8 labels.

[0036] The beneficial effects of this invention are: tensor chain decomposition is modularized in a deep learning network, and the modules are made to approximate tensor chain decomposition through learning.

[0037] This invention uses Fast Fourier Transform to fuse core features into tensor transform and transform them to the frequency domain for noise filtering. Since different spatial locations in the frequency domain correspond to different frequency components, by learning the attention module, some noise and frequency components that are not conducive to generalization can be filtered out.

[0038] This invention fuses depth image features by transforming them to their subspaces through tensor chain decomposition. This avoids or mitigates the problem that simple feature stitching cannot effectively fuse complementary information between two modalities due to the different statistical characteristics that usually exist in different modalities, resulting in significant redundancy and easily causing the curse of dimensionality in data processing.

[0039] This invention improves the robustness of the method by using an attention mechanism in the frequency domain to suppress certain frequency components. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method of the present invention.

[0041] Figure 2 This is a network structure diagram of the method of the present invention.

[0042] Figure 3 This is a comparison of the accuracy of the method of this invention with other methods using ResNet50 as the backbone network on the Seven-PointChecklist multimodal skin cancer dataset under partial labels.

[0043] Figure 4 This is the confusion matrix of test results between the method of this invention and other methods using ResNet50 as the backbone network on the first label of the Seven-PointChecklist multimodal skin cancer dataset.

[0044] Figure 5 These are sample images of two modalities from the Seven-Point Checklist multimodal skin cancer dataset, the main dataset used in this invention. Detailed Implementation

[0045] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0046] This invention is a frequency-domain denoising network-based medical image classification method based on tensor chain decomposition. The method includes a feature extractor, a tensor decomposer, a denoiser, and a classifier. To avoid overfitting and improve the model's generalization performance, common image enhancement methods such as random flipping are used on the training set. The feature extractor uses a backbone network composed of layers 1, 2, and 3 from ResNet50. The tensor decomposer decomposes the deep classification features using tensor chain decomposition, outputting one three-dimensional core feature tensor and two two-dimensional core feature tensors. The denoiser operates in the frequency domain, transforming the fusion tensor of the two modal core features to the frequency domain for denoising. Since different spatial locations in the frequency domain correspond to different frequency components, an attention module is learned in the frequency domain to filter out some noise and frequency components detrimental to generalization. The classifier is a simple module consisting of a series of average pooling, fully connected layers, and random deactivation operations, used to perform classification tasks using the fusion features from deep classification.

[0047] The method described in this invention involves a network iteration count of 1×10-1. 3 The initial learning rate is 1×10. 4 Each time 1×10 is completed 2 In the next iteration, the learning rate was adjusted to 1 / 2; the rank Rc and Rw of the tensor chain decomposition were 96 and 14, respectively; all images were resized to 224×224; and the batch size was 16.

[0048] Figure 1 , Figure 2 The flowchart and network structure diagram of the frequency domain denoising network multimodal medical image classification method based on tensor chain decomposition described in this invention include the following steps:

[0049] Step S1: Input the training set images I1 and I2 of the two modalities into two feature extractors with ResNet as the backbone network to extract the deep classification features F1 and F2 of the modal medical images. The network structure of the feature extractor is as follows: Figure 2 As shown in (1); the two modal images refer to medical images of different qualities composed of different devices, different staining agents or different perspectives that describe the same category. For example, the clinical images and dermoscopy images in the skin cancer dataset used in this invention belong to two different modal medical images; the backbone network can select different versions of ResNet according to the size and number of datasets. On the Seven-Point Checklist multimodal skin cancer dataset used in this invention, the backbone network selects layer 1, layer 2 and layer 3 of ResNet50, and the backbone networks for extracting features of the two modalities are completely consistent;

[0050] Step S2: Input the deep classification features F1 and F2 of the modal medical image into two tensor decomposers that use tensor chain decomposition to obtain the core feature tensors. The network structure of the tensor decomposers is as follows: Figure 2 As shown in (2) in the figure; the tensor chain decomposition is expressed as:

[0051]

[0052] Where X(i1,i2,…,i) N ) represents an N-dimensional tensor. Let represent the nth 3D core matrix. The first dimension of the first core matrix and the third dimension of the Nth core matrix in tensor chain decomposition are generally considered to be 1, meaning the first and Nth core matrices can be viewed as two-dimensional matrices. For 3D depth classification features, tensor chain decomposition is expressed as:

[0053]

[0054] Where X(c,h,w) represents the deep classification feature, and Rc and Rw represent the rank of the tensor chain decomposition in the channel and width dimensions, respectively. This represents the nth three-dimensional core feature tensor obtained from the decomposition. The three-dimensional feature tensor input of the tensor decomposer can be viewed as the product of a three-dimensional core feature tensor and two two-dimensional core feature tensors;

[0055] The core feature tensor The core feature tensor is generated by a series of convolutional layers. and The generation method is represented as:

[0056]

[0057]

[0058] Where GAP_c and GAP_w represent global average pooling, generating a vector that retains only one dimension of X(c,h,w). Rc and Rw vectors are concatenated to obtain... and

[0059] Step S3: The network structure of the noise reducer is as follows Figure 2 As shown in (3), the core feature tensors of the two modalities are fused using a feature-level fusion method, which is to obtain the core feature fusion tensor by weighted summation of the core feature tensors of the two modalities. The result is then input into a frequency domain noise reduction module to reduce noise generated by fusion and improve cross-domain generalization ability; the fusion of the two modal features can be expressed as:

[0060]

[0061]

[0062]

[0063] Wherein, λ1, λ2, and λ3 are learnable parameters that allow for dynamic adjustment of the weighting of the two modalities in the classification task during network training. The frequency domain noise reduction module specifically comprises:

[0064] Step S3-1: ... Perform a Fast Fourier Transform on each channel individually to obtain the corresponding frequency domain representation. It can be represented as:

[0065]

[0066] in, Let represent the core feature fusion tensor in the spatial domain, h and w represent the two dimensions of the core feature fusion tensor in the spatial domain, x and y represent the two dimensions of the core feature fusion tensor in the frequency domain, n and m increment from 0 to h and w respectively, j represents an imaginary number, and e represents 2.1718… an infinite non-repeating decimal.

[0067] Step S3-2: ... Input three consecutive frequency-domain spatial attention modules, because Different spatial locations correspond to different frequency components. By adaptively modulating these frequency components, noise and frequency components that are detrimental to generalization are filtered out. The frequency domain spatial attention module specifically includes the following steps:

[0068] Step S3-2-1: Spatial mapping is performed using a 1×1 convolutional layer, a BN layer, and a ReLU activation function;

[0069] Step S3-2-2: Feature aggregation is performed, consisting of average pooling and max pooling operations;

[0070] Step S3-2-3: Spatial mask is learned using a 7×7 convolutional layer, a BN layer, and a Sigmoid activation function;

[0071] Step S3-2-4: Multiply the outputs of step S3-2-1 and step S3-2-3 to complete the modulation of different frequency components;

[0072] Step S3-3: Perform an inverse fast Fourier transform on each channel of the output from step S3-2 individually.

[0073] Step S4: The network structure of the classifier is as follows Figure 2As shown in (4), the core feature fusion tensor output by the frequency denoising module in S3 is multiplied according to the tensor chain decomposition formula to obtain the fused feature; the output According to the formula:

[0074]

[0075] Will Multiplication yields fused features for classification.

[0076] The fused features are unfolded into a one-dimensional vector and input into a classifier consisting of a series of fully connected layers and random deactivation operations to obtain the classification result.

[0077] Step S5: Based on the loss function, train the entire model using classification labels to achieve classification of multimodal medical images; the loss function is the cross-entropy loss function. In the Seven-Point Checklist multimodal skin cancer dataset, the loss function is the mean of the losses of the eight labels, specifically expressed as:

[0078]

[0079] Where L represents the network loss, L diag ,L pn ,L str ,L pig ,L rs ,L dag ,L bwv L_vs is the cross-entropy loss for 8 labels.

[0080] Step S6: Based on the loss function, train the entire classification network, including the tensor chain decomposition module and the frequency domain noise reduction module, through classification labels to achieve classification of multimodal medical images.

[0081] To verify the effectiveness of this invention, its classification accuracy and confusion matrix were compared with those of five state-of-the-art multimodal classification methods—CAFNet, CentralNet, EmbeddingNet, HcCNN, and MMNet—on the Seven-Point Checklist multimodal skin cancer dataset. CAFNet proposed a novel cross-attention fusion network; CentralNet introduced a central network linking modality-specific networks, which not only provides general feature embeddings but also regularizes the modality-specific networks through multi-task learning; all methods were trained and tested on the same device using the same batch size, learning rate, number of iterations, and other parameters, and all used the same feature extractor and classifier.

[0082] The accuracy comparison of this invention with other methods using ResNet50 as the backbone network on the Seven-Point Checklist multimodal skin cancer dataset under partial labels is shown in the figure below. Figure 3 As shown, the labels include the diagnostic label diag, the other 5 labels, and the average of all 8 labels. Figure 3 The values ​​marked in light color represent the best performance among all methods, indicating that the multimodal medical image classification method based on tensor chain decomposition frequency domain denoising network described in this invention performs well on all labels.

[0083] The confusion matrix of the test results of the method described in this invention and other methods using ResNet50 as the backbone network on the first label (diag) of the Seven-Point Checklist multimodal skin cancer dataset shows that all methods perform well in the classification results of category nev, but perform poorly in the classification results of the other four categories, especially category sk. However, the method of this invention performs better than other methods in the other four categories. The reason for the poor performance in the other four categories is that the differences between different categories are small. Medical image classification is different from natural image classification. In medical images, different categories often differ only in some details. The distribution of images of different categories in the Seven-Point Checklist multimodal skin cancer dataset is not uniform, and there are fewer images of some analogies, which also leads to poor classification results in the other four categories. Because the denoising part of this invention has a cross-domain generalization effect, the performance of this invention in the other four categories with less data is better than some other methods.

[0084] This invention fuses depth image features by transforming them to their subspaces using tensor chain decomposition. This avoids or mitigates the problem that simple feature stitching often fails to effectively integrate complementary information from two modalities due to the different statistical characteristics typically present in different modalities, resulting in significant redundancy and potentially causing the curse of dimensionality in data processing. Furthermore, this invention improves the robustness of the method by using an attention mechanism in the frequency domain to suppress certain frequency components.

[0085] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A frequency domain denoising and multimodal medical image classification method based on tensor chain decomposition, characterized in that, The frequency domain denoising multimodal medical image classification method includes the following steps: Step 1: Input the training set images I1 and I2 of the two modalities into two backbone networks to extract the deep classification features F1 and F2 of the modal medical images; Step 2: Input the depth classification features F1 and F2 extracted from the modal medical images in Step 1 into two tensor chain decomposition modules to obtain core feature tensors, namely two three-dimensional core feature tensors ς. 11 ς 21 and four two-dimensional core feature tensors ς 12 ς 13 ς 22 ς 23 ; Step 3: The core feature tensors of the two modalities are fused using a feature-level fusion method. This involves weighted summation of the core feature tensors of the two modalities to obtain core feature fusion tensors ς1, ς2, and ς3. The obtained core feature fusion tensors ς1, ς2, and ς3 are then input into the frequency domain noise reduction module to reduce the noise generated by the fusion and improve the cross-domain generalization ability. Step 4: Multiply the core features output by the frequency domain noise reduction module 3 into tensors ς1, ς2, and ς3 according to the tensor chain decomposition formula to obtain the fused features; Step 5: Input the fused features obtained in Step 4 into a classifier consisting of a series of average pooling, fully connected layers, and random deactivation operations to obtain the classification result; Step 6: Based on the loss function, train the entire classification network, including the tensor chain decomposition module and the frequency domain noise reduction module, through classification labels to achieve classification of multimodal medical images.

2. The frequency domain denoising and multimodal medical image classification method based on tensor chain decomposition according to claim 1, characterized in that: In step 1, the backbone network uses layers 1, 2, and 3 of ResNet50 to extract deep classification features F1 and F2 from two modal medical images.

3. The frequency domain denoising and multimodal medical image classification method based on tensor chain decomposition according to claim 1, characterized in that: In step 2, when the deep classification features are three-dimensional, the tensor chain decomposition is represented as: in Representing deep classification features, and Let represent the rank of the tensor chain decomposition in the channel and width dimensions, respectively. This represents the nth three-dimensional core feature tensor obtained from the decomposition.

4. The frequency domain denoising and multimodal medical image classification method based on tensor chain decomposition according to claim 3, characterized in that: Core Feature Tensor Generated by a series of convolutional layers, and The generation method is represented as: in, and This indicates global average pooling, which generates only the elements that are retained. A one-dimensional vector and Concatenate the vectors to obtain and .

5. The frequency domain denoising and multimodal medical image classification method based on tensor chain decomposition according to claim 1, characterized in that: In step 3, the fused tensor representation of the two modal core features is as follows: in, These are learnable parameters that dynamically adjust the weight of the two modalities in the classification task during network training.

6. The frequency domain denoising and multimodal medical image classification method based on tensor chain decomposition according to claim 1, characterized in that, In step 3, the obtained core feature fusion tensors ς1, ς2, and ς3 are input into the frequency domain denoising module to reduce the noise generated by the fusion and improve the cross-domain generalization ability. Specifically, this includes the following steps: Step 3-1: Perform a Fast Fourier Transform on each channel of the core feature fusion tensor ς to obtain the corresponding frequency domain representation. , Represented as: in, This indicates that the core features of the spatial domain are fused tensors. and These represent the two dimensions of the core feature fusion tensor in the spatial domain. and These represent the two dimensions of the core feature fusion tensor in the frequency domain. and Incrementing from 0 to... and , represents an imaginary number, 2.1718… represents an infinite non-repeating decimal; Step 3-2, Input three consecutive frequency-domain spatial attention modules, and filter out noise and frequency components that are not conducive to generalization by adaptively modulating different frequency components; Step 3-3: Perform an inverse fast Fourier transform on each channel of the output from step 3-2.

7. The frequency domain denoising and multimodal medical image classification method based on tensor chain decomposition according to claim 6, characterized in that: In step 3-2, Inputting three consecutive frequency-domain spatial attention modules, and adaptively modulating different frequency components, specifically includes the following steps: Step 3-2-1: Spatial mapping is performed using a 1×1 convolutional layer, a BN layer, and a ReLU activation function; Step 3-2-2: Feature aggregation is performed, consisting of average pooling and max pooling operations; Step 3-2-3: Spatial mask is learned using a 7×7 convolutional layer, a BN layer, and a Sigmoid activation function; Step 3-2-4: Multiply the outputs of Step 3-2-1 and Step 3-2-3 to complete the modulation of different frequency components.

8. The frequency domain denoising and multimodal medical image classification method based on tensor chain decomposition according to claim 1, characterized in that: In step 4, the core feature fusion tensors ς1, ς2, and ς3 output from step 3 are multiplied together according to the following formula to obtain the fused features used for classification. : 。

Images