Multimodal esophageal cancer recognition method and system based on frequency characteristics

Abstract

The application relates to the technical field of medical image processing, and discloses a multi-modal esophageal cancer recognition method and system based on frequency characteristics, which comprises a data preprocessing unit, a multi-modal feature extraction unit, a feature fusion unit and a classification decision unit.The data preprocessing unit is used for processing input pathological images, endoscopic images and medical record text data.The multi-modal feature extraction unit is used for performing multi-scale feature extraction on the pathological images and the endoscopic images by using an adaptive frequency multi-scale feature fusion module.The feature fusion unit is used for fusing the extracted multi-modal features by using a frequency cross fusion module.The classification decision unit is used for inputting the fused features into a classifier to make a final diagnosis decision.Through fast Fourier transform and inverse fast Fourier transform, in combination with the adaptive frequency multi-scale feature fusion module and the frequency cross fusion module, the efficiency and accuracy of feature fusion can be improved.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a multimodal esophageal cancer identification method and system based on frequency features. Background Technology

[0002] In the diagnosis of esophageal cancer, endoscopic images and pathological images are two important diagnostic tools. Endoscopic images provide real-time images of the esophageal lining, while pathological images provide microscopic structural information of tissue samples. The fusion of these two types of image information, combined with patient metadata, can significantly improve diagnostic accuracy. However, there are significant differences between endoscopic and pathological images in terms of spatial resolution and metadata content, which poses a challenge to multimodal fusion. In addition, with the rapid development of artificial intelligence and deep learning technologies, multimodal diagnostic methods based on image and text data have also received widespread attention.

[0003] Traditional multimodal diagnostic methods typically rely on multi-branch structures to optimize and fuse information from different modalities at the feature level, or to allocate modal weights in stages at the decision layer. While these methods improve diagnostic performance to some extent, they also have some problems, such as asymmetry in feature levels between different modalities, cumbersome computational steps, and insufficient information fusion. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a multimodal esophageal cancer identification method and system based on frequency features. This solves the problem that traditional multimodal diagnostic methods suffer from high computational complexity, resulting in long model training and inference times that affect the efficiency and scalability of practical applications.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multimodal esophageal cancer identification system based on frequency features, comprising a data preprocessing unit, a multimodal feature extraction unit, a feature fusion unit, and a classification decision unit. The data preprocessing unit processes input pathological images, endoscopic images, and medical record text data. The multimodal feature extraction unit uses an adaptive frequency multiscale feature fusion module (AFCFBl ock) to extract multiscale features from pathological images and endoscopic images, and uses a metadata encoding module to extract features from medical record text data. The feature fusion unit uses a frequency cross-fusion module (FrequencyCrossFusionBl ock) to fuse the extracted multimodal features, enabling effective integration of different modal information. The classification decision unit inputs the fused features into a classifier for final diagnostic decision-making.

[0006] Preferably, the data preprocessing unit includes a pathological image and endoscopic image preprocessing module and a medical record text data encoding module. The pathological image and endoscopic image preprocessing module includes normalization processing, resizing, and data augmentation. The normalization processing refers to normalizing the input pathological images and endoscopic images to make the image data uniform, including adjusting the brightness and contrast of the images to make them within the same dynamic range. The resizing refers to adjusting the pathological images and endoscopic images to a fixed size and resolution. The resizing methods include scaling, cropping, and padding. The data augmentation refers to using data augmentation techniques—rotation, flipping, scaling, and noise addition.

[0007] Preferably, the medical record text data encoding module includes one-hot encoding, text cleaning, and word embedding. One-hot encoding refers to converting the category information in the medical record text data into binary vectors to effectively represent the category data. Text cleaning refers to cleaning the medical record text data, including removing stop words, punctuation marks, and special characters. Word embedding refers to using a pre-trained word embedding model - Word2Vec or GL0Ve - to convert the medical record text data into a low-dimensional dense vector representation. These word embedding vectors are used to capture the semantic relationships between words.

[0008] Preferably, the adaptive frequency multi-scale feature fusion module (AFCFB lock) includes frequency domain transformation, multi-scale feature extraction, inverse frequency domain transformation, and feature fusion. The frequency domain transformation first performs Fast Fourier Transform (FFT) on the pathological and endoscopic images to transform the image data from the time domain to the frequency domain. The FFT is used to separate different frequency components of the image. The multi-scale feature extraction refers to extracting frequency domain features at different scales in the frequency domain using multi-scale feature extraction technology. These features include low-frequency features and high-frequency features, which correspond to the overall structure and detailed information of the image, respectively. The inverse frequency domain transformation refers to using Inverse Fast Fourier Transform (IFFT) to transform the frequency domain features back to the time domain. The inverse Fourier Transform is used to restore the processed frequency domain features to the time domain image so that it is visualized. The feature fusion refers to combining a cross-attention mechanism to fuse and filter multi-scale features to extract effective frequency domain features. The cross-attention mechanism establishes connections between different modalities.

[0009] Preferably, the metadata encoding module includes One-hot encoding and a multilayer perceptron (MLP). The One-hot encoding performs one-hot encoding on the medical record text data and converts the category data into a binary vector. The MLP uses the multilayer perceptron to extract features from the encoded medical record text data and generate a text feature vector.

[0010] Preferably, the frequency cross-fusion module includes frequency domain transformation, cross-attention mechanism, and inverse frequency domain transformation. The frequency domain transformation is used to receive image features from the AFCFB lock and text features from the metadata encoding module. First, the image features and text features are subjected to Fast Fourier Transform (FFT) respectively, and then transformed to the frequency domain. The cross-attention mechanism refers to fusing frequency domain features of different modalities in the frequency domain. The cross-attention mechanism is used to dynamically adjust the attention weights according to the correlation of the input features. The inverse frequency domain transformation refers to using Inverse Fast Fourier Transform (IFFT) to transform the fused frequency domain features back to the time domain to generate a fused feature vector. The inverse Fourier transform is used to preserve the global information of the frequency domain features.

[0011] Preferably, the classification decision unit includes a visual converter ViT and a classification decision. The visual converter ViT uses a pre-trained ViT and a VisionTransformer to fine-tune the classifier and classify the fused features. The visual converter is used to effectively process image data and extract high-level feature representations. The classification decision refers to the classifier outputting the final diagnostic result, including the presence or absence of esophageal cancer and its specific classification, by classifying the input features.

[0012] A preferred method for identifying esophageal cancer based on frequency features includes the following specific steps:

[0013] S1: Data input, acquire pathological images, endoscopic images and medical record text data, and perform data preprocessing;

[0014] S2: Feature extraction, using ACFCBLOCK to extract frequency domain features from image data, and using the metadata encoding module to extract features from text data;

[0015] S3: Feature fusion, using FrequencyCrossFusionBlock to perform frequency domain fusion on the extracted multimodal features to generate a fused feature vector;

[0016] S4: Classification decision, which inputs the fused feature vector into the classifier, makes a classification decision, and outputs the diagnostic result.

[0017] This invention provides a multimodal esophageal cancer identification method and system based on frequency features. It has the following beneficial effects:

[0018] 1. This invention uses Fast Fourier Transform (FFT) and Inverse Fast Fourier Transform (IFFT) to transform multimodal features to the frequency domain for fusion. Combined with the Adaptive Frequency Multiscale Feature Fusion Module (AFCFB lock) and the Frequency Cross Fusion Module (FrequencyCrossFusionLock), it achieves feature selection and aggregation in the frequency domain, effectively reducing redundant information, thereby improving the efficiency and accuracy of feature fusion.

[0019] 2. The ACFFBL ck and FrequencyCrossFusionBl ck of the present invention achieve more efficient feature extraction and fusion than the traditional self-attention mechanism (whose computational complexity is quadratic O(N^2)) through frequency domain feature transformation and cross attention mechanism, which greatly reduces the computational complexity and thus improves the computational efficiency and scalability of the model.

[0020] 3. This invention is the first to fuse three modalities of pathological images, endoscopic images, and medical record text data for the diagnosis of esophageal cancer. Through a metadata encoding module and a multimodal feature fusion network (FreqFuse-Net), it achieves comprehensive integration and diagnosis of multimodal data, thereby improving the accuracy and robustness of esophageal cancer diagnosis and providing more comprehensive information support. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the text-visual cross-knowledge joint model of the present invention;

[0022] Figure 2 This is a schematic diagram of the architecture of the frequency-feature-based multimodal esophageal cancer identification system of the present invention;

[0023] Figure 3 This is a schematic diagram of the data preprocessing unit architecture of the present invention;

[0024] Figure 4 This is a schematic diagram of the multimodal feature extraction unit architecture of the present invention;

[0025] Figure 5 This is a schematic diagram of the feature fusion unit architecture of the present invention;

[0026] Figure 6 This is a schematic diagram of the classification decision unit architecture of the present invention;

[0027] Figure 7 This is a flowchart of the multimodal esophageal cancer identification method based on frequency features of the present invention. Detailed Implementation

[0028] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Please see the appendix Figure 1 - Appendix Figure 7 This invention provides a multimodal esophageal cancer recognition system based on frequency features, including a data preprocessing unit, a multimodal feature extraction unit, a feature fusion unit, and a classification decision unit. The data preprocessing unit processes the input pathological images, endoscopic images, and medical record text data. The multimodal feature extraction unit uses the adaptive frequency multiscale feature fusion module ACFFBLock to extract multiscale features from the pathological images and endoscopic images, and uses the metadata encoding module to extract features from the medical record text data. The feature fusion unit uses the frequency cross-fusion module FrequencyCrossFusionBlock to fuse the extracted multimodal features, enabling effective integration of different modal information. The classification decision unit inputs the fused features into a classifier for final diagnostic decision.

[0030] Specifically, the data preprocessing unit processes the input pathological images, endoscopic images, and medical record text data to ensure the consistency and applicability of the input data.

[0031] The data preprocessing unit includes a pathological image and endoscopic image preprocessing module and a medical record text data encoding module. The pathological image and endoscopic image preprocessing module includes normalization processing, resizing, and data augmentation. Normalization processing refers to normalizing the input pathological images and endoscopic images to make the image data uniform, including adjusting the brightness and contrast of the images to keep them within the same dynamic range. Resizing refers to adjusting the pathological images and endoscopic images to a fixed size and resolution. Resizing methods include scaling, cropping, and padding. Data augmentation refers to using data augmentation techniques—rotation, flipping, scaling, and noise addition.

[0032] Specifically, data augmentation techniques—rotation, flipping, scaling, and noise addition—can increase the diversity of training data and improve the robustness and generalization ability of the model.

[0033] The medical record text data encoding module includes one-hot encoding, text cleaning, and word embedding. One-hot encoding refers to converting the category information in the medical record text data into binary vectors to effectively represent the category data. Text cleaning refers to cleaning the medical record text data, including removing stop words, punctuation marks, and special characters. Word embedding refers to using a pre-trained word embedding model - Word2Vec or GL OVe - to convert the medical record text data into a low-dimensional dense vector representation. These word embedding vectors are used to capture the semantic relationships between words.

[0034] Specifically, text cleaning can ensure the purity and consistency of data, while word embedding can improve feature representation capabilities.

[0035] The adaptive frequency multi-scale feature fusion module AFCFBl ock includes frequency domain transformation, multi-scale feature extraction, inverse frequency domain transformation, and feature fusion. Frequency domain transformation first performs Fast Fourier Transform (FFT) on pathological and endoscopic images to transform the image data from the time domain to the frequency domain. The Fourier Transform is used to separate different frequency components of the image. Multi-scale feature extraction refers to extracting frequency domain features at different scales in the frequency domain using multi-scale feature extraction techniques. These features include low-frequency features and high-frequency features, which correspond to the overall structure and detailed information of the image, respectively. Inverse frequency domain transformation refers to using Inverse Fast Fourier Transform (IFFT) to transform the frequency domain features back to the time domain. The Inverse Fourier Transform is used to restore the processed frequency domain features to the time domain image so that it is visualized. Feature fusion refers to combining a cross-attention mechanism to fuse and filter multi-scale features to extract effective frequency domain features. The cross-attention mechanism establishes connections between different modalities.

[0036] The metadata encoding module includes One-hot encoding and Multilayer Perceptron (MLP). One-hot encoding encodes the medical record text data one-hot, converting the category data into binary vectors. The MLP uses the multilayer perceptron to extract features from the encoded medical record text data and generate text feature vectors.

[0037] The frequency cross-fusion module includes frequency domain transformation, cross-attention mechanism, and inverse frequency domain transformation. Frequency domain transformation is used to receive image features from the ACFFBL ock and text features from the metadata encoding module. First, fast Fourier transform (FFT) is performed on the image features and text features respectively, and then they are transformed to the frequency domain. The cross-attention mechanism refers to the fusion of frequency domain features of different modalities in the frequency domain. The cross-attention mechanism is used to dynamically adjust the attention weights according to the correlation of the input features. Inverse frequency domain transformation refers to the use of inverse fast Fourier transform (IFFT) to transform the fused frequency domain features back to the time domain to generate a fused feature vector. Inverse Fourier transform is used to preserve the global information of the frequency domain features.

[0038] The classification decision unit includes a visual converter (ViT) and a classification decision unit. The visual converter (ViT) uses a pre-trained ViSiOnTransformer to fine-tune the classifier and classify the fused features. The visual converter is used to effectively process image data and extract high-level feature representations. The classification decision unit refers to the classifier outputting the final diagnostic result by classifying the input features, including the presence or absence of esophageal cancer and its specific classification.

[0039] A frequency-based multimodal esophageal cancer identification method includes the following specific steps:

[0040] S1: Data input, acquire pathological images, endoscopic images and medical record text data, and perform data preprocessing;

[0041] S2: Feature extraction, using AFCFB lock to extract frequency domain features from image data, and using metadata encoding module to extract features from text data;

[0042] S3: Feature fusion, using FrequencyCrossFusionBlock to perform frequency domain fusion on the extracted multimodal features to generate a fused feature vector;

[0043] S4: Classification decision, which inputs the fused feature vector into the classifier, makes a classification decision, and outputs the diagnostic result;

[0044] Specifically, alternatives and scalability:

[0045] Multimodal feature fusion based on wavelet transform:

[0046] Alternative approach description: Similar to the Fast Fourier Transform (FFT), wavelet transform can be used for frequency domain feature extraction and fusion. Wavelet transform can provide better time-frequency localization characteristics, which helps to capture transient features and discontinuities in images.

[0047] Alternative steps:

[0048] Wavelet transform is applied to pathological and endoscopic images. Wavelet transform can effectively decompose the different scale components of the image and capture the local features of the image.

[0049] Feature filtering and fusion are performed in the wavelet domain. By utilizing the multi-resolution characteristics of wavelet transform, features from different modes are filtered and fused to improve feature representation capabilities.

[0050] The fused features are converted back to the time domain by inverse wavelet transform. Inverse wavelet transform can restore the processed wavelet domain features to the time domain image and preserve the structural information of the original image.

[0051] Advantages: Wavelet transform has advantages in processing non-stationary signals and can better capture detailed features in images. Compared with Fourier transform, wavelet transform has better time-frequency localization characteristics, which can improve the effect of feature fusion.

[0052] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal esophageal cancer identification system based on frequency features, characterized in that, It includes a data preprocessing unit, a multimodal feature extraction unit, a feature fusion unit, and a classification decision unit; The data preprocessing unit is used to process the input pathological images, endoscopic images, and medical record text data. The multimodal feature extraction unit includes an adaptive frequency multiscale feature fusion module (AFCFBlock) and a metadata encoding module; The Adaptive Frequency Multi-Scale Feature Fusion Module (AFCFBlock) performs Fast Fourier Transform (FFT) on the pathological image and the endoscopic image, converting them to the frequency domain. Within the frequency domain, it extracts multi-scale frequency domain features, including low-frequency and high-frequency features. The low-frequency features correspond to the overall structure of the image, while the high-frequency features correspond to detailed information. The module then performs Inverse Fast Fourier Transform (IFFT) on these multi-scale frequency domain features and, combined with a cross-attention mechanism, establishes a connection between the multi-scale features of the pathological image and the multi-scale features of the endoscopic image. Finally, it fuses and filters these multi-scale features to generate image features. The metadata encoding module includes one-hot encoding and a multilayer perceptron (MLP). One-hot encoding is used to convert the category data in the medical record text data into binary vectors, and the multilayer perceptron (MLP) is used to extract features from the encoded medical record text data to generate text feature vectors. The feature fusion unit includes a frequency cross-fusion module, FrequencyCrossFusionBlock. FrequencyCrossFusionBlock receives image features generated by the adaptive frequency multi-scale feature fusion module, AFCFBlock, and text feature vectors generated by the metadata encoding module. It performs Fast Fourier Transform (FFT) on both the image features and the text feature vectors to convert them to the frequency domain. In the frequency domain, it uses a cross-attention mechanism to dynamically adjust the attention weights based on the correlation between the image features and the text feature vectors, fusing the image features and text feature vectors. Finally, it performs an Inverse Fast Fourier Transform (IFFT) on the fused frequency domain features to generate a fused feature vector. The classification decision unit is used to input the fused feature vector into the classifier for classification decision and output the esophageal cancer identification result.

2. The multimodal esophageal cancer identification system based on frequency features according to claim 1, characterized in that, The data preprocessing unit includes a pathological image and endoscopic image preprocessing module and a medical record text data encoding module. The pathological image and endoscopic image preprocessing module is used to perform normalization, resizing, and data enhancement on the pathological images and the endoscopic images. The normalization process includes adjusting the brightness and contrast of the images to bring the pathological images and the endoscopic images into the same dynamic range. The resizing includes scaling, cropping, and padding. The data enhancement includes rotation, flipping, scaling, and noise addition.

3. The multimodal esophageal cancer identification system based on frequency features according to claim 2, characterized in that, The medical record text data encoding module includes one-hot encoding, text cleaning, and word embedding; The one-hot encoding is used to convert the category information in the medical record text data into a binary vector. The text cleaning includes removing stop words, punctuation marks, and special characters. The word embedding is used to convert the medical record text data into a low-dimensional dense vector representation using a pre-trained word embedding model Word2Vec or GloVe.

4. The multimodal esophageal cancer identification system based on frequency features according to claim 1, characterized in that, The classification decision unit includes a visual converter (ViT) and a classifier. The visual converter (ViT) is used to perform feature processing on the fused feature vector, and the classifier is used to output the presence or absence of esophageal cancer and its specific classification based on the processed fused feature vector.

5. A multimodal esophageal cancer identification method based on frequency features, characterized in that, Includes the following steps: S1, acquire pathological images, endoscopic images and medical record text data, and perform data preprocessing on the pathological images, endoscopic images and medical record text data; S2, the Adaptive Frequency Multi-Scale Feature Fusion Module (AFCFBlock) is used to perform Fast Fourier Transform (FFT) on the pathological image and the endoscopic image, converting them to the frequency domain. Multi-scale frequency domain features, including low-frequency and high-frequency features, are extracted in the frequency domain. The low-frequency features correspond to the overall structure of the image, and the high-frequency features correspond to the detailed information of the image. Inverse Fast Fourier Transform (IFFT) is then performed on the multi-scale frequency domain features, and a cross-attention mechanism is used to establish a connection between the multi-scale features of the pathological image and the multi-scale features of the endoscopic image. The multi-scale features are then fused and filtered to generate image features. The category data in the medical record text data is one-hot encoded using the metadata encoding module, converting the category data into a binary vector. Then, the encoded medical record text data is feature extracted using a multilayer perceptron (MLP) to generate a text feature vector. S3, using the FrequencyCrossFusionBlock module to receive the image features and the text feature vector, performing Fast Fourier Transform (FFT) on the image features and the text feature vector respectively, converting the image features and the text feature vector to the frequency domain, dynamically adjusting the attention weights in the frequency domain based on the correlation between the image features and the text feature vector using a cross-attention mechanism, fusing the image features and the text feature vector, and performing Inverse Fast Fourier Transform (IFFT) on the fused frequency domain features to generate a fused feature vector; S4, input the fused feature vector into the classifier for classification decision, and output the esophageal cancer identification result.

Images