An image classification system based on cross-modal feature fusion

Abstract

The application discloses an image classification system based on cross-modal feature fusion, which comprises an extraction module, a fusion module and a model training module, the extraction module extracts features of images and texts from medical images and diagnosis reports; the fusion module adopts a cross-modal attention module to fuse features of the images and the texts; and the model training module trains the fused data by using a bidirectional long short-term memory network module based on a sparse gate. The cross-modal attention mechanism can adaptively assign weights to different modal data, thereby improving the attention of the model to key information, better capturing the correlation between different modalities, and improving the fusion effect of multi-modal data; the sparse gate is applied to the gating unit of the Bi-LSTM, the sparse gate reduces the calculation complexity of the model and improves the robustness of the model, can effectively reduce the number of parameters, reduce the calculation cost of the model, and improve the operation efficiency of the model.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification and relates to an image classification system based on cross-modal feature fusion. Background Art

[0002] According to the World Health Organization (WHO), lung cancer is one of the most common cancers worldwide, resulting in approximately 1.8 million deaths annually (WHO, 2020). Data from the Global Burden of Disease Study indicate that chronic obstructive pulmonary disease (COPD) is the third leading cause of disability worldwide, resulting in approximately 3 million deaths annually. Therefore, the diagnosis and treatment of lung diseases are extremely important.

[0003] Radiographs are the most cost-effective diagnostic tool for detecting lung diseases. Diagnosing such diseases from chest X-rays requires highly skilled radiologists, while manual detection of lung diseases is a time-consuming process often subject to subjective variability, which can delay diagnosis and treatment. Computer-aided diagnosis (CAD) holds great potential in clinical practice, enabling accurate diagnosis of lung diseases in a short timeframe, reducing physician workload, lowering misdiagnosis rates, and significantly improving diagnostic efficiency. Deep learning has been widely studied due to its universal applicability to problems involving automatic feature extraction and classification. Convolutional neural network (CNN)-based evaluations have been widely used for image classification and object detection. CNNs employ spatial filters that automatically capture information about structures embedded in images. Initially, pre-trained deep learning systems, such as AlexNet, the Visual Geometry Group's DL networks (VGG16 and VGG19), and ResNet50, were used to classify selected radiograph images into normal and abnormal classes using a softmax classifier. The paper "Anthimopoulos M, Christodoulidis S, Ebner L, Christe A and Mougiakakou S. 2016. Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Transactions on Medical Imaging, 35(5): 1207-1216" uses CNN to analyze medical images to determine the severity of disease in different organs. The paper "Kawahara J, BenTaieb A and Hamarneh G. 2016. Deep features to classify skin lesions. In: 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), 1397-1400 (Classification of skin lesions using deep features, 2016 13th International Symposium on Biomedical Imaging (ISBI) of the Institute of Electrical and Electronics Engineers)" studied the combination of local information and global contextual information and designed image analysis architectures at different scales.Setio et al. (2016) "Setio AAA, Ciompi F, Litjens G, Gerke P, Jacobs C, van Riel SJ, Wille MMW, Naqibullah M, Sanchez CI and van Ginneken B. 2016. Pulmonary nodule detection in CT images: false positive reduction using multi-view convolutional networks. IEEE Transactions on Medical Imaging, 35(5): 1160-1169" uses 3DCNN to enhance classification performance. Although the above methods have good results in processing medical images, they fail to consider multimodal situations.

[0004] Scholars have proposed various methods to process multimodal medical data. For example, some studies have used convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to model medical images and time series data. The paper "Tan W, Tiwari P, Pandey HM, Moreira C and Jaiswal A K. 2020. Multimodal medical image fusion algorithm in the era of big data. Neural Computing and Applications, 1-21" proposes a multimodal medical image fusion algorithm for use in the big data era. By combining multiple medical images of different modalities and using deep learning algorithms to fuse the information, the accuracy of diagnosis can be improved. The document "Apostolopoulos ID and Mpesiana TA.2020.Covid-19: automatic detection from x-ray images utilizing transfer learning with convolutional neural networks.Physical and Engineering Sciencesin Medicine, 43, 635-640 (Covid-19: Automatic detection from X-ray images using transfer learning of convolutional neural networks, Medical Physics and Engineering Sciences)" In order to alleviate the problem of small number of samples, the transfer learning method was adopted to evaluate the recognition ability of several state-of-the-art pre-trained convolutional neural networks (VGG, Inception, etc.) in chest x-ray images, and transfer the knowledge extracted from the pre-trained model to the model to be trained. The experiment achieved good results. The traditional method of combining CNN and RNN ignores the calculation speed of its network, and it is also particularly important to improve the speed of the diagnostic system.

[0005] The literature "Hochreiter S, Schmidhuber J. Long short-term memory [J]. Neural computation, 1997, 9 (8): 1735-1780. (Long short-term memory, neural computing)" proposed a long short-term memory network (LSTM), which has the advantages of long-term memory ability, preventing gradient problems, adapting to variable-length sequences, processing information of different time scales, and generalization ability. The literature "Ni Yangfan, Yang Yuanyuan, Xie Zhe, Zheng Dezhong, Wang Weidong. 2022. Multi-feature extraction method of pulmonary nodules based on LSTM and attention structure. Journal of Shanghai Jiaotong University, 56 (08): 1078-1088" integrates the shared features between tasks through the attention mechanism to improve the feature extraction effect of the current task. The LSTM structure classifier can effectively screen the shared features between tasks and improve the information transmission efficiency of the model. The paper "Kalita A. In-depth understanding of LSTM and its recent advances in lung disease diagnosis[J]. World Journal of Advanced Research and Reviews, 2022, 14(3): 517-522. (In-depth understanding of LSTM and its recent advances in lung disease diagnosis, World Journal of Advanced Research and Reviews)" conducts an in-depth study of LSTM through several practical examples and explores the latest advances of LSTM in the diagnosis of COVID-19 and other lung diseases. The paper "Dastider AG, Sadik F and Fattah S A. 2021. An integrated autoencoder-based hybrid CNN-LSTM model for COVID-19 severity prediction from lung ultrasound. Computers in Biology and Medicine, 132, 104296" predicts the severity of lung ultrasound by introducing a long short-term memory (LSTM) layer after the proposed CNN architecture. The results show that the classification performance has been greatly improved, with an average improvement of 7-12%, which is about 17% higher than the traditional DenseNet architecture.The literature "Chowdary G J.2021.Class dependency based leaming using Bi-LSTM coupled with the transfer leaming of VGG16 for the diagnosis of Tuberculosis from chest x-rays.arXiv preprint arXiv:2108.04329 (Using Bi-LSTM and transfer learning of VGG16 for class dependency-based learning to diagnose tuberculosis from chest X-rays, arXiv preprint)" The transfer learning of VGG16 combined with bidirectional LSTM extracts high-level discriminative features from the segmented lung area, and then uses a fully connected layer for classification, which improves the diagnosis accuracy of tuberculosis on the Schezien and Montgomery datasets by 0.7% and 11.68%, respectively. The paper "Lv Qing, Zhao Kui, Cao Jilong, Wei Jingfeng. 2022. Research and prediction of lung diseases based on text and image. Acta Automatica Sinica, 48(02): 531-538" proposed a lung cancer classification method based on the combination of image and text. It introduced electronic medical record information and used Multi-head attention and Bi-LSTM to model text, further improving the performance of the image classification model. However, due to its high computational complexity and large number of parameters, it increased the computational burden. Hong Xin et al. (2023) "Hong Xin, Huang Kaifeng, Yang Chenhui. 2023. Alzheimer's disease prediction CTISS model based on Bi-ConvLSTM temporal feature extraction. Chinese Journal of Image and Graphics, 28(04): 1146-1156" proposed an Alzheimer's disease prediction CTISS model based on Bi-ConvLSTM temporal feature extraction. Through the temporal convolution bidirectional long short-term memory model Bi-ConvLSTM and the attention mechanism, temporal features are extracted on the hierarchical sections of brain images.

[0006] Although these methods have achieved certain results, they often fail to fully utilize the correlation information between modalities and suffer from low computational efficiency when processing large-scale data. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide an image classification system based on cross-modal feature fusion, which utilizes the correlation information between modalities and solves the problem of low computational efficiency when processing large-scale data.

[0008] The technical solution adopted by the present invention is: an image classification system based on cross-modal feature fusion, comprising:

[0009] Extraction module, which extracts features of images and texts from medical images and diagnostic reports;

[0010] Fusion module, which uses a cross-modal attention module to fuse the features of image and text;

[0011] The model training module uses a bidirectional long short-term memory network module based on sparse gates to train the fused data.

[0012] Furthermore, the above extraction module uses the pre-trained models ResNet50 and BERT to perform feature extraction images and texts on medical images and diagnostic reports, respectively.

[0013] Furthermore, the fusion method of the above-mentioned cross-modal attention module adopts a feature fusion method based on the cross-modal attention mechanism. The method is: the cross-modal attention mechanism is expressed as follows:

[0014]

[0015] Among them, CMA (cross-modal attention) represents the cross-modal attention mechanism, I and M represent different modalities respectively, and the role of M is to enhance the representation of I, and represents the dot product of the matrix. Represents the scaling factor. The fusion of the cross-modal attention mechanism involves the interaction between images and text. The fusion formula of the cross-modal attention mechanism is as follows:

[0016]

[0017] Among them, A i，j represents the attention weight of the i-th modality to the j-th modality, S i，j is the similarity score between two modalities, N represents the total number of modalities; for each modality i, the attention weight of i to j is calculated by calculating the similarity score between the modality i and other modalities j, and then converting these scores into probability distributions.

[0018] Furthermore, the above-mentioned bidirectional long short-term memory network module based on sparse gate is a network that combines Bi-LSTM with sparse gate. In the model combining sparse gate and Bi-LSTM, Bi-LSTM is first used to encode the sequence, and then the encoded sequence is sparsed by the sparse gate. The bidirectional long short-term memory network module based on sparse gate adds a sparse gate after the input gate, forget gate and output gate in Bi-LSTM to control the information flow at each time step in the model. The specific implementation method of the bidirectional long short-term memory network module based on sparse gate is: at each time step t, the input of the sparse gate is the output h of Bi-LSTM at that time step. tThe output of the sparse gate is a binary vector g_t∈[0,1], indicating whether h t Perform sparseness, the output g of the sparse gate t The dot product operation will be performed with the output of the input gate, forget gate and output gate in Bi-LSTM to obtain the output of the sparse gate. Suppose the output of the input gate, forget gate and output gate at time step t are i t , f t , o t , then the output of the sparse gate is:

[0019]

[0020] Among them, ⊙ represents the dot product operator, Represent the outputs of the input gate, forget gate, and output gate after sparsification, respectively.

[0021] Furthermore, the above Bi-LSTM includes two LSTMs, one LSTM traverses the input sequence forward in time, and the other LSTM traverses the input sequence backward in time, which are respectively called forward LSTM and backward LSTM. The hidden state of the forward LSTM is Only the input information of the current moment and the previous moment is considered, and the hidden state of the backward LSTM Considering only the input information at the current moment and the following moments, the outputs of the two LSTMs are concatenated to form the final Bi-LSTM output.

[0022]

[0023] Among them, “,” represents the vector splicing operation.

[0024] Furthermore, the above sparse gate is implemented by introducing a sparse matrix. Assuming that the input data is X = [x1, x2, ..., x n ]∈R n×d , where n represents the sequence length and d represents the feature dimension of each time step, then the sparse gate is expressed as:

[0025] g i，j =σ(a j (x i )+b j )·s i，j (4)

[0026] Among them, a j is a function that extracts the input x i The features in b j is the bias term, σ is the sigmoid function, s i，j Is a binary matrix, indicating sparsity, when si，j = 1, it means that the jth feature of the i-th time step participates in the operation; when s i，j = 0, it means that the jth feature of the i-th time step is ignored; i，j As a probability variable, the parameters are then learned by maximizing the marginal likelihood of the model or minimizing the reconstruction error. Specifically, the sparse coding algorithm of LISTA is used to optimize the model:

[0027]

[0028] Where X represents the given input data matrix The goal of sparse coding is to learn a dictionary The purpose is to generate a sparse code of the input data λ represents the regularization coefficient of the l1-norm.

[0029] To solve Equation (5), the traditional method is to alternately optimize W and S, which corresponds to two optimization processes: dictionary learning and sparse approximation. Specifically, by fixing S, Equation (5) is reduced to the following l2-constrained optimization problem:

[0030]

[0031] By fixing W, Equation (5) reduces to a sparse approximation problem that aims to represent the input x by a linear combination of W as follows:

[0032]

[0033] The iterative hard thresholding (ISTA) algorithm is one of the most popular solvers. It uses the iterative hard thresholding (ISTA) algorithm to decompose the objective of (7) into two parts: the differentiable part It is updated by the gradient descent method, and the other part is the l1 regularization part, which is updated by the hard threshold operator. The update formula is as follows:

[0034]

[0035] Among them, s (t) represents the sparse coding of the tth iteration, sh (λτ) =sign(s)(|s|-λT) is the definition of the contraction function, represents the differentiable part, τ is the coefficient, and the solution of formula (8) is achieved by the following update rule:

[0036]

[0037] Among them, W u =I-τW T W, W v=τW T .

[0038] Beneficial effects of the present invention: Compared with the prior art, in response to the limitations of these existing methods, the present invention introduces a cross-modal attention mechanism and a sparse gate to improve multimodal medical data analysis. First, based on the cross-modal attention mechanism, a method is designed that can dynamically learn the correlation between different modalities. This mechanism can adaptively assign weights to different modal data, thereby increasing the model's attention to key information. By introducing the attention mechanism, the correlation between different modalities can be better captured, thereby improving the fusion effect of multimodal data. Secondly, the sparse gate is applied to the gating unit of Bi-LSTM to further optimize the performance of the model. The role of the sparse gate is to filter out unimportant information, thereby reducing the computational complexity of the model and improving the robustness of the model. By introducing the sparse gate mechanism, the number of parameters can be effectively reduced, the computational overhead of the model can be reduced, and the operating efficiency of the model can be improved.

[0039] The innovation of this invention lies in the application of a cross-modal attention mechanism and sparse gates to multimodal medical data analysis. This approach fully exploits the correlation information between different modalities and improves the computational efficiency of the model when processing large-scale data. Compared to existing methods, the system of this invention can better mine the potential features of multimodal data, thereby improving model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is the structural diagram of the proposed model;

[0041] Figure 2 This is the Bi-LSTM network framework diagram;

[0042] Figure 3 This is the SG-LSTM network framework diagram;

[0043] Figure 4 is the confusion matrix heat map;

[0044] Figure 5 ROC curve diagram comparing the model of the present invention with other baseline models. DETAILED DESCRIPTION

[0045] The present invention will be further described below with reference to specific embodiments.

[0046] Example 1: Figure 1-5 As shown, an image classification system based on cross-modal feature fusion includes:

[0047] Extraction module, which extracts features of images and texts from medical images and diagnostic reports;

[0048] Fusion module, which uses a cross-modal attention module to fuse the features of image and text;

[0049] The model training module uses a bidirectional long short-term memory network module based on sparse gates to train the fused data.

[0050] The main goal of this paper is to perform feature extraction and fusion for medical images and medical reports, and based on the Bi-LSTM model, a Bi-LSTM model based on sparse gate is proposed to improve the accuracy and efficiency of feature processing.

[0051] First, the ResNet50 model is used to extract features from medical images. ResNet (residual network) is a deep convolutional neural network architecture that effectively addresses the vanishing and exploding gradient problems in deep network training. By using ResNet, useful feature information can be extracted from medical images. Second, the BERT model is used to extract features from text. BERT (Bidirectional Encoder Representation Transformer) is a pre-trained language model with powerful text feature extraction capabilities. BERT captures contextual information and semantic associations in text data. Next, image and text features are fused using a cross-modal attention mechanism. The cross-modal attention mechanism automatically learns the associations between image and text, enabling the fused features to better represent the combined information of both data types. Finally, the SG-Bi-LSTM model is proposed to train and test the fused features. Sparse gates are a sparsification method that effectively removes noise and processes redundant information in network models, thereby improving model efficiency and accuracy.

[0052] In summary, the basic principle of the proposed model CMASG-Bi-LSTM is to extract features from medical images through ResNet50, extract text features through the BERT model, then fuse the image and text features using a cross-modal attention mechanism, and finally train and test the fused features through the SG-Bi-LSTM model. The combined application of these methods can improve the feature expression capabilities of medical image and text data, and by introducing sparse gates, further optimize the efficiency and accuracy of the model. The model structure is shown in the figure below. Figure 1 shown.

[0053] Among them, the extraction module uses the pre-trained models ResNet50 and BERT to perform feature extraction images and text on medical images and diagnostic reports respectively.

[0054] Among them, the fusion method of the cross-modal attention module adopts a feature fusion method based on the cross-modal attention mechanism. The method is: the cross-modal attention mechanism is expressed as follows:

[0055]

[0056] Among them, CMA (cross-modal attention) represents the cross-modal attention mechanism, I and M represent different modalities respectively, and the role of M is to enhance the representation of I, and represents the dot product of the matrix. Represents the scaling factor. The fusion of the cross-modal attention mechanism involves the interaction between images and text. The fusion formula of the cross-modal attention mechanism is as follows:

[0057]

[0058] Among them, A i，j represents the attention weight of the i-th modality to the j-th modality, S i，j is the similarity score between the two modalities, calculated using the dot product method, where N represents the total number of modalities. For each modality i, the similarity score between modality i and other modality j is calculated, and then these scores are converted into a probability distribution to calculate the attention weight of i on j. This fusion of cross-modal attention mechanisms can help the model better utilize information between different modalities and improve model performance.

[0059] By extracting the features of images and texts, and fusing the image features and text features across modalities through the cross-modal attention mechanism, we can finally obtain better features for training and testing the classification model, thereby improving the model classification effect.

[0060] The sparse gate-based bidirectional long short-term memory network module is a network that combines Bi-LSTM with sparse gates. In the sparse gate and Bi-LSTM combined model, the Bi-LSTM is first used to encode the sequence, and then the encoded sequence is sparsed using the sparse gate. Because Bi-LSTM can fully utilize the information in the input sequence to model the input sequence, thus obtaining the sequence representation H. The sparse gate can further reduce the redundant information in the sequence while retaining the important information of the sequence, thereby improving the efficiency of the model.

[0061] Bi-LSTM (Bidirectional Long Short-Term Memory) is a commonly used recurrent neural network (RNN) model, often used to process sequence data, such as language modeling, natural language processing, and speech recognition. Compared with the traditional unidirectional LSTM (Long Short-Term Memory) model, Bi-LSTM can simultaneously utilize forward and backward context information, thereby improving the performance of the model. Figure 2shown.

[0062] Bi-LSTM consists of two LSTMs, one of which traverses the input sequence forward in time, and the other LSTM traverses the input sequence backward in time, respectively called forward LSTM and backward LSTM. The hidden state of the forward LSTM is Only the input information of the current moment and the previous moment is considered, and the hidden state of the backward LSTM Considering only the input information at the current moment and the following moments, the outputs of the two LSTMs are concatenated to form the final Bi-LSTM output.

[0063]

[0064] Among them, “,” represents the vector splicing operation.

[0065] Among them, the sparse gate is a new gating mechanism, whose main function is to control the flow of information and improve the representation and generalization capabilities of the neural network. Compared with traditional gating mechanisms (such as LSTM and GRU), the sparse gate can more effectively filter out noise information and improve the robustness and interpretability of the network. The sparse gate is implemented by introducing a sparse matrix. Assume that the input data is X = [x1, x2, ..., x n ]∈R n×d , where n represents the sequence length and d represents the feature dimension of each time step, then the sparse gate is expressed as:

[0066] g i，j =σ(a j (x i )+b j )·s i，j (4)

[0067] Among them, a j is a function that extracts the input x i The features in b j is the bias term, σ is the sigmoid function, s i，j Is a binary matrix, indicating sparsity, when s i，j = 1, it means that the jth feature of the i-th time step participates in the operation; when s i，j = 0, it means that the jth feature of the i-th time step is ignored; i，j As a probability variable, the parameters are then learned by maximizing the marginal likelihood of the model or minimizing the reconstruction error. Specifically, the sparse coding algorithm of LISTA is used to optimize the model:

[0068]

[0069] Where X represents the given input data matrix The goal of sparse coding is to learn a dictionary The purpose is to generate a sparse code of the input data λ represents the regularization coefficient of the l1-norm.

[0070] To solve Equation (5), the traditional method is to alternately optimize W and S, which corresponds to two optimization processes: dictionary learning and sparse approximation. Specifically, by fixing S, Equation (5) is reduced to the following l2-constrained optimization problem:

[0071]

[0072] By fixing W, Equation (5) reduces to a sparse approximation problem that aims to represent the input x by a linear combination of W as follows:

[0073]

[0074] The iterative hard thresholding (ISTA) algorithm is one of the most popular solvers. It uses the iterative hard thresholding (ISTA) algorithm to decompose the objective of (7) into two parts: the differentiable part It is updated by the gradient descent method, and the other part is the l1 regularization part, which is updated by the hard threshold operator. The update formula is as follows:

[0075]

[0076] Among them, s (t) represents the sparse coding of the tth iteration, sh (λτ) =sign(s)(|s|-λT) is the definition of the contraction function, represents the differentiable part, τ is the coefficient, and the solution of formula (8) is achieved by the following update rule:

[0077]

[0078] Among them, W u =I-τW T W, W v =τW T .

[0079] The bidirectional long short-term memory network module based on sparse gates adds a sparse gate after the input gate, forget gate and output gate in Bi-LSTM to control the information flow at each time step in the model. The specific implementation method of the bidirectional long short-term memory network module based on sparse gates is: at each time step t, the input of the sparse gate is the output h of Bi-LSTM at that time step tThe output of the sparse gate is a binary vector g_t∈[0,1], indicating whether h t Perform sparseness, the output g of the sparse gate t The dot product operation will be performed with the output of the input gate, forget gate and output gate in Bi-LSTM to obtain the output of the sparse gate. Suppose the output of the input gate, forget gate and output gate at time step t are i t , f t , o t , then the output of the sparse gate is:

[0080]

[0081] Among them, ⊙ represents the dot product operator, Represent the outputs of the input gate, forget gate, and output gate after sparsification, respectively.

[0082] In order to illustrate the effect of the present invention, the following simulation experiment is carried out:

[0083] 1) Dataset and preprocessing

[0084] The NLMCXR dataset, sourced from the National Library of Medicine at the National Institutes of Health (NIH), is part of the NLM Open-i Indiana Chest X-ray Dataset, primarily used for research in automatic diagnosis and disease classification of medical images. The dataset contains a series of chest X-rays and their corresponding diagnostic reports, including 7,470 images (including frontal and lateral views) and 3,955 diagnostic reports.

[0085] 2) Experimental Setup

[0086] This experiment was conducted in Python in a GPU-accelerated environment, using the Pytorch deep learning framework. The computer configuration was Windows 11 system, 128G memory, and RTX Geforce 4090 24G video memory. This experiment used the ResNet50 and BERT pre-trained models to preprocess the images and reports of the dataset and obtain data features. In order to make full use of the feature information of the image report, the present invention proposes a feature fusion method based on a cross-modal attention mechanism to fuse the features of image and text information, and use the fused features for training and decision-making. SGBi-LSTM is a gating mechanism for improving Bi-LSTM. The standard gating mechanism of Bi-LSTM includes an input gate, a forget gate, and an output gate, which control the flow of information in LSTM. However, the standard gating mechanism may be affected by the accumulation and interference of information, resulting in a decrease in model performance. By adding a sparse gate to reduce the information flow controlled by the gate, the accumulation and interference of information are alleviated, thereby improving the performance of the model. Table 1 lists the parameter settings of this experiment.

[0087] Table 1 Experimental parameter settings

[0088]

[0089] 3) Experimental evaluation indicators

[0090] The detection / diagnosis performance of CAD systems is usually measured by indicators such as recall, accuracy, receiver operating characteristic curves, and confusion matrices. For the ROI area of ​​a medical image, it can be described as lesion or non-lesion using positive or negative, and the correctness of the judgment can be expressed as true or false. Then the detection and diagnosis results output by the CAD system may be: (1) True Positive (TP) - the diagnosis is positive, and the object's true value is also positive; (2) True Negative (TN) - the diagnosis is negative, and the object's true value is also negative; (3) False Positive (FP) - the diagnosis is positive, and the object's true value is negative; (4) False Negative (FN) - the diagnosis is negative, and the object's true value is positive. The calculation formulas for recall and accuracy are expressed as follows:

[0091] Recall=TPR=TP / (TP+FN)×100% (11)

[0092] Accuracy=(TP+TN) / (TP+TN+FP+FN)×100% (12)

[0093] FPR=FP / (TP+FN)×100% (13)

[0094] Recall, also known as the true positive rate (TPR), is the ratio of samples correctly identified as positive in anomaly regions and is a measure of a system's true positive recognition performance. Accuracy is the ratio of true positive and true negative individuals in an object that are correctly identified. The ROC curve shows the relationship between the true positive rate (TPR) and false positive rate (FPR) of a binary classifier at various thresholds. The true positive rate refers to the proportion of samples predicted as positive by the classifier when they are actually positive, while the false positive rate refers to the proportion of samples predicted as positive by the classifier when they are actually negative.

[0095] The horizontal axis of the ROC curve is FPR, and the vertical axis is TPR. The closer the ROC curve is to the upper left corner, the better the classifier performance. In addition, by calculating the area under the ROC curve, an evaluation metric can be obtained. That is, the larger the area under the curve, the better the classifier performance. This metric is called AUC (Area Under the Curve).

[0096] The confusion matrix is ​​a common method for evaluating classifier performance. A confusion matrix is ​​a two-dimensional matrix where rows represent actual categories and columns represent predicted categories. The four elements in a confusion matrix are: true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). Using the confusion matrix, we can calculate multiple metrics, such as accuracy, recall, precision, and F1-score, which are also important indicators for evaluating model performance.

[0097] 4) Experimental results and analysis

[0098] After training, the model proposed in this invention was tested on 20% of the samples. The experiment showed that after testing on the public dataset NLMCXR, the accuracy of the network model proposed in this invention reached 95.98%. In order to observe the data more intuitively, the confusion matrix heat map was drawn as follows: Figure 5 shown.

[0099] In order to illustrate the effectiveness and superiority of the algorithm proposed in this paper, we will conduct baseline model comparison and ablation experiment comparison. First, on the same dataset, we compare the proposed model with the traditional baseline models VGG16, ResNet50, and the existing model Inceptionv3+Bi-LSTM-Attention. The ROC curves of the proposed model compared with other baseline models are shown in the figure below. Figure 5 As shown in Table 3, the comparison results of the relevant indicators of this model and other traditional baseline models are shown. Figure 5 It can be intuitively seen that the ROC of the proposed model is significantly better than other baseline models and existing models. As can be clearly seen from Table 3, the VGG16 model performed the worst in both accuracy and AUC in the experiment, while the ResNet50 performed relatively well. The proposed model performed the best in terms of both metrics. In terms of recall, the proposed model was second only to the Inceptionv3+Bi-LSTM-Attention model, verifying the effectiveness of the proposed model.

[0100] Table 3 Comparison of the algorithm of the present invention with other baseline models on the NLMCXR dataset

[0101]

[0102] Secondly, ablation experiments were conducted on the same dataset using Bi-LSTM, CMA+Bi-LSTM, SG+Bi-LSTM, and CMA+SG+Bi-LSTM combined with a CNN feature extractor (ResNet50). The experimental results were compared using relevant evaluation indicators, as shown in Table 4. From the comparison results, it can be seen that the model of the present invention performs best in both accuracy and AUC results, among which Bi-LSTM performs the worst in the ablation experiment. The SG+Bi-LSTM model introduces a sparse gate on the basis of Bi-LSTM, and its average test inference time (T_infer) is reduced by 60%, giving full play to the role of the sparse gate, reducing the amount of computation and improving the computing speed. When CMA+SG+Bi-LSTM (our) is compared with CMA+Bi-LSTM, the role of the sparse gate reduces its inference time by approximately 65%, demonstrating the superiority of the model of the present invention.

[0103] Table 4 Comparison results of the model of the present invention with other ablation models

[0104]

[0105] The present invention proposes a medical image classification model based on CMASG-Bi-LSTM, which can effectively fuse cross-modal features and improve the performance and efficiency of the model. First, the cross-modal attention mechanism allows the model to dynamically adjust the weight of each modality, thereby better utilizing the information of different modalities. Secondly, the sparse gate mechanism can reduce the impact of redundant information and improve the computational efficiency and robustness of the model. Finally, by comparing with the traditional models VGG16, ResNet50 and the existing model Inceptionv3+Bi-LSTM-Attention, and combining the model of the present invention with the CNN feature extractor (ResNet50) for ablation experiments, the comparison results are all highly competitive, verifying the effectiveness and superiority of this model.

[0106] In summary, the model proposed in this paper has significant advantages in the field of medical image processing, especially in medical image classification tasks, and provides valuable tools and methods for the field of medical image analysis.

[0107] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An image classification system based on cross-modal feature fusion, characterized by: It includes,extraction module, which performs feature extraction of images and texts on medical images and,diagnosis reports; Fusion module, which uses a cross-modal attention module to fuse the features of image and text; The model training module uses a bidirectional long short-term memory network module based on sparse gates to train the fused data; The bidirectional long short-term memory network module based on sparse gate is a network that combines Bi-LSTM with sparse gate. In the model combining sparse gate and Bi-LSTM, Bi-LSTM is first used to encode the sequence, and then the encoded sequence is sparsed by the sparse gate. The bidirectional long short-term memory network module based on sparse gate adds a sparse gate after the input gate, forget gate and output gate in Bi-LSTM to control the information flow of each time step in the model. The specific implementation method of the bidirectional long short-term memory network module based on sparse gate is: at each time step t, the input of the sparse gate is the output h of Bi-LSTM at that time step. t , the output of the sparse gate is a binary vector g t ∈[0,1], indicating whether h t Perform sparseness, the output g of the sparse gate t The dot product operation will be performed with the output of the input gate, forget gate and output gate in Bi-LSTM to obtain the output of the sparse gate. Suppose the output of the input gate, forget gate and output gate at time step t are i t 、f t 、o t , then the output of the sparse gate is: Among them, ⊙ represents the dot product operator, Represent the outputs of the input gate, forget gate, and output gate after sparsification, respectively.

2. The image classification system based on cross-modal feature fusion according to claim 1, characterized in that: The extraction module uses the pre-trained models ResNet50 and BERT to perform feature extraction on medical images and diagnostic reports, respectively.

3. The image classification system based on cross-modal feature fusion according to claim 1, characterized in that: The fusion method of the cross-modal attention module adopts a feature fusion method based on the cross-modal attention mechanism. The method is: the cross-modal attention mechanism is expressed as follows: Among them, CMA (cross-modal attention) represents the cross-modal attention mechanism, I and M represent different modalities respectively, and the role of M is to enhance the representation of I, and represents the dot product of the matrix. Represents the scaling factor. The fusion of the cross-modal attention mechanism involves the interaction between images and text. The fusion formula of the cross-modal attention mechanism is as follows: Among them, A i，j represents the attention weight of the i-th modality to the j-th modality, S i,j is the similarity score between two modalities, N represents the total number of modalities; for each modality i, the attention weight of i to j is calculated by calculating the similarity score between the modality i and other modalities j, and then converting these scores into probability distributions.

4. The image classification system based on cross-modal feature fusion according to claim 1, characterized in that: Bi-LSTM consists of two LSTMs, one of which traverses the input sequence forward in time, and the other LSTM traverses the input sequence backward in time, respectively called forward LSTM and backward LSTM. The hidden state of the forward LSTM is Only the input information of the current moment and the previous moment is considered, and the hidden state of the backward LSTM Considering only the input information at the current moment and the following moments, the outputs of the two LSTMs are concatenated to form the final Bi-LSTM output. Among them, "," represents the vector splicing operation.

5. The image classification system based on cross-modal feature fusion according to claim 1, characterized in that: The sparse gate is implemented by introducing a sparse matrix. Assume that the input data is X = [x1, x2, ..., x n ]∈R n×d , where n represents the sequence length and d represents the feature dimension of each time step, then the sparse gate is expressed as: g i，j =σ(a j (x i )+b j )·s i，j (4) Among them, a j is a function that extracts the input x i The features in b j is the bias term, σ is the sigmoid function, s i,j Is a binary matrix, indicating sparsity, when s i,j = 1, it means that the jth feature of the i-th time step participates in the operation; when s i，j = 0, it means that the jth feature of the i-th time step is ignored; i,j As a probability variable, the parameters are then learned by maximizing the marginal likelihood of the model or minimizing the reconstruction error. Specifically, the sparse coding algorithm of LISTA is used to optimize the model: Where X represents the given input data matrix The goal of sparse coding is to learn a dictionary The purpose is to generate a sparse code of the input data λ represents -norm regularization coefficient; To solve Equation (5), W and S are optimized alternately, which corresponds to two optimization processes: dictionary learning and sparse approximation. Specifically, by fixing S, Equation (5) is reduced to the following Constrained optimization problem: By fixing W, Equation (5) reduces to a sparse approximation problem that aims to represent the input x by a linear combination of W as follows: The iterative hard threshold algorithm is used to decompose the target of (7) into two parts: the differentiable part Updated by gradient descent, the other part is The regularization part is updated by the hard threshold operator, and the update formula is as follows: Among them, s (t) represents the sparse coding of the tth iteration, sh (λτ) =sign(s)(|s|-λτ) is the definition of the contraction function, represents the differentiable part, τ is the coefficient, and the solution of formula (8) is achieved by the following update rule: Among them, W u =I-τW T W,W v =τW T 。

Images