Medical image diagnosis system based on image and text feature fusion

Abstract

The invention belongs to the technical field of medical image processing, and discloses a medical image diagnosis system based on image and text feature fusion, and the system comprises an image processing module which is responsible for carrying out the enhancement, segmentation and feature extraction of a medical image; the text analysis module is responsible for extracting key clinical information from the electronic medical record; the semantic alignment and feature fusion module is used for embedding image features and text features into a unified vector space to realize fusion of multi-modal data; the model training and optimizing module is responsible for training and optimizing a medical diagnosis system model; and the clinical application module is responsible for applying the trained medical diagnosis system model to an actual medical environment to assist doctors in disease diagnosis. According to the medical image diagnosis system based on image and text feature fusion, by combining image enhancement, morphological processing, NLP analysis and multi-modal feature fusion, the illness state of a patient can be understood more comprehensively, the diagnosis accuracy is improved, and a more reliable auxiliary decision making basis is provided for doctors.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, and particularly relates to a medical image diagnosis system based on image and text feature fusion. BACKGROUND

[0002] In the modern medical field, the fusion of image and text features has become an important means to improve the accuracy and efficiency of disease diagnosis, especially in the process of burn diagnosis. Through the collaborative analysis of image and text data, the patient's condition can be more comprehensively understood, and more reliable support can be provided for clinical decision-making. However, there are still many challenges in practical application, among which data quality and medical diagnosis system model generalization ability are key factors affecting system performance and promotion.

[0003] Firstly, data quality is one of the main challenges faced by medical artificial intelligence systems. Medical data usually has high heterogeneity and incompleteness. Images collected by different hospitals and different equipment may have differences in resolution, different lighting conditions or noise interference, affecting the accuracy of feature extraction. In addition, the text information in the electronic medical record often lacks structure, and different doctors' writing habits and terminology use may cause parsing difficulties.

[0004] In addition, the generalization ability of the medical diagnosis system model is a key factor affecting the applicability of the medical AI system in different environments. Because the distribution of medical data may differ due to differences in hospitals, regions or patient groups, the medical diagnosis system model performs well on training data but causes performance degradation on new data.

[0005] Therefore, in view of the problems in the prior art, the present application proposes a medical image diagnosis system based on image and text feature fusion. SUMMARY

[0006] The purpose of the present application is to provide a medical image diagnosis system based on image and text feature fusion, which comprehensively understands the patient's condition by combining image enhancement, morphological processing, NLP analysis and multi-modal feature fusion, improves the accuracy of diagnosis, and provides more reliable auxiliary decision-making basis for doctors.

[0007] To achieve the above-mentioned purpose, the present application provides a medical image diagnosis system based on image and text feature fusion, which comprises an image processing module, a text analysis module, a semantic alignment and feature fusion module, a model training and optimization module and a clinical application module.

[0008] Preferably, the image processing module is responsible for enhancing, segmenting and feature extracting medical images, improving medical image quality and extracting key lesion information, and the specific process is as follows:

[0009] Firstly, histogram equalization, contrast limited adaptive histogram equalization, and deep learning based image enhancement method are used for image enhancement; wherein the image enhancement formula is as follows:

[0010] I enhanced =CLAHE(I)⊕M multi-scale (I);

[0011] In the formula, I enhanced represents the enhanced image; CLAHE represents the contrast limited adaptive histogram equalization operation; ⊕ represents the morphological superposition operation; M multi-scale represents the multi-scale morphological operation; I represents the original image;

[0012] Subsequently, morphological processing is used to remove noise, and multi-scale structural element optimization processing effect is combined; wherein the morphological operation includes erosion, dilation, opening operation and closing operation, which eliminates isolated noise points in the image and preserves the structural characteristics;

[0013] Then, in the image segmentation stage, threshold segmentation, edge detection, region growing and deep learning segmentation model are used to realize the extraction of the lesion area;

[0014] Among them, the deep learning based segmentation model is used to realize image segmentation, and its loss function is as follows:

[0015] L seg =DiceLoss+λ·FocalLoss;

[0016] In the formula, L seg represents the total loss value of the segmentation model; DiceLoss represents the Dice coefficient loss; λ represents the adjustment weight coefficient; FocalLoss represents the focal loss;

[0017] Finally, the shape, color and texture features of the image are extracted by the feature extraction algorithm, and they are converted into numerical text; wherein the color feature is quantified by RGB or HSV color space; the texture feature is extracted by using gray level co-occurrence matrix or convolutional neural network.

[0018] Preferably, the text analysis module is responsible for extracting key clinical information from the electronic medical record, including patient complaints, medical history, physical examination description;

[0019] Firstly, the natural language processing NLP model is used to perform word segmentation, part-of-speech tagging and syntax analysis on the text;

[0020] Subsequently, an entity recognition NER model is established by using the entity recognition NER technology, which is used to extract key information; combined with the rule engine and the deep learning model, the physical examination description is automatically identified and analyzed;

[0021] Wherein, the entity recognition model is as follows:

[0022] P(y|x) = BiLSTM-CRF(BERT med (x));

[0023] In the formula, P(y|x) represents the probability of predicting the label sequence y under the condition of giving the input original text sequence x; BiLSTM represents a bidirectional long short-term memory network; CRF represents a conditional random field; BERT med represents a BERT model pre-trained in the medical field; x represents an original text sequence; y represents the optimal entity label sequence of the input original text sequence x;

[0024] Finally, the extracted text information is structured and semantically aligned with the image features.

[0025] Preferably, the semantic alignment and feature fusion module embeds the image features and the text features into a unified vector space by establishing a mapping relationship between the image and text features, realizes the fusion of multi-modal data, and dynamically weights the fusion through an attention mechanism. The specific process is as follows:

[0026] Firstly, a multi-modal pre-training model based on deep learning is used to encode the text and image features into high-dimensional vectors, so that they can be trained on the dataset while understanding the image and text information and establishing semantic associations between different modalities. In the training process, the fusion strategy is optimized to keep the image and text information consistent in the vector space. In addition, through a contrastive learning method, the image and text features are aligned in the training process.

[0027] Subsequently, a multi-head attention mechanism is used to focus on multiple different feature dimensions simultaneously, and the weight of the image features is dynamically adjusted according to the text content. The semantic alignment formula is as follows:

[0028]

[0029] In the formula, Align represents the output of the semantic alignment function; V i represents the image feature vector; V t represents the text feature vector; N represents the number of multi-head attention heads, and n represents the head index; Attn represents a scaled dot-product attention mechanism; W q and W k represent learnable weight matrices.

[0030] Preferably, a gating mechanism is combined to control the contribution of different modalities, and the fusion strategy is adaptively adjusted according to the input content. Self-supervised learning is introduced to pre-train using unlabeled data, so that semantic information can be learned without human annotation, and the fusion strategy is adaptively adjusted according to the input content.

[0031] where the gating fusion mechanism is as follows:

[0032] G = σ(W g [V i ,V t ]+b g )V fused = G ⊙ V i + (1 - G) ⊙ V t ;

[0033] where G represents the gating signal; σ is the sigmoid activation function; W g represents the gating weight matrix; b g represents the gating bias term; V fused represents the final fused feature vector; and ⊙ represents element-wise multiplication.

[0034] Preferably, the model training and optimization module is responsible for training and optimizing the medical diagnosis system model, and the specific process is as follows:

[0035] First, pre-training is performed using a medical data set, so that the medical diagnosis system model learns medical knowledge;

[0036] Subsequently, a multi-task learning method is used to simultaneously train the medical diagnosis system model for image classification, text classification, and multi-modal feature fusion;

[0037] In addition, a transfer learning method is introduced to adapt the medical diagnosis system model to different hospitals and different data sets, and a self-adaptive learning strategy is combined to dynamically adjust the parameters of the model according to different data distributions.

[0038] Preferably, the clinical application module is responsible for applying the trained medical diagnosis system model to the actual medical environment to assist doctors in disease diagnosis, and the specific process is as follows:

[0039] First, medical images of patients are acquired through image acquisition devices, and electronic medical record data are analyzed in combination;

[0040] Subsequently, the trained medical diagnosis system model is used for feature extraction, text analysis, and multi-modal fusion to generate a diagnosis result;

[0041] Finally, the diagnosis result is presented to the doctor in a visual manner, and an explainable diagnosis basis is provided;

[0042] In addition, the clinical application module combines knowledge graph technology to associate medical knowledge with the prediction results of the medical diagnosis system model, and provides diagnosis suggestions.

[0043] Therefore, the medical image diagnosis system based on image and text feature fusion has the following beneficial effects:

[0044] (1) The present application comprehensively understands the patient's condition by combining image enhancement, morphological processing, NLP analysis and multi-modal feature fusion, improves the accuracy of diagnosis, and provides more reliable auxiliary decision basis for doctors.

[0045] (2) The present application not only improves the integration effect of image and text features, but also enhances the generalization ability of the medical diagnosis system model, so that it can maintain high performance in different medical scenarios; combined with federated learning and adaptive learning method, effectively cope with the heterogeneity and privacy protection problem of medical data, improve the adaptability and scalability of the medical diagnosis system model.

[0046] The technical solutions of the present application will be further described below with the help of drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a medical image diagnosis system architecture based on image and text feature fusion of the present application;

[0048] Figure 2 is a workflow diagram of burn diagnosis scenario of the present application. DETAILED DESCRIPTION

[0049] The technical solutions of the present application will be further described below with the help of drawings and examples.

[0050] EMBODIMENT

[0051] As shown in Figure 1 , the present application proposes a medical image diagnosis system based on image and text feature fusion, which combines image enhancement, text analysis, semantic alignment and feature fusion technology to improve the accuracy and efficiency of disease diagnosis. Among them, the system mainly includes the following several core modules: image processing module, text analysis module, semantic alignment and feature fusion module, model training and optimization module and clinical application module.

[0052] 1. Image processing module

[0053] The image processing module is responsible for enhancing, segmenting and feature extracting medical images to improve image quality and extract key lesion information.

[0054] First, histogram equalization, contrast limited adaptive histogram equalization (CLAHE) or deep learning-based image enhancement methods (such as generative adversarial network (GAN) and autoencoder (Autoencoder)) are used for image enhancement, which can improve the contrast and clarity of the image, adjust the texture and details of the image more finely, effectively improve the brightness distribution of the image, enhance the contrast between different tissues, make the features such as wound edge and color change more obvious, and improve the accuracy of feature extraction.

[0055] wherein the image enhancement formula is as follows:

[0056] I enhanced = CLAHE (I) O M multi-scale (I);

[0057] In the formula, I enhanced represents the enhanced image; CLAHE represents the contrast limited adaptive histogram equalization operation; O represents the morphological overlay operation; M multi-scale represents the multi-scale morphological operation; I represents the original image.

[0058] Subsequently, morphological processing is used to remove noise, and multi-scale structural elements are combined to optimize the processing effect. The morphological operation mainly includes erosion, dilation, opening operation and closing operation, which can effectively eliminate isolated noise points in the image while preserving the main structural features.

[0059] For example, in wound image analysis, the erosion operation can be used to remove small noise points, while the dilation operation can be used to fill the holes in the wound edge, making the lesion area more complete. In addition, the morphological processing method based on multi-scale structural elements can adapt to different sizes of noise, improving the robustness of image processing.

[0060] Then, in the image segmentation stage, threshold segmentation, edge detection, region growing and deep learning segmentation model (such as U-Net, MaskR-CNN) are used to realize accurate extraction of the lesion area.

[0061] Among them, the improved U-Net structure is used to realize image segmentation, and the loss function is as follows:

[0062] L seg = DiceLoss + λ · FocalLoss;

[0063] In the formula, L seg represents the total loss value of the segmentation model; DiceLoss represents the Dice coefficient loss; λ represents the adjustment weight coefficient; FocalLoss represents the focal loss.

[0064] Finally, the shape, color, and texture features of the image are extracted using feature extraction algorithms such as HOG, SIFT, and CNN, and converted into numerical text to provide data support for subsequent feature integration.

[0065] The color features can be quantified using RGB or HSV color spaces, and the texture features can be extracted using a gray-level co-occurrence matrix (GLCM) or a convolutional neural network.

[0066] In medical image analysis, the optimization of image feature extraction and processing is crucial for improving diagnostic accuracy and efficiency. Although traditional image enhancement methods can improve image quality, they may not be precise enough when dealing with complex lesions. Therefore, combining adaptive enhancement methods with deep learning and multi-scale morphological processing can further enhance the effectiveness of image processing. Meanwhile, converting image features into numerical text not only helps feature integration but also provides more abundant input data for subsequent machine learning models, improving the generalization ability of the model.

[0067] 2. Text analysis module.

[0068] In the process of medical diagnosis, the text information in electronic medical records contains a large amount of key clinical data, including patient complaints, medical history, physical examination descriptions, etc. In order to fully utilize these information, it is necessary to analyze and extract them through natural language processing (NLP) models to obtain features related to the disease. The application of NLP technology in medical text analysis mainly includes entity recognition, relationship extraction, sentiment analysis, and semantic understanding, which can automatically extract key information from medical records and structure them, providing data support for subsequent feature integration.

[0069] Among them, patient complaints are an important basis for clinical diagnosis, usually containing information such as pain level, wound appearance, and allergy history. For example, "pain sensitive" may indicate a superficial Ⅱ degree burn, and "no pain" may indicate a Ⅲ degree burn. Through NLP models, these key descriptions are identified and matched with standard terms in medical knowledge bases, thereby improving the accuracy of diagnosis.

[0070] In addition, medical history information such as diabetes and infection risk is also an important factor affecting wound healing. NLP models can automatically extract these medical history information and use it as an auxiliary diagnostic basis to help doctors make more comprehensive judgments.

[0071] Physical examination descriptions are another important type of textual information, often containing details such as blister shape, eschar hardness, and wound color. These details are crucial for determining wound type and severity. NLP models can extract these descriptions through keyword matching and context analysis, and convert them into structured features. For example, the size of the blister, its shape (e.g., "blister border clear" or "blister blurred"), and the hardness of the eschar (e.g., "hard knot" or "soft") can all be automatically identified and recorded, providing data support for subsequent feature integration.

[0072] Therefore, the text analysis module is responsible for extracting key clinical information from electronic medical records, such as patient complaints, medical history, and physical examination descriptions.

[0073] First, natural language processing (NLP) models such as BERT, RoBERTa, or BiLSTM are used to perform word segmentation, part-of-speech tagging, and syntax analysis on the text.

[0074] Subsequently, an entity recognition (NER) model is established through NER technology to extract key information such as "pain sensitivity," "no pain," "diabetes," and "infection risk." In addition, a rule engine and deep learning model can be combined to automatically identify and analyze physical examination descriptions such as blister shape and eschar hardness.

[0075] Among them, the entity recognition model is as follows:

[0076] P(y|x) = BiLSTM-CRF(BERT med (x));

[0077] In the formula, P(y|x) represents the probability of predicting the label sequence y given the input original text sequence x; BiLSTM represents a bidirectional long short-term memory network; CRF represents a conditional random field; BERT med represents a BERT model pre-trained in the medical field; x represents the original text sequence; y represents the optimal entity label sequence of the input original text sequence x.

[0078] Finally, the extracted text information is structured and semantically aligned with the image features, providing a data foundation for subsequent multi-modal fusion.

[0079] This process analyzes electronic medical records through NLP models and combines semantic alignment techniques to effectively integrate text information with image features, providing more comprehensive data support for medical diagnosis, improving the accuracy of diagnosis, and enhancing the intelligent level of the medical system, providing more reliable information for clinical decision-making.

[0080] 3. Semantic alignment and feature fusion module.

[0081] The semantic alignment and feature fusion module is the core part of the entire system, which establishes the mapping relationship between image and text features, embeds image and text features into a unified vector space to realize the fusion of multi-modal data, and dynamically weights the fusion through attention mechanism. For example, when the text description is "red and white", the weight of the image feature is automatically adjusted to strengthen the feature extraction of the corresponding region. This dynamic fusion method can improve the understanding ability of the medical diagnosis system for multi-modal data, making the diagnosis result more accurate.

[0082] Firstly, a multi-modal pre-training model based on deep learning (such as multi-layer perception (MLP), Transformer architecture, CLIP, ALIGN, graph neural network (GNN) or ViLT) is used to encode text and image features into high-dimensional vectors, so that they can be trained on large-scale datasets, understand image and text information at the same time, and establish semantic association between different modalities. During the training process, the fusion strategy is continuously optimized to make them consistent in the vector space. In addition, through the contrastive learning method, the image and text features are gradually aligned during the training process, improving the generalization ability of the medical diagnosis system.

[0083] Subsequently, the multi-head attention (Multi-head Attention) mechanism is used to focus on multiple different feature dimensions at the same time, dynamically adjust the weight of the image feature according to the text content, so that the medical diagnosis system pays more attention to the image area related to the current text description, improving the accuracy and interpretability of the diagnosis.

[0084] Among them, the semantic alignment formula is as follows:

[0085]

[0086] In the formula, Align represents the output of the semantic alignment function; V i represents the image feature vector; V t represents the text feature vector; N represents the number of multi-head attention heads, n represents the head index; Attn represents the scaled dot-product attention mechanism; W q and W k represent the learnable weight matrix.

[0087] In addition, a gating mechanism (such as using a gated recurrent unit (GRU) or a long short-term memory network (LSTM)) can be combined to control the contribution of different modalities, and the fusion strategy can be adaptively adjusted according to the input content. This method not only improves the accuracy of feature fusion, but also enhances the explainability of the medical diagnosis system, allowing doctors to more intuitively understand the decision-making process. Self-supervised learning is introduced, and unlabelled data is used for pre-training, so that more rich semantic information can be learned without human annotation, and the fusion strategy can be adaptively adjusted according to the input content to improve the generalization ability of the medical diagnosis system.

[0088] wherein the gating fusion mechanism is as follows:

[0089] G = σ (W g [V i ,V t ]+b g )V fused = G o V i +(1-G) o V t ;

[0090] In the formula, G represents the gating signal; σ is the sigmoid activation function; W g represents the gating weight matrix; b g represents the gating bias term; V fused represents the final fused feature vector; and o represents element-wise multiplication.

[0091] The fusion method based on attention mechanism can also use a multi-task learning (Multi-task Learning) framework to jointly train on multiple related tasks, thereby improving the robustness of feature fusion. For example, in the medical diagnosis task, the medical diagnosis system model can be trained to perform image classification, text classification, and multi-modal feature fusion simultaneously, so that the medical diagnosis system model shares feature representations between different tasks and improves overall performance through joint optimization.

[0092] This process can effectively integrate image and text features through semantic alignment and dynamic weighted fusion, improving the accuracy of medical diagnosis, not only enhancing the understanding ability of the medical diagnosis system for multi-modal data, but also improving the explainability of the diagnosis result, allowing doctors to more intuitively understand the basis for decision-making.

[0093] 4. Model training and optimization module.

[0094] The model training and optimization module is responsible for training and optimizing the medical diagnosis system model to improve its performance and adaptability.

[0095] First, pre-training is performed using large-scale medical datasets to enable the medical diagnosis system model to learn rich medical knowledge.

[0096] Subsequently, a multi-task learning method is used to simultaneously train the medical diagnosis system model for image classification, text classification, and multi-modal feature fusion, improving the model's generalization ability.

[0097] In addition, transfer learning is introduced to enable the medical diagnosis system model to quickly adapt to different hospitals or different datasets, improving its applicability in different environments. At the same time, adaptive learning strategies are combined to enable the model to dynamically adjust parameters according to different data distributions, improving its performance in different scenarios.

[0098] 5. Clinical application module.

[0099] The clinical application module is responsible for applying the trained medical diagnosis system model to actual medical environments to assist doctors in disease diagnosis.

[0100] First, medical images of patients are obtained through image acquisition devices and analyzed in combination with electronic medical record data.

[0101] Subsequently, the trained medical diagnosis system model is used for feature extraction, text analysis, and multi-modal fusion to generate diagnosis results.

[0102] Finally, the diagnosis results are presented to doctors in a visual manner, and interpretable diagnostic evidence such as attention visualization and feature importance analysis is provided to improve doctors' trust in the medical diagnosis system model.

[0103] In addition, the clinical application module can also combine knowledge graph technology to associate medical knowledge with the prediction results of the medical diagnosis system model, enabling the model to provide more clinically meaningful diagnostic recommendations.

[0104] In actual medical environments, image and text feature fusion technology has been widely applied in various clinical scenarios, especially in chronic wound management, postoperative monitoring, and burn assessment, among others. The workflow diagram for the burn diagnosis scenario is as follows: Figure 2For example, in chronic wound management, this technology can automatically identify wound types, assess healing progress, and provide personalized treatment recommendations by analyzing image features and textual information in electronic medical records. Additionally, in postoperative monitoring, combining image analysis and medical record data can track patient recovery in real-time, detect potential complications promptly, and improve the timeliness and accuracy of medical interventions. In burn assessment, this technology can quickly determine burn depth through image enhancement and text analysis, and provide more accurate treatment plans by combining patient medical history information.

[0105] To verify the actual effect of this technology, various evaluation methods are usually used, including quantitative analysis and clinical verification. Quantitative analysis mainly relies on performance indicators of machine learning models, such as accuracy, precision, recall, and F1-score, to measure the performance of medical diagnosis system models in image feature extraction, text analysis, and multi-modal fusion. In addition, cross-validation and AUC-ROC curve analysis can be used to evaluate the generalization ability of medical diagnosis system models on different data sets. In clinical verification, the diagnosis results of doctors are usually compared to evaluate the auxiliary diagnosis effect of medical diagnosis system models. For example, a blind test method can be used to let doctors independently judge the disease without knowing the output of the medical diagnosis system model, and the prediction results of the medical diagnosis system model are compared to evaluate its clinical applicability.

[0106] Therefore, the present application adopts the above-mentioned medical image diagnosis system based on image and text feature fusion, which can more comprehensively understand the patient's condition by combining image enhancement, morphological processing, NLP analysis and multi-modal feature fusion, improve the accuracy of diagnosis, and provide more reliable auxiliary decision-making basis for doctors. In addition, the present application shows good application prospect in chronic wound management, postoperative monitoring and burn assessment, which can effectively improve medical efficiency, reduce misdiagnosis rate and optimize treatment plan.

[0107] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application and not to limit them, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that: it can still modify or equivalently replace the technical solutions of the present application, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A medical image diagnosis system based on image and text feature fusion, characterized in that: It includes an image processing module, a text analysis module, a semantic alignment and feature fusion module, a model training and optimization module, and a clinical application module. 2.The medical image diagnosis system based on image and text feature fusion according to claim 1, characterized in that: The image processing module is responsible for image enhancement, segmentation, and feature extraction, improving image quality and extracting key lesion information. The specific process is as follows: First, histogram equalization, contrast limited adaptive histogram equalization, and deep learning-based image enhancement methods are used for image enhancement; the image enhancement formula is as follows: wherein I enhanced denotes an enhanced image; CLAHE denotes a contrast limited adaptive histogram equalization operation; denotes a morphological superposition operation; M multi-scale denotes a multi-scale morphological operation; I denotes an original image; Then, morphological processing is used to remove noise, and multi-scale structural elements are used to optimize the processing effect; morphological operations include erosion, dilation, opening operation, and closing operation, which eliminate isolated noise points in the image while preserving structural features; Then, in the image segmentation stage, threshold segmentation, edge detection, region growing, and deep learning segmentation models are used to extract lesion regions; The loss function of the deep learning-based segmentation model is as follows: L seg = DiceLoss + λ · FocalLoss; In the formula, L seg represents the total loss value of the segmentation model; DiceLoss represents the Dice coefficient loss; λ represents the adjustment weight coefficient; FocalLoss represents the focal loss; Finally, shape, color, and texture features of the image are extracted through feature extraction algorithms and converted into numerical text; color features are quantified through RGB or HSV color space; texture features are extracted using a gray-level co-occurrence matrix or a convolutional neural network. 3.The medical image diagnosis system based on image and text feature fusion according to claim 1, characterized in that: The text analysis module is responsible for extracting key clinical information from electronic medical records, including patient complaints, medical history, and physical examination descriptions. First, natural language processing (NLP) models are used to perform word segmentation, part-of-speech tagging, and syntactic analysis on the text; Then, an entity recognition (NER) model is established to extract key information; combined with rule engines and deep learning models, physical examination descriptions are automatically identified and analyzed; The entity recognition model is as follows: P(y | x) = BiLSTM-CRF(BERT med (x)); Where P(y|x) represents the probability of predicting the label sequence y given the input original text sequence x; BiLSTM represents a bidirectional long short-term memory network; CRF denotes conditional random field; BERT med denotes a BERT model pre-trained in the medical field; x denotes an original text sequence; y denotes an optimal entity label sequence of the input original text sequence x; Finally, the extracted text information is structured and semantically aligned with the image features. 4.The medical image diagnosis system based on image and text feature fusion according to claim 1, characterized in that: The semantic alignment and feature fusion module establishes a mapping relationship between image and text features, embeds image and text features into a unified vector space, realizes the fusion of multi-modal data, and dynamically weights the fusion through an attention mechanism. The specific process is as follows: First, a deep learning-based multi-modal pre-training model is used to encode text and image features into high-dimensional vectors, allowing them to be trained on a dataset while understanding image and text information and establishing semantic associations between different modalities. During training, the fusion strategy is optimized to ensure consistency between image and text information in the vector space. In addition, through a contrast learning method, image and text features are aligned during training; Then, a multi-head attention mechanism is used to focus on multiple different feature dimensions simultaneously, dynamically adjusting the weight of image features based on text content; the semantic alignment formula is as follows: where Align denotes the output of the semantic alignment function; V i denotes the image feature vector; V t denotes the text feature vector; N denotes the number of multi-head attention heads, n denotes the head index; Attn denotes the scaled dot-product attention mechanism; W q and W k denote learnable weight matrices. 5.The medical image diagnosis system based on image and text feature fusion according to claim 4, characterized in that: The gating mechanism is combined to control the contribution of different modalities, and the fusion strategy is adaptively adjusted according to the input content; self-supervised learning is introduced to pre-train with unlabeled data, so that semantic information is learned without manual annotation, and the fusion strategy is adaptively adjusted according to the input content; The gating fusion mechanism is as follows: G=σ(W g [V i ,V t ]+b g )V fused =G⊙V i +(1-G)⊙V t ; In the formula, G represents a gate signal; σ is a sigmoid activation function; W g represents a gate weight matrix; b g represents a gate bias term; V fused represents a final fused feature vector; and ⊙ represents element-wise multiplication. 6.The medical image diagnosis system based on image and text feature fusion according to claim 1, characterized in that: The model training and optimization module is responsible for training and optimizing the medical diagnosis system model, and the specific process is as follows: First, pre-training is performed using a medical data set, so that the medical diagnosis system model learns medical knowledge; Subsequently, a multi-task learning method is used to simultaneously train the medical diagnosis system model for image classification, text classification, and multi-modal feature fusion; In addition, a transfer learning method is introduced to adapt the medical diagnosis system model to different hospitals and different data sets, and an adaptive learning strategy is combined to dynamically adjust the parameters of the model according to different data distributions. 7.The medical image diagnosis system based on image and text feature fusion according to claim 1, characterized in that: The clinical application module is responsible for applying the trained medical diagnosis system model to the actual medical environment to assist doctors in disease diagnosis, and the specific process is as follows: First, medical images of patients are acquired through image acquisition devices and analyzed in combination with electronic medical record data; Subsequently, the trained medical diagnosis system model is used for feature extraction, text analysis, and multi-modal fusion to generate a diagnosis result; Finally, the diagnosis result is presented to the doctor in a visual manner, and an explainable diagnosis basis is provided; In addition, the clinical application module combines knowledge graph technology to associate medical knowledge with the prediction results of the medical diagnosis system model to provide diagnosis suggestions.

Images