Medical diagnosis intelligent decision-making system based on multimodal data fusion

Abstract

The present invention discloses a medical diagnosis intelligent decision-making system based on multimodal data fusion. The medical images, biological features and electronic medical record data of the patient are obtained through a patient information collection unit, and feature vectors are extracted from different modal data using a feature processing unit. The core of the system is a modal fusion engine, which combines a deep neural network and a Bayesian reasoning engine to achieve deep learning and probabilistic dependency modeling of the correlation relationship between features. Through an adaptive feature optimizer, an attention mechanism is used to optimize the weights of different modal features to generate optimized feature representations. The intelligent diagnosis analysis unit calculates the diagnosis probability based on the optimized features, and performs auxiliary diagnosis in combination with a case knowledge base when the confidence level is insufficient, and outputs a diagnosis report containing diagnosis categories, bases and suggestions. The present invention integrates multi-source heterogeneous data, combines deep learning and probabilistic reasoning, realizes intelligent medical auxiliary diagnosis, and improves the accuracy and interpretability of diagnosis.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a medical diagnosis intelligent decision-making system based on multimodal data fusion. Background Art

[0002] With the continuous development of medical technology, intelligent medical diagnosis has become an important research direction in the current medical field. The current intelligent diagnosis system mainly uses deep learning methods to process medical imaging data, or uses natural language processing technology to analyze electronic medical record texts. At the same time, some systems also analyze biometric data. These systems have made certain progress in their respective fields and provide valuable auxiliary references for doctors' clinical diagnosis work.

[0003] However, existing medical diagnosis systems generally have the problem of insufficient data utilization. Most systems only focus on a single data modality and fail to effectively integrate the rich diagnostic information contained in multi-source heterogeneous data; even in some systems that attempt multimodal fusion, they often use simple feature splicing or average weighting methods, which cannot fully explore the deep correlation between different modal data, and lack targeted analysis of individual patient characteristics and dynamic adjustment mechanisms, resulting in limited accuracy and reliability of diagnostic results.

[0004] Based on this, the present invention proposes a new medical diagnosis intelligent decision-making system based on multimodal data fusion. Summary of the invention

[0005] The present application provides a medical diagnosis intelligent decision-making system based on multimodal data fusion to improve the accuracy of medical diagnosis.

[0006] The present application provides a medical diagnosis intelligent decision-making system based on multimodal data fusion, comprising:

[0007] Patient information collection unit, used to collect patients' medical imaging data, biometric data and electronic medical record text data;

[0008] Feature processing unit, used to extract visual feature vectors from medical image data; extract physiological index feature vectors from biometric data; extract semantic feature vectors from electronic medical record text;

[0009] A modality fusion engine, comprising a deep neural network and a Bayesian reasoning engine, wherein the deep neural network is used to receive the visual feature vector, the physiological index feature vector and the semantic feature vector output by the feature processing unit, and learn the association relationship between the features of different modalities through multi-layer nonlinear transformation to obtain a fused feature representation, wherein the fused feature representation corresponds to a plurality of feature nodes, and each feature node is associated with a specific medical concept; the Bayesian reasoning engine is used to receive the fused feature representation, establish a conditional probability dependency model between feature nodes based on a pre-built expert knowledge graph, and generate a probabilistic feature association representation;

[0010] An adaptive feature optimizer is used to combine and integrate the fused feature representation and the probabilistic feature association representation to generate a comprehensive feature representation; based on the comprehensive feature representation, the similarity between samples is calculated, and in combination with the characteristics of individual patient data, the weight coefficients of different modal features are adaptively generated using an attention mechanism; the weight coefficients are used to weight the different modal features in the enhanced feature matrix; the weighted different modal features are recombined to generate an optimized feature representation;

[0011] The intelligent diagnosis analysis unit is used to input the optimized feature representation into a preset disease diagnosis model, calculate the probability score of each diagnosis category, and generate a preliminary diagnosis result; calculate the confidence score of the preliminary diagnosis result, and when the confidence score is lower than a preset threshold, extract the diagnosis information of similar cases from a preset case knowledge base as an auxiliary diagnosis basis; and generate a diagnosis report including main diagnosis categories, diagnosis basis and recommended examination items based on the preliminary diagnosis result, confidence score and auxiliary diagnosis basis.

[0012] This application has the following beneficial technical effects:

[0013] (1) Through the dual-engine fusion mechanism of deep neural network and Bayesian reasoning engine, the advantages of data-driven deep learning and knowledge-driven probabilistic reasoning are complemented, which not only fully utilizes the implicit patterns in massive data, but also integrates the prior information of expert knowledge, significantly improving the effect of multimodal data fusion. (2) The attention mechanism is used to dynamically optimize the weights of different modal features, and adaptively adjusts them according to the characteristics of individual patient data, so that the system can automatically adjust the importance of various types of data according to the specific situation of each patient, realizing personalized diagnostic analysis. (3) A diagnostic decision mechanism based on confidence threshold is established. When the diagnostic confidence is insufficient, similar cases are automatically extracted from the case knowledge base as auxiliary basis, forming a double-guarantee diagnostic decision system, which effectively reduces the risk of misdiagnosis. (4) The diagnostic report output by the system not only contains the diagnostic results, but also provides detailed diagnostic basis and recommended examination items, which enhances the interpretability and traceability of the diagnostic results, and is conducive to doctors' understanding and verification of the system's diagnostic recommendations. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of a medical diagnosis intelligent decision-making system based on multimodal data fusion provided in the first embodiment of the present application. DETAILED DESCRIPTION

[0015] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0016] The first embodiment of the present application provides a medical diagnosis intelligent decision-making system based on multimodal data fusion. Figure 1 , which is a schematic diagram of the first embodiment of the present application. Figure 1 The first embodiment of the present application provides a medical diagnosis intelligent decision-making system based on multimodal data fusion for detailed description.

[0017] The medical diagnosis intelligent decision-making system based on multimodal data fusion comprises a patient information acquisition unit 101, a feature processing unit 102, a modality fusion engine 103, an adaptive feature optimizer 104 and an intelligent diagnosis analysis unit 105.

[0018] The patient information collection unit 101 is used to collect the patient's medical image data, biometric data and electronic medical record text data.

[0019] In this embodiment, the patient information collection unit 101 is mainly responsible for acquiring various types of medical data of the patient from multiple data sources in real time. Specifically, the module collects multimodal data including medical imaging data, biometric data and electronic medical record text data through the standard interface of the hospital information system.

[0020] In terms of the collection of medical imaging data, the module connects to the hospital's imaging archiving and communication system through a standard interface to obtain the patient's CT, MRI, X-ray and other medical imaging data. The collected imaging data includes the original image file and its related metadata information, such as scanning parameters, imaging time, etc. The system supports a variety of common medical imaging formats.

[0021] For the collection of biometric data, the module can connect with the hospital's test information system and electronic medical record system through standard interfaces to obtain the patient's various physiological index data in real time. These data include blood routine, biochemical indicators, electrocardiogram data, body temperature, blood pressure, respiratory rate and other data. The system uses standard medical data exchange formats for data transmission and storage to ensure the standardization and interoperability of data.

[0022] In terms of electronic medical record text data, the patient information collection unit obtains structured and unstructured text data including chief complaint, current medical history, past medical history, physical examination, etc. through the hospital's electronic medical record system API interface. For unstructured text, the system uses a standard coding format for storage and retains the original document structure information.

[0023] To ensure data security and privacy protection, the module implements data desensitization during data collection and encrypts the patient's identity information. At the same time, the module also includes a data quality control mechanism that can detect and mark quality issues such as outliers and missing values ​​in the data and generate corresponding data quality reports.

[0024] After data collection is completed, the module will perform preliminary preprocessing on the collected data, including data format unification, timestamp alignment and other operations, to provide standardized input data for subsequent feature processing units. In addition, the module also maintains a data cache mechanism that can temporarily store data when the network connection is interrupted to ensure the continuity and integrity of data collection.

[0025] In practical applications, this module can be implemented using a distributed architecture, supporting multi-node parallel data collection, and ensuring system stability and scalability through a load balancing mechanism. The module also provides a detailed logging function to record various events and abnormal situations during data collection, facilitating system maintenance and problem diagnosis.

[0026] Furthermore, the patient information collection unit is specifically used for:

[0027] For medical imaging data, it communicates with imaging devices through interfaces that comply with the DICOM standard, obtains image files and their metadata, including imaging parameters and scanning time, and improves the quality of image data through pre-processing steps such as denoising and format standardization;

[0028] For biometric data, it interacts with biosensors through wireless or wired connections to obtain patients' physiological indicators in real time, and synchronizes multiple sampled data in combination with timestamp information, while using anomaly detection mechanisms to mark possible inaccurate data;

[0029] For electronic medical record text data, the patient information collection unit extracts the patient's structured and unstructured text content from the hospital's information system through an API interface based on the HL7 protocol standard, and performs encoding conversion on the unstructured text to ensure format uniformity and data integrity.

[0030] The patient information acquisition unit is designed to efficiently and accurately collect the patient's medical imaging data, biometric data and electronic medical record text data from multiple sources, providing high-quality input for multimodal data fusion. This module adopts targeted acquisition and processing methods according to the characteristics of different data sources to ensure the quality, consistency and integrity of the data to meet the processing requirements of subsequent modules.

[0031] For medical imaging data, the patient information acquisition unit communicates with the imaging equipment (such as CT, MRI, and X-ray equipment) used in the hospital through an interface that complies with the DICOM (Digital Imaging and Communications in Medicine) standard, thereby directly acquiring image files and their related metadata. Metadata includes key information such as imaging parameters (such as resolution, contrast, equipment model) and scanning time, which can provide support for subsequent image processing and diagnosis. During the data acquisition process, the module first pre-processes the original image data and uses denoising technology to filter out random noise introduced by equipment performance or imaging environment, such as using an algorithm based on Gaussian filtering to improve image clarity. At the same time, the image data is format-standardized during the acquisition stage and uniformly converted into a format that can be directly recognized by subsequent modules, such as PNG format. Through these steps, the module can effectively improve the quality and consistency of image data and provide high-precision visual feature input for the feature processing unit.

[0032] For biometric data, the patient information collection unit establishes a connection with the biosensor devices (such as blood pressure monitors, heart rate monitors, and blood oximeters) worn by the patient through wireless (such as Bluetooth, Wi-Fi) or wired interfaces to collect key physiological indicators including blood pressure, heart rate, and blood oxygen saturation in real time. In order to ensure the time consistency of these data in multiple samplings, the module uses timestamp information to synchronize the collected data to ensure that the data provided by different devices can be analyzed and fused under the same time reference. In addition, the module's built-in anomaly detection mechanism can identify and mark the collected values ​​that may have errors, such as abnormal fluctuations caused by sensor failure, patient movement, or signal interference. For the detected abnormal data, the module will generate corresponding quality tags for subsequent modules to screen or process in the analysis. This mechanism ensures the high reliability and analytical applicability of biometric data.

[0033] For electronic medical record text data, the patient information collection unit extracts the patient's electronic medical record content from the hospital information system through an API interface based on the HL7 (Health Level 7) protocol standard, including structured data (such as diagnosis codes, test results) and unstructured data (such as chief complaint description, medical history records). For structured data, the module directly parses and stores it according to the established field mapping rules, while for unstructured text data, encoding conversion technology is used to convert the text content into a unified format, such as UTF-8 encoding format. This encoding conversion ensures the consistency of multi-source text data in content and format. In addition, the module pre-processes the unstructured text data, such as removing irrelevant special characters, retaining key information, and marking possible incomplete or missing content, so as to provide high-quality input for subsequent semantic feature extraction.

[0034] Through the above-mentioned methods, the patient information collection unit can make full use of data interfaces and standards from different sources to ensure that the collected medical imaging data, biometric data and electronic medical record text data meet the requirements of multimodal data fusion processing in terms of quality, consistency and integrity. The implementation technology of this module has been carefully designed to meet the data collection needs in the modern medical environment and provide a solid data foundation for the intelligent diagnosis system.

[0035] The feature processing unit 102 is used to extract visual feature vectors from medical image data; extract physiological index feature vectors from biometric data; and extract semantic feature vectors from electronic medical record text.

[0036] The function of the feature processing unit 102 is to extract the corresponding feature vectors from medical image data, biometric data and electronic medical record text data, and provide structured input data for the modality fusion engine. This module aims to efficiently and accurately extract the core features of different modal data, and designs specific extraction methods and optimized processing steps for each modality to ensure that subsequent modules can fully utilize these features for deep learning and diagnostic modeling.

[0037] For medical image data, the feature processing unit uses specialized image processing and feature extraction techniques. First, the module preprocesses the input medical images, including denoising, grayscale normalization, and resizing. The denoising process uses Gaussian filters or other noise reduction algorithms to remove random noise in the image caused by equipment or environmental factors to improve data quality. The purpose of grayscale normalization is to adjust the pixel values ​​of the image to a uniform dynamic range, such as [0, 1] or [0,255], to reduce the differences in data distribution. Resizing adjusts the width and height of the image according to the input requirements of the model, for example, to 224×224 pixels to ensure that the input dimensions of subsequent feature extraction are consistent. After preprocessing, the module uses pre-trained deep convolutional neural networks (such as ResNet) to extract multi-level visual features. These networks extract low-level features such as edges, textures, and local structures in the image through convolution operations, and capture higher-level semantic information in the image layer by layer. The final extracted visual feature vector represents potential diagnosis-related information in the medical image data, such as texture patterns or structural abnormalities in certain lesion areas.

[0038] For biometric data, the module designs a series of feature extraction algorithms for time series or numerical data. First, the module receives the patient's physiological indicator data, such as blood pressure, heart rate, blood oxygen saturation, blood sugar level, etc. For these data, the module will perform missing value filling and outlier detection. If there is missing data, the module uses interpolation or prediction methods based on time series correlation to fill the missing values; for outliers, the module marks and replaces them through statistical analysis or machine learning algorithms. Then, the module calculates statistical features from each physiological indicator, including mean, variance, maximum, minimum, peak, rate of change, etc. For indicators with time series properties (such as heart rate changes), the module further extracts frequency domain features, such as extracting frequency components or rhythm features through fast Fourier transform (FFT). In addition, the module normalizes all biometric data to a unified numerical range, such as zero mean unit variance, through standardization processing to reduce the magnitude difference between features. The final generated physiological indicator feature vector represents the patient's physiological state with a fixed length to support subsequent diagnostic analysis.

[0039] For electronic medical record text data, the module uses natural language processing (NLP) technology to extract semantic features. First, the module performs word segmentation and preprocessing on the input text data, including removing stop words, punctuation marks, and meaningless characters. At the same time, for medical terms, the module uses the vocabulary or ontology library in the medical field for standardization and uniform encoding. After preprocessing, the module uses a pre-trained language model (such as BERT) to semantically encode the text data. Each piece of text is decomposed into a series of word embeddings, and the contextual dependencies are learned through the multi-layer Transformer structure of the language model, and finally a fixed-dimensional text feature vector is generated. In addition, in order to enhance semantic expression, the module also uses the attention mechanism to focus on key content related to disease diagnosis in the text, such as symptom descriptions, test results, and treatment recommendations. The generated semantic feature vector accurately captures the implicit medical information in the electronic medical record and provides a structured expression of language data for multimodal fusion.

[0040] Through the above steps, the feature processing unit 102 completes the feature extraction task of the three modal data, and generates visual feature vectors, physiological indicator feature vectors and semantic feature vectors. These feature vectors have a unified format and dimension, and can seamlessly connect to the subsequent modal fusion engine. The module is designed with full consideration of the characteristics and requirements of different modal data, and ensures data quality and feature expression capabilities through targeted preprocessing, feature extraction and optimization methods. The implementation of this module not only adapts to the complexity of medical diagnostic data, but also lays a solid foundation for subsequent intelligent diagnosis processes.

[0041] Furthermore, the feature processing unit includes a medical image preprocessing submodule for performing size normalization, denoising and multi-scale feature enhancement on the collected medical image data;

[0042] The multi-scale feature enhancement extracts local and global features based on images of different resolutions, so that low-resolution features highlight the overall structure of the image and high-resolution features retain detailed information of local lesions, thereby generating accurate visual feature vectors.

[0043] The medical image preprocessing submodule in the feature processing unit is designed to perform comprehensive preprocessing and feature extraction on the collected medical image data to ensure that high-quality and accurate visual feature vectors are provided for subsequent modules. This module combines a variety of image processing technologies and is suitable for processing different types of medical image data (such as CT, MRI, X-ray, etc.) to meet the needs of complex medical diagnosis scenarios.

[0044] First, this submodule performs a size normalization operation on the input medical image data to ensure that the image data has a uniform spatial resolution. This process is mainly achieved through interpolation methods, which adjusts all input images to a specified size, such as 224×224 pixels or other standard sizes suitable for deep learning networks. The purpose of size normalization is to eliminate the resolution differences caused by different image acquisition devices or parameters, so that the scale characteristics of the image data are consistent, thereby avoiding affecting the accuracy of subsequent feature extraction.

[0045] Next, the submodule denoises the image data to remove random noise caused by equipment noise, imaging conditions, or patient motion. Specifically, this submodule uses denoising algorithms based on the spatial domain or frequency domain, such as Gaussian filtering, bilateral filtering, or adaptive noise removal algorithms. During the denoising process, the module prioritizes retaining key structural information in the image, such as edge and texture features, while reducing the impact of noise on image quality. This approach strikes a balance between denoising and detail preservation, providing clearer, interference-free input data for feature extraction.

[0046] In addition, in order to further improve the effect of feature extraction, this submodule designs a multi-scale feature enhancement technology to extract global and local feature information by processing different resolutions of image data. In multi-scale feature enhancement, image data is decomposed into multiple resolutions, such as low-resolution and high-resolution versions. For low-resolution images, the module focuses on extracting global structural features, such as the shape of organs and the overall distribution of lesions; while for high-resolution images, the module focuses on capturing local detail information, such as subtle changes in the edges of lesions or abnormal texture patterns. This process is usually achieved through pyramid decomposition methods or multi-scale filtering techniques, which can simultaneously highlight the global information and detail features of the image.

[0047] After extracting global and local features, the module integrates features of different scales to generate the final visual feature vector. These feature vectors accurately represent important diagnostic information in medical images, such as the shape, location, and detail features of the lesion area. This multi-scale feature fusion method can not only improve the expressiveness of visual features, but also enhance the adaptability of image data in multimodal fusion.

[0048] Through the above steps, the medical image preprocessing submodule can efficiently process and optimize the collected medical image data, making it have a uniform size, high-quality denoising effect, and rich multi-scale feature representation. The final generated visual feature vector provides a high-quality basic input for multimodal data fusion and medical diagnosis.

[0049] Furthermore, the feature processing unit includes a biometric analysis submodule for receiving the patient's biometric data and extracting short-term rate of change and long-term trend features based on time series analysis;

[0050] The analysis submodule includes an outlier detection function, which identifies and corrects abnormal biometric data by combining patient historical data and population baseline data to ensure the accuracy and consistency of the generated physiological indicator feature vector.

[0051] The biometric analysis submodule in the feature processing unit is designed to process the patient's biometric data, comprehensively extract its short-term and long-term change characteristics, and ensure the accuracy and consistency of the data, providing reliable physiological indicator feature vectors for subsequent multimodal fusion and medical diagnosis. This submodule combines time series analysis technology and outlier detection function to deeply analyze and optimize the collected biometric data.

[0052] In the process of processing biometric data, the submodule first receives a variety of physiological indicator data of the patient, such as blood pressure, heart rate, blood oxygen saturation, body temperature, etc. These data are usually of time series nature, and record the changes of the patient's physiological state over time through dynamic acquisition. The submodule uses time series analysis methods to extract short-term change rate and long-term trend characteristics from the original data. The short-term change rate reflects the fluctuation amplitude and frequency of the patient's physiological indicators in a short period of time (such as minutes to hours), which is crucial for identifying acute lesions or emergencies. The long-term trend characteristics reveal the changing trend of the patient's overall health status, such as the progression of chronic diseases or the recovery process, by observing the change trajectory of biometrics over a longer period of time (such as days or weeks). This combination of short-term and long-term features can provide a comprehensive description of biometrics and enhance the ability to dynamically assess the health status of patients.

[0053] In order to ensure that the generated physiological indicator feature vectors are highly accurate and consistent, this submodule also includes an outlier detection function. Outlier detection analyzes the collected biometric data by combining the patient's historical data and the population baseline data to identify outliers that may have errors. For example, when the heart rate data suddenly deviates from the patient's normal range and is significantly inconsistent with its historical records, the system will mark the data as abnormal. At the same time, through the population baseline data (such as the average indicator range of healthy people or people with similar symptoms obtained by large-scale statistics), the system can further determine whether the abnormal data is caused by equipment failure, environmental interference or patient behavior. For detected outliers, the submodule adopts a correction strategy, such as filling and replacing the data through interpolation methods or based on time series prediction models to ensure the integrity and availability of the data.

[0054] Throughout the data processing process, the biometric analysis submodule focuses on the precise alignment and consistency maintenance of timestamps to ensure that all feature data can be analyzed and fused with the same time reference. Through these steps, the physiological indicator feature vector generated by the module can efficiently represent the dynamic changes of the patient's biometrics and provide standardized and information-rich input for subsequent modules.

[0055] The design of this submodule combines the technical advantages of time series analysis and anomaly detection, and uses personalized patient data and group statistics to achieve accurate processing of biometric data. Whether it is real-time response to acute lesions in clinical applications or tracking long-term trends in chronic disease management, it can provide highly reliable data support, thereby ensuring the performance and stability of the entire medical diagnosis intelligent decision-making system.

[0056] Furthermore, the feature processing unit includes a text semantic parsing submodule for performing word segmentation and standardization of key medical terms on the unstructured text in the electronic medical record;

[0057] The text semantic analysis submodule combines the medical ontology knowledge base to perform semantic enhancement on the extracted text features, ensuring that the generated semantic feature vector can accurately reflect the medical meaning and diagnostic relevance in the text.

[0058] The text semantic parsing submodule in the feature processing unit is specifically used to process unstructured text data in electronic medical records. It aims to convert medical information in natural language form into structured semantic feature vectors to accurately reflect the medical meaning and diagnostic relevance in the text, and provide high-quality text feature input for subsequent multimodal data fusion.

[0059] In the unstructured text processing of electronic medical records, the text semantic parsing submodule first performs a segmentation operation on the input text. This operation breaks down continuous natural language text into basic units with independent meanings by segmenting words or phrases in sentences. For example, "the patient complained of chest pain for three days" will be segmented into "patient", "complaint", "chest pain", "three days", etc. This process uses a segmentation algorithm optimized in the medical field, which can identify common medical terms, abbreviations, and key words related to patient symptoms, diagnosis, and treatment.

[0060] After completing the word segmentation, the submodule performs standardization of key medical terms. Standardization aims to map the medical terms in the word segmentation results to a unified medical ontology or vocabulary, such as ICD codes (International Classification of Diseases). This process not only solves the inconsistency problem of synonyms and abbreviations, such as "hypertension" and "high blood pressure" will be standardized into a unified term, but also gives the terms a clearer meaning through the definition and context information in the medical ontology knowledge base. This standardization process ensures the consistency and comparability of terms in different texts, laying the foundation for generating semantic features.

[0061] After completing the word segmentation and standardization processing, the text semantic parsing submodule combines the medical ontology knowledge base to semantically enhance the extracted text features. The medical ontology knowledge base is a structured knowledge representation that contains medical concepts such as diseases, symptoms, examination items, drugs and treatment methods and their interrelationships. The submodule uses the conceptual relationships defined in the knowledge base (such as causal relationships, hierarchical relationships, and association relationships) to expand and refine text features. For example, the term "chest pain" can be associated with "myocardial infarction", "coronary artery disease" or "non-specific chest pain" in the knowledge base. The text semantic parsing submodule enhances the diagnostic relevance and medical significance of the text through these associations. In addition, the module highlights the features most relevant to the patient's current condition through a semantic weighting mechanism, making the generated semantic feature vector more accurate and targeted.

[0062] Finally, the semantic feature vectors generated by the text semantic parsing submodule are high-dimensional numerical representations that can comprehensively reflect the key information and medical significance in the text. These feature vectors not only retain the important diagnosis-related information in the original text, but also highlight the domain-specific background information provided by the medical ontology knowledge base through semantic enhancement. This multi-level text parsing and semantic processing enables the submodule to process complex medical text data with high accuracy and reliability.

[0063] The design of the text semantic analysis submodule fully considers the unstructured characteristics and domain expertise of medical texts. Through the combination of word segmentation, standardization processing and semantic enhancement, the text information is effectively converted into structured features that can be used for multimodal fusion. This module can adapt to electronic medical record text data in different formats and plays an important role in improving diagnostic accuracy and decision support capabilities.

[0064] The modal fusion engine 103 includes a deep neural network and a Bayesian reasoning engine, wherein the deep neural network is used to receive the visual feature vector, the physiological indicator feature vector and the semantic feature vector output by the feature processing unit, and learn the association relationship between different modal features through multi-layer nonlinear transformation to obtain a fused feature representation; the Bayesian reasoning engine is used to receive the fused feature representation, establish a conditional probability dependency model between features based on a pre-built expert knowledge graph, and generate a probabilistic feature association representation.

[0065] The modality fusion engine 103 includes a deep neural network and a Bayesian reasoning engine, and its function is to fuse the data features of different modalities, realize deep correlation relationship learning and probabilistic modeling, and provide reliable data support for medical diagnosis decisions.

[0066] The deep neural network receives multimodal feature data output by the feature processing unit, including visual feature vectors extracted from medical images, physiological indicator feature vectors extracted from biometric data, and semantic feature vectors extracted from electronic medical record text. Due to the different sources and structures of these features, they first need to be mapped to a unified feature space through feature alignment operations. At this stage, visual features, physiological features, and semantic features are preprocessed through specific mapping methods, such as using a fully connected layer to adjust the dimension of the feature, and combining batch normalization operations to normalize the numerical range of the feature, thereby enhancing the stability and generalization ability of the deep network.

[0067] After feature alignment is completed, the deep neural network further extracts the correlation information between modalities through multi-layer nonlinear transformation and interactive learning modules. The interactive learning module adopts a layer-by-layer processing method, and considers the correlation between different modal features in each layer. For example, visual features can be combined with physiological features and semantic features to extract characteristic patterns related to a certain disease. This nonlinear transformation and interactive learning can capture the implicit complex relationships between multimodal data. For example, a symptom may appear as a change in a specific structure in the image, accompanied by abnormal physiological indicators and the patient's chief complaint description.

[0068] The deep neural network ultimately generates a set of fused feature representations, which are further subdivided into multiple logical feature nodes, each of which corresponds to a specific medical concept. For example, one feature node may correspond to the concept of "hypertension" and another node may correspond to "cardiac hypertrophy." These feature nodes are the core results of multimodal feature fusion in the system, providing a logical and data-based representation of the disease or symptom.

[0069] The task of the Bayesian inference engine is to further model the conditional probability relationship between feature nodes based on the fusion features generated by the deep neural network and the expert knowledge graph in the medical field. The knowledge graph is pre-built and contains a large number of medical concepts and their relationships, such as the relationship between diseases and symptoms, and the relationship between symptoms and test results. The Bayesian inference engine identifies the dependency structure between these feature nodes by analyzing the joint distribution of them. For example, "high blood pressure" may be associated with "heart hypertrophy" through a probabilistic relationship, and "heart hypertrophy" may be further associated with "chest pain".

[0070] In the specific implementation, the Bayesian reasoning engine calculates the conditional probability of each feature node based on the fused feature values ​​of the feature node and its association in the knowledge graph. These conditional probabilities are gradually constructed into a global probability distribution in an iterative manner, describing the joint behavior of all feature nodes in the system. This joint probability distribution not only reflects the diagnostic information of the patient's current data, but also combines prior knowledge in the medical field to provide a more comprehensive description of feature associations.

[0071] Ultimately, the output of the Bayesian inference engine is a probabilistic feature association representation that combines the fusion features learned by the deep neural network and the domain prior knowledge provided by the knowledge graph. These probabilistic features provide a reliable input basis for subsequent modules, supporting the accuracy and interpretability of the system in diagnostic decisions. The module adopts an efficient computing framework in its implementation to cope with the processing needs of multimodal big data, while ensuring the stability and robustness of the system in complex medical scenarios through outlier detection and data recovery mechanisms.

[0072] Furthermore, the deep neural network is specifically used for:

[0073] According to the following formula 1, the modal feature space is uniformly mapped:

[0074] ;

[0075] in, Represents input modality features, including visual feature vectors , physiological index feature vector and semantic feature vector , ; is the modal eigenvector after unified feature space mapping; It is The weight matrix corresponding to the modalities is used to map the modal features to a unified feature space; It is The bias vector corresponding to the mode; is the activation function, including ReLU or tanh; the weight matrix The initial value of can be set by random initialization, and the recommended value range is normal distribution . Bias vector Can be initialized to a zero vector.

[0076] According to the following formula 2, the interaction optimization between modalities is achieved:

[0077] ;

[0078] in, In order to generate interactively optimized feature representations through inter-modal association learning, this process captures the correlation between features of different modalities through the attention mechanism.

[0079] Represents the query vector, which is used to capture the relationship between the current modality and other modal features; represents the key vector, which represents the reference for modal feature comparison; represents the dimension of the key vector; represents a value vector, which is used to store the actual feature information of each modality; the query vector, key vector and value vector are calculated according to the following formula 3-5:

[0080] ;

[0081] ;

[0082] ;

[0083] in, It is the visual feature vector after mapping to the unified space; It is the characteristic vector of physiological indicators after being mapped to the unified space; It is the semantic feature vector after being mapped to the unified space; is the weight matrix of the query vector; is the weight matrix of the key vector; is the weight matrix of the value vector;

[0084] According to the following formula 6, the fusion feature representation is calculated :

[0085] ;

[0086] in, is the adaptive weight, calculated according to the following formula 7:

[0087] ;

[0088] in, Learnable weight matrices; is a learnable bias term.

[0089] Furthermore, the Bayesian reasoning engine is specifically used for:

[0090] According to the following formula 8, the fusion feature is expressed as Mapped to the preliminary probability space, generating preliminary probability values ​​for each feature node:

[0091] ;

[0092] in, Indicates The initial probability of feature nodes;

[0093] is the weight matrix of the feature map; is the bias term of the feature map; is the Sigmoid function; is the feature vector after multimodal fusion, representing the feature node The data representation comes from the output of a deep neural network.

[0094] The Bayesian reasoning engine combines the expert knowledge graph to construct the conditional probability dependency relationship between feature nodes and calculates the feature node according to the following formula 9. The conditional probability :

[0095] ;

[0096] in, It is a feature node defined in the knowledge graph The parent node set of It is a feature node With its parent node The association weight of Characteristic node The eigenvector of For the parent node The eigenvector of is the feature vector and The Euclidean distance of

[0097] According to the following formula 10, the global joint probability distribution is obtained :

[0098] ;

[0099] in, , represents the set of global feature nodes; , represents the corresponding feature vector set; is the number of feature nodes; Characteristic node The conditional probability of

[0100] The initial probability is obtained by variational inference method and conditional probability Perform joint optimization to generate optimized posterior probability distribution , the optimization objective function adopts the following formula 11:

[0101] ;

[0102] in, refers to the KL divergence; the optimization objective function dynamically adjusts the approximate distribution by minimizing the KL divergence , making it as close to the target distribution as possible ; is the optimized approximate distribution, representing the characteristic node The final probability value of The result of modeling conditional probability; optimized posterior probability Represents the final probabilistic association representation of feature nodes, which is used for new diagnosis decisions. , is the number of feature nodes. Through the above process, the Bayesian reasoning engine not only realizes the modeling of complex relationships between feature nodes, but also ensures the accuracy and applicability of the final probability distribution through dynamic optimization.

[0103] The adaptive feature optimizer 104 is used to combine and integrate the fused feature representation and the probabilistic feature association representation to generate a comprehensive feature representation; calculate the similarity between samples based on the comprehensive feature representation, and use the attention mechanism to adaptively generate weight coefficients of different modal features in combination with the characteristics of individual patient data; use the weight coefficients to weight different modal features in the enhanced feature matrix; and recombine the weighted different modal features to generate an optimized feature representation.

[0104] The adaptive feature optimizer 104 is responsible for further optimizing and integrating the feature representation output by the modal fusion engine. The module first receives the fused feature representation and the probabilistic feature association representation through the feature combiner, and uses nonlinear transformation to map the two 1024-dimensional feature vectors to a unified feature space. In the specific implementation, a multi-layer perceptron is used to transform the two features separately, and then the transformed features are added through the residual connection mechanism to generate the initial comprehensive feature representation.

[0105] After obtaining the comprehensive feature representation, the module implements a feature enhancement mechanism based on sample similarity. First, a feature index tree is constructed, and a local sensitive hashing algorithm is used to accelerate the similarity calculation process. For each input sample, the cosine similarity is calculated to find the nearest neighbor sample set in the feature space. The local structure of the feature is taken into account when calculating the similarity, and the number of nearest neighbor samples is determined by an adaptive threshold, which is usually controlled between 5 and 10 samples.

[0106] The core of the module is the adaptive attention mechanism, which can dynamically adjust the importance of different modal data according to the individual characteristics of the patient. The attention calculation is divided into two steps: first, a query vector is constructed based on the patient's clinical characteristics, which encodes the patient's key medical information; then, the correlation score between the query vector and each modal feature is calculated to obtain the modality-level attention weight. The attention weight is normalized by the softmax function to ensure that the sum of the weights is 1.

[0107] After obtaining the attention weights, the module performs weighted reorganization of the features of different modalities. Specifically, the attention weights are multiplied element-by-element with the feature vectors of the corresponding modality, and then the weighted feature vectors are concatenated. In order to maintain the expressiveness of the features, the module adopts a feature recalibration mechanism to adjust the amplitude of the features through a learnable scaling factor.

[0108] In order to improve the optimization effect, the module also implements an iterative optimization strategy. In each iteration, the sample similarity and attention weight are recalculated based on the current feature representation, and then the feature representation is updated. The iteration process is controlled by an early stopping strategy, and the iteration is stopped when the change in the feature representation is less than the preset threshold. Convergence is usually achieved after 3-5 iterations.

[0109] The entire process of feature optimization uses gradient checkpoint technology, which reduces memory usage while ensuring performance. The module also implements real-time visualization of feature representation, mapping high-dimensional features to a two-dimensional plane through the t-SNE dimensionality reduction method, making it easier to observe the optimization effect. All intermediate states have detailed log records and support breakpoint recovery, ensuring the traceability and reproducibility of the processing process.

[0110] The optimized feature representation of the final output maintains a unified dimension of 1024 dimensions, which not only contains the complementary information of multimodal data, but also retains the differences in individual patient characteristics. The module adopts an efficient parallel computing architecture, and the average processing time is controlled within 100 milliseconds to meet the needs of real-time processing. In addition, the module also provides a feature interpretation interface that can output the medical semantic interpretation corresponding to each feature dimension, enhancing the interpretability of the feature representation.

[0111] Furthermore, the adaptive feature optimizer combines the similarity between samples to perform feature enhancement, and by comparing the multimodal feature representation of the patient with the feature representation of known cases in the database, identifies similar cases and extracts relevant features, thereby enhancing the feature matrix of the current patient;

[0112] The enhanced feature matrix is ​​combined with the attention mechanism to optimize the feature importance between modalities and improve the diagnostic sensitivity of multimodal data fusion.

[0113] The design of the adaptive feature optimizer aims to enhance and optimize the feature representation of patients by combining the similarity between samples and the attention mechanism, thereby improving the sensitivity and accuracy of multimodal data fusion in medical diagnosis. The implementation of the module involves multiple steps to ensure dynamic processing of patient data and accurate optimization of feature representation.

[0114] First, the adaptive feature optimizer starts with the optimized multimodal feature representation of the patient and compares it with the feature representation of known cases stored in the database. The known case database contains historical case data confirmed by experts, and its feature representation has been processed by multimodal fusion, which is highly representative and diagnostically relevant. By calculating the similarity between the patient feature representation and the feature representation of each case in the database, the module can identify the most similar case groups. The similarity calculation not only considers the matching of single modality data, but also evaluates the overall similarity by combining the correlation between multimodal features, such as the common patterns of visual features, semantic features, and physiological features in specific diagnostic scenarios. The identification of similar cases provides a reference for further optimization of the current patient feature representation.

[0115] After identifying similar cases, the module extracts relevant feature information from the feature representations of these cases to enhance the feature matrix of the current patient. The feature enhancement process mines patterns that are important to the current patient's features by analyzing the diagnostic results and corresponding features of similar cases. For example, the combination of certain imaging features and biological features may indicate the presence of a specific disease. This enhancement mechanism supplements the potential missing information in the patient's features by weighted fusion of the patient's feature matrix with the highly correlated features of similar cases, thereby improving the overall diagnostic accuracy.

[0116] Next, the enhanced feature matrix is ​​further optimized through the attention mechanism to adjust the importance of different modal features. The attention mechanism dynamically assigns weights by analyzing the contribution value of each modality in the enhanced feature matrix to highlight the modal features that are most critical to diagnosis. For example, for some patients, physiological indicator features may be more important in diagnosis. The attention mechanism will adaptively increase their weights while suppressing modal features that contribute less to diagnosis. This weight allocation process is dynamically adjusted according to the specific characteristics of the patient and the diagnostic context, ensuring that the final optimized feature representation can fully reflect the individual characteristics of the patient and is highly relevant to the diagnostic goal.

[0117] Finally, the adaptive feature optimizer outputs an enhanced and optimized feature matrix that contains comprehensive information of multimodal data and key features of personalized diagnosis. By combining feature enhancement of similar cases and dynamic optimization of attention mechanism, this module significantly improves the diagnostic sensitivity of multimodal data fusion, enabling the system to more accurately identify complex disease patterns and provide high-quality input for subsequent intelligent diagnosis and analysis units.

[0118] This design not only takes into account the utilization of historical case data, but also enhances the diagnostic relevance of multimodal data through intelligent feature optimization strategies.

[0119] Furthermore, the adaptive feature optimizer adopts a multi-round iterative optimization strategy, recalculates the comprehensive feature representation and modal weight distribution in each iteration, suppresses low-confidence modal features, and gradually optimizes the final feature representation;

[0120] In each iteration, the optimization step size is dynamically adjusted according to the degree of feature convergence and changes in patient characteristics to ensure the accuracy and stability of the final feature representation.

[0121] The multi-round iterative optimization strategy of the adaptive feature optimizer aims to achieve high accuracy and stability of the final feature representation by gradually adjusting the comprehensive feature representation and modality weight distribution, providing a solid data foundation for subsequent diagnostic decisions. The design of the module fully considers the heterogeneity of multimodal data and the dynamic changes of individual patient characteristics, and gradually optimizes the feature representation through a flexible iterative mechanism to ensure that the final result can accurately reflect the patient's medical information.

[0122] In each iteration, the adaptive feature optimizer first recalculates the comprehensive feature representation based on the current fused feature representation. This calculation combines the contributions of all modal features and considers the correlation between multiple modalities to generate a feature representation from a global perspective. The comprehensive feature representation is continuously updated in each round of iteration, gradually integrating the advantageous information of each modality while eliminating redundancy and noise, thereby improving the overall quality of the representation.

[0123] At the same time, the module dynamically adjusts the weight distribution of the importance of each modal feature. By analyzing the diagnostic relevance of each modal feature in the current iteration, the module adaptively reallocates the modal weights. This weight distribution takes diagnostic sensitivity as the core principle, highlighting those modal features that have the greatest impact on the diagnostic results. For example, in some disease scenarios, visual features may dominate, while in other cases, biological or text features may be more critical. By iteratively updating the modal weight distribution, the module can continuously optimize the performance of multimodal fusion.

[0124] For low-confidence modal features, the adaptive feature optimizer adopts a suppression strategy in iteration. Low-confidence features are usually caused by low data quality or weak correlation with the current diagnostic target. The module dynamically lowers the weight of low-confidence features to reduce their impact on the comprehensive feature representation. This strategy can effectively improve the robustness of the optimization process and avoid negative interference of low-quality features on the diagnostic results.

[0125] During multiple rounds of iterations, the adaptive feature optimizer dynamically adjusts the optimization step size according to the convergence degree of the features and the changes in patient characteristics. Step size adjustment is a key link in the optimization strategy, which directly affects the convergence speed and accuracy of the feature representation. When the feature representation is close to convergence, the module automatically reduces the step size to achieve more refined optimization; in the early iteration stage or when the patient characteristics change significantly, the module increases the step size to speed up the optimization process. This dynamic adjustment mechanism can effectively balance the optimization efficiency and accuracy, ensuring that the final feature representation can converge stably.

[0126] Through multiple rounds of iterations, the feature representation output by the adaptive feature optimizer has higher accuracy and stability based on the full integration of multimodal data. The optimized feature representation can not only fully reflect the individual characteristics of the patient, but also significantly improve the reliability and accuracy of subsequent diagnostic decisions. The implementation method of this module is logically clear and technologically advanced. Those skilled in the art can accurately implement it according to the above instructions, thereby realizing the core value of dynamic optimization in medical diagnosis scenarios.

[0127] The intelligent diagnosis analysis unit 105 is used to input the optimized feature representation into a preset disease diagnosis model, calculate the probability score of each diagnosis category, and generate a preliminary diagnosis result; calculate the confidence score of the preliminary diagnosis result, and when the confidence score is lower than a preset threshold, extract the diagnosis information of similar cases from a preset case knowledge base as an auxiliary diagnosis basis; and generate a diagnosis report including main diagnosis categories, diagnosis basis and recommended examination items based on the preliminary diagnosis result, confidence score and auxiliary diagnosis basis.

[0128] As the last processing link of the system, the intelligent diagnosis and analysis unit 105 is responsible for generating the final medical diagnosis result. This module adopts a multi-stage diagnosis decision process, combined with a machine learning model and an expert knowledge base, to achieve a reliable and explainable diagnosis process.

[0129] In the preliminary diagnosis stage, the module first inputs the optimized 1024-dimensional feature representation into the preset disease diagnosis model. The model uses an ensemble learning framework and contains multiple basic classifiers, including gradient boosting decision trees, deep neural networks, and support vector machines. Each basic classifier processes the input features independently and outputs the probability score of each diagnostic category. The ensemble model integrates the prediction results of each classifier through weighted voting to obtain a more reliable preliminary diagnosis result. The weight value is dynamically adjusted based on the performance on the validation set to ensure the optimal performance of the ensemble model.

[0130] For the preliminary diagnosis results, the module will calculate its confidence score. The confidence calculation adopts the Bayesian uncertainty estimation method, which takes into account the entropy value and prediction variance of the prediction probability distribution. In the specific implementation, the Monte Carlo dropout technology is used to make multiple predictions for the same input and the discreteness of the prediction results is statistically analyzed. The confidence score is normalized between 0 and 1, where 1 represents the highest confidence.

[0131] When the calculated confidence score is lower than the preset threshold (usually set to 0.85), the module will start the auxiliary diagnosis mechanism based on the knowledge base. The system maintains a structured case knowledge base, which contains a large number of diagnosis cases confirmed by experts. Each case in the knowledge base contains complete medical records, diagnosis results and treatment plans. The module retrieves the historical cases most similar to the current case from the knowledge base through vector similarity search, and usually selects the top three cases with the highest similarity as the basis for auxiliary diagnosis.

[0132] When generating the final diagnosis report, the module comprehensively considers the preliminary diagnosis results, confidence assessment, and knowledge base search results. The diagnosis report adopts a hierarchical structure, first giving the main diagnosis categories and their probability scores, and then listing in detail the key evidence supporting the diagnosis, including abnormal test indicators, imaging features, and clinical manifestations. For diagnostic results with low confidence, the report will specially mark the items that need further confirmation, and list the diagnostic points in similar cases as reference.

[0133] In order to provide more valuable clinical advice, the module will automatically generate a list of recommended examination items based on the diagnosis results. These recommendations are based on medical guidelines and expert consensus, taking into account the necessity, urgency and economy of the examination items. At the same time, the module will mark the completed and unfinished examination items to help doctors quickly understand the progress of the diagnosis.

[0134] At the implementation level, the module adopts a microservice architecture, and diagnostic model reasoning, confidence calculation, and knowledge base retrieval are deployed in independent service units. Services communicate asynchronously through message queues to ensure the responsiveness of the system. The module also implements a complete exception handling mechanism, which can properly handle abnormal situations such as model prediction failure and knowledge base query timeout. Detailed logs are recorded for each step of the diagnostic process to support retrospective analysis of diagnostic decisions.

[0135] Through this multi-level diagnostic decision-making mechanism, the module can not only give accurate diagnostic suggestions, but also provide sufficient diagnostic evidence and reasonable examination suggestions, effectively assisting doctors in their clinical decision-making process. At the same time, the design of the module fully considers scalability, which can easily update the diagnostic model and expand the knowledge base to meet the evolving medical diagnosis needs.

[0136] Furthermore, the intelligent diagnosis and analysis unit is specifically used for:

[0137] The optimized feature representation is input into the disease diagnosis model, the probability score of each diagnosis category is calculated, and the prediction results of multiple basic classifiers are integrated through ensemble learning methods to generate a preliminary diagnosis result;

[0138] The confidence of the preliminary diagnosis result is evaluated, and the evaluation combines the discrete degree of the prediction probability, the consistency between the classifiers and the patient's personalized data characteristics to generate a confidence score. When the confidence is lower than a preset threshold, the auxiliary diagnosis mechanism of the case knowledge base is activated;

[0139] Retrieve the case that is most similar to the current patient's characteristics from the case knowledge base, and extract the diagnostic information, treatment plan and related basis of similar cases;

[0140] A diagnostic report is generated based on the preliminary diagnosis results, confidence assessment results and similar case information, and the report includes the main diagnostic category, key evidence supporting the diagnosis, comparative analysis with similar cases and recommended further examinations or treatment items.

[0141] The intelligent diagnosis and analysis unit plays a core role in the medical diagnosis intelligent decision-making system based on multimodal data fusion. It is designed to extract meaningful information from the optimized feature representation and generate accurate and interpretable diagnosis results and reports through efficient classification and auxiliary diagnosis mechanisms.

[0142] First, the intelligent diagnosis and analysis unit receives the optimized feature representation output by the adaptive feature optimizer and passes it as input to the preset disease diagnosis model. The model uses an ensemble learning method to integrate the prediction results of multiple basic classifiers to improve the stability and accuracy of the diagnosis. Basic classifiers may include different types of algorithms such as deep neural networks, support vector machines, random forests, etc. Each classifier processes the input features independently and outputs its own probability score for each diagnostic category. The module weightedly synthesizes these probability scores through ensemble learning, making full use of the advantages of each classifier to generate an overall consistent preliminary diagnosis result.

[0143] After generating the preliminary diagnosis result, the module performs a confidence assessment to determine the reliability of the result. The confidence assessment takes into account multiple factors, including the discreteness of the prediction probabilities of each classifier, the consistency of the prediction results between different classifiers, and the degree of matching with the patient's personalized data characteristics (such as age, gender, and medical history). This comprehensive evaluation process ensures that the confidence score can fully reflect the credibility of the preliminary diagnosis result. When the confidence score is higher than the preset threshold, the preliminary diagnosis result is directly used to generate a diagnosis report; when the confidence score is lower than the threshold, the module automatically starts the auxiliary diagnosis mechanism of the case knowledge base.

[0144] The case knowledge base is an important part of the module, which contains extensive historical case data verified by experts, including case feature representation, diagnosis results, and treatment plans. The intelligent diagnosis analysis unit retrieves the cases with the most similar characteristics to the current patient from the case knowledge base through feature similarity analysis. The characteristics of these similar cases include but are not limited to medical imaging features, biometric features, and semantic features extracted from electronic medical records. The module extracts the diagnostic information, treatment plans, and related evidence of these similar cases as reference data for auxiliary diagnosis.

[0145] Finally, the intelligent diagnostic analysis unit generates a comprehensive diagnostic report based on the preliminary diagnosis results, confidence assessment results, and similar case information. The diagnostic report adopts a clear, hierarchical structure. First, it lists the main diagnostic categories and their basis, including key indicators or patterns of abnormal performance in the characteristics; second, it conducts comparative analysis with similar cases to clarify the similarities and differences between the current patient and historical cases; finally, it provides recommended further examinations or treatment items, such as imaging examinations, laboratory tests, or clinical interventions that need to be performed. Through this design, the diagnostic report not only provides doctors with clear diagnostic conclusions, but also provides in-depth background support for subsequent clinical decisions.

[0146] The entire intelligent diagnosis and analysis unit focuses on data reliability, result interpretability and operation automation during implementation.

[0147] The following is a detailed code implementation, covering all contents of the modal fusion engine, adaptive feature optimizer, and intelligent diagnosis and analysis unit:

[0148] import numpy as np

[0149] import torch

[0150] import torch.nn as nn

[0151] import torch.optim as optim

[0152] import torch.nn.functional as F

[0153] # Modal Fusion Engine

[0154] class MultiModalFusion(nn.Module):

[0155] def __init__(self, input_dims, unified_dim, attention_dim):

[0156] super(MultiModalFusion, self).__init__()

[0157] # Initialize weight matrix and bias vector

[0158] self.visual_transform = nn.Linear(input_dims['visual'],unified_dim)

[0159] self.physical_transform = nn.Linear(input_dims['physical'],unified_dim)

[0160] self.semantic_transform = nn.Linear(input_dims['semantic'],unified_dim)

[0161] self.query_transform = nn.Linear(unified_dim 3, attention_dim)

[0162] self.key_transform = nn.Linear(unified_dim 3, attention_dim)

[0163] self.value_transform = nn.Linear(unified_dim 3, unified_dim)

[0164] self.alpha_transform = nn.Linear(unified_dim, 1)

[0165] def forward(self, features):

[0166] """

[0167] :param features: dict, including visual, physiological indicators and semantic features

[0168] :return: The fused feature vector

[0169] """

[0170] # Unified feature space mapping

[0171] F_v_prime = F.relu(self.visual_transform(features['visual']))

[0172] F_p_prime = F.relu(self.physical_transform(features['physical']))

[0173] F_s_prime = F.relu(self.semantic_transform(features['semantic']))

[0174] # Inter-modal interaction optimization

[0175] combined_features = torch.cat([F_v_prime, F_p_prime, F_s_prime], dim=-1)

[0176] Q = self.query_transform(combined_features)

[0177] K = self.key_transform(combined_features)

[0178] V = self.value_transform(combined_features)

[0179] d_k = Q.size(-1)

[0180] attention_weights = F.softmax((Q @ K.transpose(-2, -1)) / np.sqrt(d_k), dim=-1)

[0181] F_inter = attention_weights @ V

[0182] # Fusion feature representation

[0183] alpha_v = F.softmax(self.alpha_transform(F_v_prime + F_inter),dim=-1)

[0184] alpha_p = F.softmax(self.alpha_transform(F_p_prime + F_inter),dim=-1)

[0185] alpha_s = F.softmax(self.alpha_transform(F_s_prime + F_inter),dim=-1)

[0186] F_fusion = alpha_v F_v_prime + alpha_p F_p_prime + alpha_s F_s_prime

[0187] return F_fusion

[0188] # Adaptive feature optimizer

[0189] class DynamicOptimization(nn.Module):

[0190] def __init__(self, fusion_dim, attention_dim):

[0191] super(DynamicOptimization, self).__init__()

[0192] self.attention_layer = nn.Linear(fusion_dim, attention_dim)

[0193] self.weight_layer = nn.Linear(attention_dim, 1)

[0194] def forward(self, fused_features, historical_cases):

[0195] """

[0196] :param fused_features: The fused features of the current patient

[0197] :param historical_cases: Historical case characteristics

[0198] :return: Optimized feature representation

[0199] """

[0200] # Calculate the similarity with historical cases

[0201] similarities = F.cosine_similarity(fused_features.unsqueeze(1), historical_cases, dim=-1)

[0202] most_similar_cases = historical_cases[similarities.argmax(dim=-1)]

[0203] # Attention mechanism optimization

[0204] attention_input = fused_features + most_similar_cases

[0205] attention_weights = F.softmax(self.weight_layer(F.relu(self.attention_layer(attention_input))), dim=-1)

[0206] optimized_features = attention_weights fused_features + (1- attention_weights) most_similar_cases

[0207] return optimized_features

[0208] # Intelligent Diagnosis and Analysis Unit

[0209] class DiagnosticDecision(nn.Module):

[0210] def __init__(self, feature_dim, num_classes):

[0211] super(DiagnosticDecision, self).__init__()

[0212] self.disease_model = nn.Linear(feature_dim, num_classes)

[0213] self.knowledge_base = {} # Case knowledge base, used to assist diagnosis

[0214] def forward(self, optimized_features, confidence_threshold=0.7):

[0215] """

[0216] :param optimized_features: optimized patient features

[0217] :param confidence_threshold: confidence threshold

[0218] :return: Diagnostic report

[0219] """

[0220] # Calculate the probability score for each category

[0221] logits = self.disease_model(optimized_features)

[0222] probabilities = F.softmax(logits, dim=-1)

[0223] diagnosis = probabilities.argmax(dim=-1)

[0224] # Confidence evaluation

[0225] confidence_scores = probabilities.max(dim=-1).values

[0226] report = {"diagnosis": diagnosis.tolist(),"confidence": confidence_scores.tolist()}

[0227] # Auxiliary diagnosis

[0228] if confidence_scores.min().item() <confidence_threshold:

[0229] for i, score in enumerate(confidence_scores):

[0230] if score <confidence_threshold:

[0231] similar_cases = self.search_knowledge_base(optimized_features[i])

[0232] report[f"patient_{i}_similar_cases"] =similar_cases

[0233] return report

[0234] def search_knowledge_base(self, feature):

[0235] """

[0236] Search for similar cases in the knowledge base

[0237] """

[0238] if not self.knowledge_base:

[0239] return "Knowledge base is empty"

[0240] similarities = {case_id: F.cosine_similarity(feature, case_feature, dim=-1).item()

[0241] for case_id, case_feature in self.knowledge_base.items()}

[0242] return sorted(similarities.items(), key=lambda x: -x[1])[:3]

[0243] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A medical diagnosis intelligent decision-making system based on multimodal data fusion, characterized in that: include: Patient information collection unit, used to collect patients' medical imaging data, biometric data and electronic medical record text data; Feature processing unit, used to extract visual feature vectors from medical image data; extract physiological index feature vectors from biometric data; extract semantic feature vectors from electronic medical record text; A modality fusion engine, comprising a deep neural network and a Bayesian reasoning engine, wherein the deep neural network is used to receive the visual feature vector, the physiological index feature vector and the semantic feature vector output by the feature processing unit, and learn the association relationship between the features of different modalities through multi-layer nonlinear transformation to obtain a fused feature representation, wherein the fused feature representation corresponds to a plurality of feature nodes, and each feature node is associated with a specific medical concept; the Bayesian reasoning engine is used to receive the fused feature representation, establish a conditional probability dependency model between feature nodes based on a pre-built expert knowledge graph, and generate a probabilistic feature association representation; An adaptive feature optimizer is used to combine and integrate the fused feature representation and the probabilistic feature association representation to generate a comprehensive feature representation; based on the comprehensive feature representation, the similarity between samples is calculated, and in combination with the characteristics of individual patient data, the weight coefficients of different modal features are adaptively generated using an attention mechanism; the weight coefficients are used to weight the different modal features in the enhanced feature matrix; the weighted different modal features are recombined to generate an optimized feature representation; An intelligent diagnosis analysis unit is used to input the optimized feature representation into a preset disease diagnosis model, calculate the probability score of each diagnosis category, and generate a preliminary diagnosis result; calculate the confidence score of the preliminary diagnosis result, and when the confidence score is lower than a preset threshold, extract the diagnosis information of similar cases from a preset case knowledge base as an auxiliary diagnosis basis; generate a diagnosis report including the main diagnosis category, diagnosis basis and recommended examination items based on the preliminary diagnosis result, confidence score and auxiliary diagnosis basis; Wherein, the deep neural network is specifically used for: According to the following formula 1, the modal feature space is uniformly mapped: ； in, Represents input modality features, including visual feature vectors , physiological index feature vector and semantic feature vector , ; is the modal eigenvector after unified feature space mapping; It is The weight matrix corresponding to the modalities is used to map the modal features to a unified feature space; It is The bias vector corresponding to the mode; is the activation function, including ReLU or tanh; According to the following formula 2, the interaction optimization between modalities is achieved: ； in, To generate interactively optimized feature representations through inter-modality association learning; Represents the query vector, which is used to capture the relationship between the current modality and other modal features; represents the key vector, which represents the reference for modal feature comparison; represents the dimension of the key vector; represents a value vector, which is used to store the actual feature information of each modality; the query vector, key vector and value vector are calculated according to the following formula 3-5: ； ； ； in, It is the visual feature vector after mapping to the unified space; It is the characteristic vector of physiological indicators after being mapped to the unified space; It is the semantic feature vector after being mapped to the unified space; is the weight matrix of the query vector; is the weight matrix of the key vector; is the weight matrix of the value vector; According to the following formula 6, the fusion feature representation is calculated : ； in, is the adaptive weight, calculated according to the following formula 7: ； in, Learnable weight matrices; is a learnable bias term; The Bayesian inference engine is specifically used for: According to the following formula 8, the fusion feature is expressed as Mapped to the preliminary probability space, generating preliminary probability values ​​for each feature node: ； in, Indicates The initial probability of feature nodes; is the weight matrix of the feature map; is the bias term of the feature map; is the Sigmoid function; is the feature vector after multimodal fusion, representing the feature node Data representation; Combined with the expert knowledge graph, the conditional probability dependency relationship between feature nodes is constructed. According to the following formula 9, the feature node The conditional probability : ； in, Nodes defined in the knowledge graph The parent node set of Is a node With its parent node The association weight of For Node The eigenvector of For the parent node The eigenvector of is the feature vector and The Euclidean distance of According to the following formula 10, the global joint probability distribution is obtained : ； in, , represents the set of global feature nodes; , represents the corresponding feature vector set; is the number of feature nodes; Characteristic node The conditional probability of The initial probability is obtained by variational inference method and conditional probability Perform joint optimization to generate optimized posterior probability distribution , the optimization objective function adopts the following formula 11: ； in, refers to the KL divergence; the optimization objective function dynamically adjusts the approximate distribution by minimizing the KL divergence , making it as close as possible to the target distribution ; is the optimized approximate distribution, representing the characteristic node The final probability value of The result of modeling conditional probability; optimized posterior probability Represents the final probabilistic association representation of feature nodes, which is used for new diagnosis decisions. , is the number of feature nodes.

2. The medical diagnosis intelligent decision-making system based on multimodal data fusion according to claim 1 is characterized in that: The patient information collection unit is specifically used for: For medical imaging data, it communicates with imaging devices through interfaces that comply with the DICOM standard, obtains image files and their metadata, including imaging parameters and scanning time, and improves the quality of image data through pre-processing steps such as denoising and format standardization; For biometric data, it interacts with biosensors through wireless or wired connections to obtain patients' physiological indicators in real time, and synchronizes multiple sampled data in combination with timestamp information, while using anomaly detection mechanisms to mark possible inaccurate data; For electronic medical record text data, the patient information collection unit extracts the patient's structured and unstructured text content from the hospital's information system through an API interface based on the HL7 protocol standard, and performs encoding conversion on the unstructured text to ensure format uniformity and data integrity.

3. The medical diagnosis intelligent decision-making system based on multimodal data fusion according to claim 1 is characterized in that: The feature processing unit includes a medical image preprocessing submodule for performing size normalization, denoising and multi-scale feature enhancement on the collected medical image data; The multi-scale feature enhancement extracts local and global features based on images of different resolutions, so that low-resolution features highlight the overall structure of the image and high-resolution features retain detailed information of local lesions, thereby generating accurate visual feature vectors.

4. The medical diagnosis intelligent decision-making system based on multimodal data fusion according to claim 1 is characterized in that: The feature processing unit includes a biometric analysis submodule for receiving the patient's biometric data and extracting short-term change rate and long-term trend features based on time series analysis; The analysis submodule includes an outlier detection function, which identifies and corrects abnormal biometric data by combining patient historical data and population baseline data to ensure the accuracy and consistency of the generated physiological indicator feature vector.

5. The medical diagnosis intelligent decision-making system based on multimodal data fusion according to claim 1 is characterized in that: The feature processing unit includes a text semantic parsing submodule for performing word segmentation and standardization of key medical terms on unstructured text in electronic medical records; The text semantic analysis submodule combines the medical ontology knowledge base to perform semantic enhancement on the extracted text features, ensuring that the generated semantic feature vector can accurately reflect the medical meaning and diagnostic relevance in the text.

6. The medical diagnosis intelligent decision-making system based on multimodal data fusion according to claim 1 is characterized in that: The adaptive feature optimizer combines the similarity between samples to perform feature enhancement, identifies similar cases and extracts relevant features by comparing the multimodal feature representation of the patient with the feature representation of known cases in the database, and enhances the feature matrix of the current patient; The enhanced feature matrix is ​​combined with the attention mechanism to optimize the feature importance between modalities and improve the diagnostic sensitivity of multimodal data fusion.

7. The medical diagnosis intelligent decision-making system based on multimodal data fusion according to claim 1 is characterized in that: The adaptive feature optimizer adopts a multi-round iterative optimization strategy, recalculates the comprehensive feature representation and modal weight distribution in each iteration, suppresses low-confidence modal features, and gradually optimizes the final feature representation; In each iteration, the optimization step size is dynamically adjusted according to the degree of feature convergence and changes in patient characteristics to ensure the accuracy and stability of the final feature representation.

8. The medical diagnosis intelligent decision-making system based on multimodal data fusion according to claim 1 is characterized in that: The intelligent diagnosis and analysis unit is specifically used for: The optimized feature representation is input into the disease diagnosis model, the probability score of each diagnosis category is calculated, and the prediction results of multiple basic classifiers are integrated through ensemble learning methods to generate a preliminary diagnosis result; The confidence of the preliminary diagnosis result is evaluated, and the evaluation combines the discrete degree of the prediction probability, the consistency between the classifiers and the patient's personalized data characteristics to generate a confidence score. When the confidence is lower than a preset threshold, the auxiliary diagnosis mechanism of the case knowledge base is activated; Retrieve the case that is most similar to the current patient's characteristics from the case knowledge base, and extract the diagnostic information, treatment plan and related basis of similar cases; A diagnostic report is generated based on the preliminary diagnosis results, confidence assessment results and similar case information, and the report includes the main diagnostic category, key evidence supporting the diagnosis, comparative analysis with similar cases and recommended further examinations or treatment items.

Images