A method, device, equipment and product for constructing a medical data analysis model

Abstract

The application provides a kind of medical data analysis model construction method, device, equipment and product, related to medical image technology field.The method comprises: based on medical data source, construct medical knowledge graph, including multiple medical entities and the semantic relationship between multiple medical entities;Construct multi-modal data processing module for data processing and feature extraction of multi-modal medical image data, obtain medical image features;Construct data analysis processing module, based on medical knowledge graph, the semantic association of medical image features and the semantic relationship constraint between medical entities, obtain medical data analysis results.Through the analysis of multi-modal medical image data using cross-modal feature alignment and fusion technology, realize the joint analysis of PET, MRI, CT and other multi-source data, solve the problem that the prior art is limited to single modal data and lacks cross-modal feature fusion capability.

Description

Technical Field

[0001] This invention relates to the field of medical imaging technology, and in particular to a method, apparatus, equipment and product for constructing a medical data analysis model. Background Technology

[0002] In the field of intelligent medical image analysis, existing technologies primarily employ a single-modality data processing architecture. They typically rely on traditional machine learning algorithms or shallow neural networks to extract features from single-type image data such as positron emission tomography (PET), computed tomography (CT), and magnetic resonance imaging (MRI). These systems generally use standardized data preprocessing workflows, including basic processing techniques such as noise filtering and image normalization. Feature extraction largely depends on manually designed image processing algorithms, such as edge detection and texture analysis. Regarding knowledge integration, existing solutions typically use static medical knowledge bases, such as the International Classification of Diseases (ICD) 10, with standardized terminology systems for rule-based reasoning. The knowledge update cycle for these bases can be as long as months or even years.

[0003] Existing system architectures mostly adopt a centralized processing model, achieving one-way data transmission with hospital and Picture Archiving and Communication Systems (PACS) through standardized interfaces. In terms of inference mechanisms, they primarily rely on rule-based expert systems or traditional decision tree models to generate suggested solutions through predefined diagnostic paths. Some advanced systems attempt to introduce deep learning models (such as Residual Networks (ResNet) and U-Net) for lesion detection, but their model training is often limited to single-modal data, lacking cross-modal feature fusion capabilities. Regarding inter-system collaboration, existing solutions typically only achieve basic data interface with Hospital Information Systems (HIS) / PACS systems, lacking real-time bidirectional data interaction mechanisms, and security protection is mostly limited to basic transmission encryption measures. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, device, and product for constructing a medical data analysis model, which solves the problem that existing medical image intelligent analysis methods are limited to single-modal data and lack cross-modal feature fusion capabilities.

[0005] To achieve the above objectives, embodiments of the present invention provide a method for constructing a medical data analysis model, wherein the medical data analysis model includes a medical knowledge graph, a multimodal data processing module, and a data analysis and processing module, and the method includes:

[0006] Based on medical data sources, a medical knowledge graph is constructed; the medical knowledge graph includes multiple medical entities and the semantic relationships between these multiple medical entities.

[0007] The multimodal data processing module is constructed; the multimodal data processing module is used to perform data processing and feature extraction on multimodal medical image data to obtain medical image features;

[0008] A data analysis and processing module is constructed; the data analysis and processing module performs semantic association on the medical image features and semantic relationship constraints between medical entities based on the medical knowledge graph, and obtains medical data analysis results.

[0009] Optionally, the method, wherein constructing the medical knowledge graph includes:

[0010] Knowledge extraction is performed on the medical data source to obtain multiple medical characteristics and the semantic relationships between the multiple medical characteristics;

[0011] The medical entities in the medical knowledge graph are constructed based on the multiple medical characteristics, and the semantic relationships between the multiple medical entities in the medical knowledge graph are constructed based on the semantic relationships between the multiple medical characteristics.

[0012] Optionally, in the method, the data analysis and processing module includes:

[0013] A data-driven channel is constructed; the data-driven channel is used to perform semantic association on the medical image features according to the medical knowledge graph and output the probability of a first diagnostic result; wherein, the probability of the first diagnostic result is obtained based on a first medical entity in the medical knowledge graph whose similarity to the medical image features is greater than a preset threshold;

[0014] A knowledge constraint channel is constructed; the knowledge constraint channel is used to impose knowledge constraints on the first medical entity based on the second medical entity in the medical knowledge graph to obtain the probability of a second diagnostic result; wherein, the second medical entity is a neighboring entity of the first medical entity;

[0015] A result output channel is constructed; the result output channel outputs the medical data analysis results based on the probability of the first diagnostic result and the probability of the second diagnostic result.

[0016] To achieve the above objectives, embodiments of the present invention provide a medical data analysis method, wherein the medical data analysis model described above is applied, and the method includes:

[0017] Multimodal medical image data is input into the multimodal data processing module of the medical data analysis model;

[0018] Medical image features are extracted using the multimodal data processing module.

[0019] The data analysis and processing module in the medical data analysis model performs semantic association on the medical image features and semantic relationship constraints between medical entities based on the medical knowledge graph in the medical data analysis model to obtain medical data analysis results; wherein, the medical knowledge graph includes multiple medical entities and semantic relationships between the multiple medical entities.

[0020] Optionally, the method, wherein extracting medical image features through the multimodal data processing module includes:

[0021] Multimodal medical image data is input into the multimodal data processing module to preprocess the medical image data and obtain processed medical data; wherein, when the medical image data is of multiple types, the preprocessing includes alignment processing;

[0022] The processed medical data is used to extract features using a deep learning algorithm to obtain the medical image features.

[0023] Optionally, the method, wherein the medical data analysis and processing module in the medical data analysis model performs semantic association of the medical image features and semantic relationship constraints between medical entities based on the medical knowledge graph in the medical data analysis model to obtain medical data analysis results, includes:

[0024] The data-driven channel performs semantic association on the medical image features based on the medical knowledge graph, and outputs the probability of a first diagnostic result; wherein, the probability of the first diagnostic result is obtained based on a first medical entity in the medical knowledge graph whose similarity to the medical image features is greater than a preset threshold; the data analysis and processing module includes the data-driven channel;

[0025] The probability of a second diagnostic result is obtained by applying knowledge constraints to the first medical entity based on the second medical entity in the medical knowledge graph through a knowledge constraint channel; wherein the second medical entity is a neighboring entity of the first medical entity; the data analysis and processing module includes the knowledge constraint channel;

[0026] The medical data analysis results are output through the result output channel based on the probabilities of the first diagnostic result and the second diagnostic result; the data analysis and processing module includes the result output channel.

[0027] Optionally, the method, wherein semantic association of the medical image features based on the medical knowledge graph via a data-driven channel, and outputting the probability of a first diagnostic result, includes:

[0028] The medical image features are input into the data-driven channel, and a knowledge context is obtained based on the first medical entity and the fusion weight; wherein, the fusion weight is the feature similarity between the medical image features and the first medical entity.

[0029] Based on the knowledge context and the medical image features, obtain the knowledge context vector;

[0030] Based on the knowledge context vector and the medical image features, a knowledge sequence is obtained;

[0031] Autoregressive modeling is performed on the knowledge sequence to obtain the probability of the first diagnostic result.

[0032] Optionally, the method, wherein obtaining the probability of a second diagnostic result by applying knowledge constraints to the first medical entity based on a second medical entity in the medical knowledge graph through a knowledge constraint channel includes:

[0033] The first medical entity is input into the knowledge constraint channel, and the second medical entity is determined based on the first medical entity and the medical knowledge graph.

[0034] The probability of the second diagnostic result is obtained by linearly transforming and aggregating the node features of the second medical entity.

[0035] Optionally, the method, wherein outputting the medical data analysis results via a result output channel based on the probability of the first diagnostic result and the probability of the second diagnostic result, includes:

[0036] The probabilities of the first and second diagnostic results are input into the result output channel. The product of the probability of the first diagnostic result and the first weight, and the product of the probability of the second diagnostic result and the second weight are added together to determine the medical data analysis result, wherein the sum of the first weight and the second weight is 1.

[0037] To achieve the above objectives, embodiments of the present invention provide a model building apparatus, comprising:

[0038] The first construction module is used to construct the medical knowledge graph based on medical data sources; the medical knowledge graph includes multiple medical entities and the semantic relationships between the multiple medical entities;

[0039] The second construction module is used to construct the multimodal data processing module; the multimodal data processing module is used to perform data processing and feature extraction on multimodal medical image data to obtain medical image features;

[0040] The third construction module is used to construct a data analysis and processing module; the data analysis and processing module performs semantic association on the medical image features and semantic relationship constraints between medical entities based on the medical knowledge graph to obtain medical data analysis results.

[0041] To achieve the above objectives, embodiments of the present invention provide a medical data analysis device, wherein the medical data analysis model obtained by the model building method described above is applied, and the device includes:

[0042] The first input module is used to input multimodal medical image data into the multimodal data processing module in the medical data analysis model;

[0043] The first processing module is used to extract medical image features through the multimodal data processing module;

[0044] The first acquisition module is used to obtain medical data analysis results by performing semantic association on the medical image features and semantic relationship constraints between medical entities based on the medical knowledge graph in the medical data analysis model through the data analysis and processing module in the medical data analysis model; wherein, the medical knowledge graph includes multiple medical entities and semantic relationships between the multiple medical entities.

[0045] To achieve the above objectives, embodiments of the present invention provide a medical data analysis device, comprising: a processor, a memory, and a program or instructions stored in the memory and executable on the processor; wherein, when the processor executes the program or instructions, it implements the method for constructing a medical data analysis model as described above, or the medical data analysis method as described above.

[0046] To achieve the above objectives, embodiments of the present invention provide a readable storage medium having a program or instructions stored thereon, wherein the program or instructions, when executed by a processor, implement the method for constructing a medical data analysis model as described above, or the steps in the medical data analysis method as described above.

[0047] To achieve the above objectives, embodiments of the present invention provide a computer program product, comprising computer instructions that, when executed by a processor, implement the method for constructing a medical data analysis model as described above, or the steps of the medical data analysis method as described above.

[0048] The beneficial effects of the above-described technical solution of the present invention are as follows:

[0049] This invention employs a multimodal data processing module to process and extract features from multimodal medical image data, yielding medical image features. A data analysis module then performs semantic association on these features based on a medical knowledge graph constructed from medical data sources, constraining semantic relationships between medical entities to obtain medical data analysis results. By utilizing cross-modal feature alignment and fusion technology to analyze multimodal medical image data, it achieves joint analysis of multi-source data such as PET, MRI, and CT, solving the problem of existing technologies being limited to single-modal data and lacking cross-modal feature fusion capabilities. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the method for constructing a medical data analysis model according to an embodiment of the present invention;

[0051] Figure 2 This is a model architecture diagram of the medical data analysis model construction method described in the embodiments of the present invention;

[0052] Figure 3 This is a schematic diagram of the medical knowledge graph in the method for constructing a medical data analysis model according to an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the multimodal data processing module in the medical data analysis model construction method described in this embodiment of the invention;

[0054] Figure 5 This is a schematic diagram of the medical data analysis method described in an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the apparatus for constructing a medical data analysis model according to an embodiment of the present invention;

[0056] Figure 7 This is a schematic diagram of the medical data analysis device described in an embodiment of the present invention. Detailed Implementation

[0057] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0058] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0059] In various embodiments of the present invention, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0060] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0061] In the embodiments provided by this invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0062] For ease of understanding, the following describes some aspects of the embodiments of the present invention:

[0063] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for constructing a medical data analysis model, wherein the medical data analysis model includes a medical knowledge graph, a multimodal data processing module, and a data analysis processing module, and the method includes:

[0064] Step S10: Based on medical data sources, construct the medical knowledge graph; the medical knowledge graph includes multiple medical entities and the semantic relationships between the multiple medical entities;

[0065] It should be noted that, as Figure 2 As shown, the medical data analysis model includes a medical knowledge graph construction module, used to construct the medical knowledge graph. Figure 3 As shown, the medical knowledge graph stores knowledge through a graph database. The multiple medical entities included in the knowledge graph are nodes in the graph, and the semantic relationships between the multiple medical entities are edges between the nodes.

[0066] Step S20: Construct the multimodal data processing module; the multimodal data processing module is used to process and extract features from multimodal medical image data to obtain medical image features;

[0067] It should be noted that, as Figure 2As shown, the medical data analysis model includes the multimodal data processing module. Figure 4 As shown, the multimodal data processing module extracts image features (i.e., medical image features) from multimodal images (i.e., multimodal medical image data) through image preprocessing (data augmentation, image alignment, noise removal, image enhancement, and standardization), i.e., data processing, and deep learning models (i.e., feature extraction). The main task of the multimodal data processing module is to integrate, align, and process medical data from different sources and modalities, providing rich and accurate input for subsequent inference and analysis. The multimodal data processing module needs to fuse medical image data from multiple sources such as CT, MRI, and ultrasound, and perform necessary data processing and feature extraction. It adopts adaptive alignment algorithms and deep learning models to handle the spatial and temporal differences of different modalities, ensuring that the data can be effectively aligned at the same scale.

[0068] Step S30: Construct a data analysis and processing module; the data analysis and processing module performs semantic association on the medical image features and semantic relationship constraints between medical entities based on the medical knowledge graph, and obtains medical data analysis results;

[0069] It should be noted that, as Figure 2 As shown, the medical data analysis model includes a large-scale model reasoning module (i.e., the data analysis and processing module). The core purpose of the large-scale model reasoning module is to perform intelligent reasoning based on fused multimodal data (i.e., the multimodal medical image data) and the medical knowledge graph, using a pre-trained large-scale language model. Based on the medical knowledge graph, it performs semantic association of the medical image features and semantic relationship constraints between medical entities to obtain medical data analysis results.

[0070] Furthermore, the medical data analysis model also includes an intelligent gateway module. The intelligent gateway module ensures seamless integration between all modules and external medical systems, thereby guaranteeing smooth and secure data flow. The intelligent gateway module communicates with external systems using standardized interface protocols, enabling real-time acquisition of patient image data, electronic medical records, etc., while simultaneously feeding back reasoning results and suggestions to relevant systems or users. The intelligent gateway module not only achieves efficient data exchange but also ensures data security through authentication and encryption technologies, complying with relevant regulations on medical data privacy protection. During data exchange, it guarantees data integrity and accuracy, thus achieving efficient and intelligent support for medical services.

[0071] The medical data analysis model also includes a user interaction module, which provides doctors with a user-friendly interface supporting real-time queries, diagnostic support, and intelligent feedback. The user interaction module displays patient imaging data, historical medical records, and system inference results through a graphical interface, helping doctors better understand and apply the system's recommendations. It provides semantic search functionality, allowing doctors to quickly retrieve relevant medical knowledge by inputting simple symptoms or disease names. The user interaction module can also provide real-time feedback based on doctor input (such as medical record summaries and diagnostic queries), offering intelligent diagnostic support and treatment recommendations. Furthermore, the user interaction module supports tight integration with the inference module, optimizing the large-scale inference model (i.e., the data analysis and processing module) based on doctor feedback (such as adjusting diagnostic results) and forming a closed-loop feedback mechanism to further improve the accuracy and reliability of the inference.

[0072] In this embodiment, a multimodal data processing module processes and extracts features from multimodal medical image data to obtain medical image features. A data analysis and processing module then performs semantic association on these medical image features based on a medical knowledge graph constructed using medical data sources, and imposes semantic relationship constraints between medical entities to obtain medical data analysis results. By employing cross-modal feature alignment and fusion technology to analyze multimodal medical image data, joint analysis of multi-source data such as PET, MRI, and CT is achieved, solving the problem of existing technologies being limited to single-modal data and lacking cross-modal feature fusion capabilities.

[0073] Optionally, the method, wherein step S10 includes:

[0074] Knowledge extraction is performed on the medical data source to obtain multiple medical characteristics and the semantic relationships between the multiple medical characteristics;

[0075] The medical entities in the medical knowledge graph are constructed based on the multiple medical characteristics, and the semantic relationships between the multiple medical entities in the medical knowledge graph are constructed based on the semantic relationships between the multiple medical characteristics.

[0076] In this embodiment, the core function of the medical knowledge graph construction module is to construct a structured and semantic medical knowledge graph by integrating multiple data sources such as medical literature, electronic medical records, and diagnostic standards. In knowledge representation learning, a graph convolutional network is used to achieve deep modeling of entity relationships, as shown in the following formula:

[0077] ;

[0078] in, To add self-connected adjacency matrices, This is the corresponding degree matrix. Represents the node features of the l-th layer. σ is a trainable parameter and the activation function. The above formula supports the hierarchical propagation of node features in the medical knowledge graph, enhancing the semantic relationships between entities such as diseases, symptoms, and drugs. During the data acquisition and preprocessing stages, the medical knowledge graph construction module collects data from multiple data sources (such as PubMed medical literature, electronic medical records, drug databases, etc., i.e., the medical data sources), and ensures data consistency and quality through deduplication, formatting, and standardization. The knowledge extraction process employs advanced natural language processing technologies (such as named entity recognition, relation extraction, etc.) to extract medical entities such as diseases, symptoms, and drugs from the text, and identifies the relationships between them, such as "drug-treatment-disease" or "disease-complications," etc.

[0079] like Figure 3 As shown, the medical knowledge graph construction module supports incremental learning of the knowledge graph using various knowledge representation learning methods, including matrix factorization, translation models, and neural network models. This transforms the knowledge graph into computable "node vectors (embeddings)" and projects them into a low-dimensional space (i.e., the node embedding space). Through an automated update mechanism, the latest medical research findings and clinical data are integrated into the knowledge graph, ensuring that the system's knowledge remains up-to-date. The medical knowledge graph construction module also supports efficient knowledge retrieval and reasoning using graph query languages, further enhancing the system's ability to support medical decision-making.

[0080] Optionally, the method, wherein step S30 includes:

[0081] A data-driven channel is constructed; the data-driven channel is used to perform semantic association on the medical image features according to the medical knowledge graph and output the probability of a first diagnostic result; wherein, the probability of the first diagnostic result is obtained based on a first medical entity in the medical knowledge graph whose similarity to the medical image features is greater than a preset threshold;

[0082] A knowledge constraint channel is constructed; the knowledge constraint channel is used to impose knowledge constraints on the first medical entity based on the second medical entity in the medical knowledge graph to obtain the probability of a second diagnostic result; wherein, the second medical entity is a neighboring entity of the first medical entity;

[0083] A result output channel is constructed; the result output channel outputs the medical data analysis results based on the probability of the first diagnostic result and the probability of the second diagnostic result.

[0084] In this embodiment, the data analysis and processing module includes a data-driven channel, a knowledge-constrained channel, and a result output channel. In the data-driven channel, autoregressive modeling is performed on multimodal features (i.e., medical image features) based on a Transformer decoder to generate preliminary diagnostic hypotheses (i.e., the probability of the first diagnostic result). For example, based on nodule features in a patient's lung CT image, the model can generate a preliminary conclusion of "suspected malignant tumor." In the knowledge-constrained channel, a Graph Neural Network (GNN) is used to logically verify disease relationships in the knowledge graph. For example, when the preliminary diagnosis is "lung cancer," the GNN will verify the presence of bone metastasis features based on the "lung cancer-metastatic site" relationship in the knowledge graph to obtain the probability of the second diagnostic result.

[0085] In the result output channel, the final diagnostic result (i.e., the medical data analysis result) is determined by the weighted joint probability of the dual-channel output, calculated as follows:

[0086] ;

[0087] in, The output of the data-driven channel (i.e., the probability of the first diagnostic result). λ is the output of the knowledge constraint channel (i.e., the probability of the second diagnostic result), and λ is a learnable confidence parameter that dynamically balances statistical data regularity and medical prior knowledge.

[0088] like Figure 5 As shown, to achieve the above objectives, embodiments of the present invention provide a medical data analysis method, wherein the medical data analysis model described above is applied, and the method includes:

[0089] Step A10: Input the multimodal medical image data into the multimodal data processing module of the medical data analysis model;

[0090] It should be noted that, as Figure 4 As shown, multimodal images (i.e., the multimodal medical image data) are input into the multimodal data processing module in the medical data analysis model to provide accurate input for subsequent reasoning, thereby improving the accuracy of diagnosis and treatment recommendations.

[0091] Step A20: Extract medical image features through the multimodal data processing module;

[0092] It should be noted that the multimodal data processing module performs noise removal, image enhancement, and standardization on the image data to improve data quality. Furthermore, deep learning algorithms such as Convolutional Neural Networks (CNNs) are used to extract features from the image data, transforming the images into high-dimensional feature vectors. These extracted features are combined with information from the medical knowledge graph to provide accurate input for subsequent reasoning, thereby improving the accuracy of diagnostic and treatment recommendations.

[0093] Step A30: The data analysis and processing module in the medical data analysis model performs semantic association on the medical image features and semantic relationship constraints between medical entities based on the medical knowledge graph in the medical data analysis model to obtain medical data analysis results; wherein, the medical knowledge graph includes multiple medical entities and semantic relationships between the multiple medical entities;

[0094] It should be noted that the data analysis and processing module is used to semantically associate the medical image features with the medical knowledge graph and constrain the semantic relationships between medical entities to obtain the medical data analysis results.

[0095] In this embodiment, a multimodal data processing module processes and extracts features from multimodal medical image data to obtain medical image features. A data analysis and processing module then performs semantic association on these medical image features based on a medical knowledge graph constructed using medical data sources, and imposes semantic relationship constraints between medical entities to obtain medical data analysis results. By employing cross-modal feature alignment and fusion technology to analyze multimodal medical image data, joint analysis of multi-source data such as PET, MRI, and CT is achieved, solving the problem of existing technologies being limited to single-modal data and lacking cross-modal feature fusion capabilities.

[0096] Optionally, the method, wherein step A20 includes:

[0097] Multimodal medical image data is input into the multimodal data processing module to preprocess the medical image data and obtain processed medical data; wherein, when the medical image data is of multiple types, the preprocessing includes alignment processing;

[0098] The processed medical data is used to extract features using a deep learning algorithm to obtain the medical image features.

[0099] In this embodiment, medical image data from different sources and modalities are integrated, aligned, and processed to obtain processed medical data. The deep learning algorithm is then used to extract features from the processed medical data, obtaining the medical image features and providing rich and accurate input for subsequent reasoning and analysis.

[0100] Optionally, the method, wherein step A30 includes:

[0101] The data-driven channel performs semantic association on the medical image features based on the medical knowledge graph, and outputs the probability of a first diagnostic result; wherein, the probability of the first diagnostic result is obtained based on a first medical entity in the medical knowledge graph whose similarity to the medical image features is greater than a preset threshold; the data analysis and processing module includes the data-driven channel;

[0102] The probability of a second diagnostic result is obtained by applying knowledge constraints to the first medical entity based on the second medical entity in the medical knowledge graph through a knowledge constraint channel; wherein the second medical entity is a neighboring entity of the first medical entity; the data analysis and processing module includes the knowledge constraint channel;

[0103] The medical data analysis results are output through the result output channel based on the probabilities of the first diagnostic result and the second diagnostic result; the data analysis and processing module includes the result output channel.

[0104] In this embodiment, in the data-driven channel, autoregressive modeling is performed on multimodal features (i.e., the medical image features) based on the Transformer decoder to generate preliminary diagnostic hypotheses (i.e., the probability of the first diagnostic result). For example, based on nodule features in a patient's lung CT image, the model can generate a preliminary conclusion of "suspected malignant tumor". In the knowledge-constrained channel, a graph neural network (GNN) is used to logically verify the disease relationships in the knowledge graph. For example, when the preliminary diagnosis is "lung cancer", the GNN will verify whether bone metastasis features exist based on the "lung cancer-metastatic site" relationship in the knowledge graph to obtain the probability of the second diagnostic result.

[0105] In the result output channel, the final diagnostic result (i.e., the medical data analysis result) is determined by the weighted joint probability of the dual-channel output, calculated as follows:

[0106] ;

[0107] in, The output of the data-driven channel (i.e., the probability of the first diagnostic result). λ is the output of the knowledge constraint channel (i.e., the probability of the second diagnostic result), and λ is a learnable confidence parameter that dynamically balances statistical data regularity and medical prior knowledge.

[0108] Optionally, the method, wherein semantic association of the medical image features based on the medical knowledge graph via a data-driven channel, and outputting the probability of a first diagnostic result, includes:

[0109] The medical image features are input into the data-driven channel, and a knowledge context is obtained based on the first medical entity and the fusion weight; wherein, the fusion weight is the feature similarity between the medical image features and the first medical entity.

[0110] Based on the knowledge context and the medical image features, obtain the knowledge context vector;

[0111] Based on the knowledge context vector and the medical image features, a knowledge sequence is obtained;

[0112] Autoregressive modeling is performed on the knowledge sequence to obtain the probability of the first diagnostic result.

[0113] In this embodiment, a cross-modal attention mechanism is used to achieve semantic association between image features and knowledge graph. A fusion weight matrix (i.e., the feature similarity between the medical image features and the first medical entity) is defined, and the formula is as follows:

[0114] ;

[0115] in, Let i be the feature vector of the i-th image. Let represent the embedding of the j-th entity in the knowledge graph, and let ϕ() and ψ() be the nonlinear mapping functions between image features and knowledge embedding, respectively. The output can be interpreted as evidence indicating the set of medical knowledge entities that the current image features are primarily aligned with, i.e., the first medical entities whose feature similarity is greater than the threshold. This mechanism addresses the semantic gap problem in multimodal data by dynamically calculating cross-modal feature similarity to accurately associate lesion regions with disease entities.

[0116] The first is obtained from cross-modal attention. The knowledge context corresponding to each image feature (i.e., the medical image feature) is given by the following formula:

[0117] ;

[0118] in, Indicate image features With knowledge entity embedding The fusion weight.

[0119] Based on this weight, the knowledge entities are weighted and aggregated to obtain the knowledge context vector, as shown in the following formula:

[0120] ;

[0121] in, Can be spliced Linear mapping or gated fusion, This will be used as a conditional input to the data-driven channel to output the diagnostic probability distribution. .

[0122] All joint representations are arranged into a sequence (i.e., the knowledge sequence). , as the conditional context input for the Transformer decoder: ;

[0123] Based on Transformer decoder Autoregressive modeling is performed to obtain the probability of the first diagnostic result. The formula is as follows:

[0124] ;

[0125] in, This represents the historical hidden state (i.e., the knowledge sequence). This is the output layer weight matrix.

[0126] Optionally, the method, wherein obtaining the probability of a second diagnostic result by applying knowledge constraints to the first medical entity based on a second medical entity in the medical knowledge graph through a knowledge constraint channel includes:

[0127] The first medical entity is input into the knowledge constraint channel, and the second medical entity is determined based on the first medical entity and the medical knowledge graph.

[0128] The probability of the second diagnostic result is obtained by linearly transforming and aggregating the node features of the second medical entity.

[0129] In this embodiment, a Generative Neural Network (GNN) is used to logically verify the disease relationships in the knowledge graph. For example, when a preliminary diagnosis of "lung cancer" is made, the GNN will verify the presence of bone metastasis features based on the "lung cancer-metastatic site" relationship in the knowledge graph. The node update equation is as follows:

[0130] ;

[0131] in, Representing nodes in a knowledge graph In the The updated representation vector of the layer, For nodes The set of neighboring nodes connected to the first medical entity (i.e., the second medical entity). For neighboring nodes In the Layer representation vector; and The first The trainable weight matrix and bias vector of the layer are used to perform linear transformation and aggregation on the features of neighboring nodes (i.e., the node features of the second medical entity). This is a non-linear activation function. The knowledge probability distribution is then obtained through Softmax normalization. (i.e., the probability of the second diagnostic result).

[0132] Optionally, the method, wherein outputting the medical data analysis results via a result output channel based on the probability of the first diagnostic result and the probability of the second diagnostic result, includes:

[0133] The probabilities of the first and second diagnostic results are input into the result output channel. The product of the probability of the first diagnostic result and the first weight, and the product of the probability of the second diagnostic result and the second weight are added together to determine the medical data analysis result, wherein the sum of the first weight and the second weight is 1.

[0134] In this embodiment, the final diagnostic result is determined by the weighted joint probability of the dual-channel output:

[0135] ;

[0136] in, The output of the data-driven channel (i.e., the probability of the first diagnostic result). λ is the output of the knowledge constraint channel (i.e., the probability of the second diagnostic result), and λ is a learnable confidence parameter that dynamically balances statistical data regularity and medical prior knowledge.

[0137] It should be noted that the embodiments of the present invention have the following advantages:

[0138] 1. Multimodal medical image fusion and dynamic alignment technology:

[0139] This invention proposes an adaptive cross-modal feature alignment algorithm that supports spatiotemporal alignment and fusion of multi-source heterogeneous medical image data such as CT, MRI, and ultrasound, overcoming the limitations of single-modal analysis and improving lesion localization accuracy to over 95%. A deep differentiable feature selection network is designed, and a dynamic gating mechanism enhances the capture capability of sub-millimeter-level micro-lesions, significantly improving the sensitivity of early lung cancer detection to 92%. The system proposed in this application can effectively integrate and process various types of medical data, including multimodal medical image data such as CT, MRI, and ultrasound, as well as non-image data such as electronic medical records. Through the application of the adaptive alignment algorithm and deep learning model, not only can accurate data alignment and fusion be achieved, but also the comprehensive analytical capability of the inference model can be improved based on the complementary characteristics of multimodal data, providing more accurate and comprehensive diagnostic support.

[0140] 2. Dynamic Medical Knowledge Graph Construction and Incremental Update Mechanism:

[0141] This invention, based on Graph Convolutional Networks (GNNs) and Natural Language Processing (NLP) technologies, achieves automated extraction and structured storage of medical entities (diseases, symptoms, drugs) and their relationships, supporting dynamic updates of the knowledge graph. An incremental learning mechanism is introduced to integrate the latest data from medical literature, electronic medical records, and clinical guidelines in real time, ensuring the timeliness of the knowledge graph and providing accurate medical background knowledge for reasoning. The technical solution of this invention innovatively introduces a dynamic update mechanism for the medical knowledge graph, which can automatically extract medical entities and relationships based on the latest medical literature, electronic medical records, and diagnostic standards through NLP technology, and continuously update the graph content. This dynamic update mechanism ensures that the knowledge graph reflects the latest developments in the medical field, enabling the reasoning module to utilize the latest medical knowledge in each reasoning iteration, improving the accuracy and timeliness of the reasoning results.

[0142] 3. Deep collaborative reasoning architecture of large models and knowledge graphs:

[0143] This invention proposes a dual-channel reasoning framework. The data-driven channel uses a Transformer decoder to perform autoregressive modeling on multimodal features (images, clinical text) to generate preliminary diagnostic hypotheses. The knowledge-constrained channel uses a Generative Neural Network (GNN) to verify the logical consistency between the diagnostic hypotheses and the knowledge graph (e.g., disease-complication relationships), ensuring that the reasoning results conform to prior medical knowledge. A cross-modal attention mechanism is designed to dynamically associate image features with knowledge graph entities, addressing the multimodal semantic gap and enhancing the interpretability of diagnostic results. This invention deeply integrates large models (such as pre-trained Transformer models) with medical knowledge graphs. By fusing image features with structured knowledge in the knowledge graph, the reasoning process is not only based on image data but also incorporates medical background knowledge and disease knowledge for more comprehensive and personalized reasoning. It enables refined reasoning based on the patient's specific medical history, symptoms, and medical image features, providing doctors with personalized diagnostic and treatment plans.

[0144] 4. Seamless integration of the smart gateway with external systems:

[0145] The intelligent gateway module in this embodiment of the invention can seamlessly interface with hospital information systems (HIS), picture archiving systems (PACS), and other medical platforms, supporting real-time data interaction through standardized interfaces (such as HL7, DICOM, etc.). This not only ensures data flow and consistency but also guarantees efficient system operation and data security, complies with regulatory requirements in the medical industry, and avoids delays and data loss during data exchange.

[0146] 5. Real-time feedback and closed-loop optimization mechanism:

[0147] This invention, through the introduction of a real-time feedback mechanism and a closed-loop optimization strategy, enables the inference model to be continuously adjusted based on physician feedback. This closed-loop optimization mechanism improves the model's accuracy and personalization level. As medical data accumulates and physician experience is gathered, the inference capability continuously strengthens, thereby achieving higher-quality intelligent diagnostic support.

[0148] like Figure 6 As shown, to achieve the above objectives, embodiments of the present invention provide a model building apparatus, comprising:

[0149] The first construction module 601 is used to construct the medical knowledge graph based on medical data sources; the medical knowledge graph includes multiple medical entities and semantic relationships between the multiple medical entities;

[0150] The second construction module 602 is used to construct the multimodal data processing module; the multimodal data processing module is used to perform data processing and feature extraction on multimodal medical image data to obtain medical image features;

[0151] The third construction module 603 is used to construct a data analysis and processing module; the data analysis and processing module performs semantic association on the medical image features and semantic relationship constraints between medical entities based on the medical knowledge graph, and obtains medical data analysis results.

[0152] Optionally, in the apparatus, the first building module 601 includes:

[0153] The first acquisition unit is used to extract knowledge from the medical data source and acquire multiple medical characteristics and semantic relationships between the multiple medical characteristics.

[0154] The first construction unit is used to construct medical entities in the medical knowledge graph based on the multiple medical characteristics, and to construct semantic relationships between the multiple medical entities in the medical knowledge graph based on the semantic relationships between the multiple medical characteristics.

[0155] Optionally, in the aforementioned apparatus, the third building module 603 includes:

[0156] The second construction unit is used to construct a data-driven channel; the data-driven channel is used to perform semantic association on the medical image features according to the medical knowledge graph and output the probability of a first diagnostic result; wherein, the probability of the first diagnostic result is obtained based on a first medical entity in the medical knowledge graph whose similarity to the medical image features is greater than a preset threshold;

[0157] The third construction unit is used to construct a knowledge constraint channel; the knowledge constraint channel is used to impose knowledge constraints on the first medical entity based on the second medical entity in the medical knowledge graph to obtain the probability of a second diagnostic result; wherein, the second medical entity is a neighboring entity of the first medical entity;

[0158] The fourth construction unit is used to construct a result output channel; the result output channel outputs the medical data analysis results based on the probability of the first diagnostic result and the probability of the second diagnostic result.

[0159] like Figure 7 As shown, to achieve the above objectives, embodiments of the present invention provide a medical data analysis device, wherein the medical data analysis model obtained by the model construction method described above is applied, and the device includes:

[0160] The first input module 701 is used to input multimodal medical image data into the multimodal data processing module in the medical data analysis model;

[0161] The first processing module 702 is used to extract medical image features through the multimodal data processing module;

[0162] The first acquisition module 703 is used to acquire medical data analysis results by performing semantic association on the medical image features and semantic relationship constraints between medical entities based on the medical knowledge graph in the medical data analysis model through the data analysis and processing module in the medical data analysis model; wherein, the medical knowledge graph includes multiple medical entities and semantic relationships between the multiple medical entities.

[0163] Optionally, in the aforementioned apparatus, the first processing module 702 includes:

[0164] The second acquisition unit is used to input multimodal medical image data into the multimodal data processing module, preprocess the medical image data, and acquire processed medical data; wherein, when the medical image data is of multiple types, the preprocessing includes alignment processing;

[0165] The third acquisition unit is used to extract features from the processed medical data using a deep learning algorithm to obtain the medical image features.

[0166] Optionally, in the aforementioned apparatus, the first acquisition module 703 includes:

[0167] The first output unit is used to perform semantic association of the medical image features based on the medical knowledge graph through a data-driven channel, and output the probability of a first diagnostic result; wherein, the probability of the first diagnostic result is obtained based on a first medical entity in the medical knowledge graph whose similarity to the medical image features is greater than a preset threshold; the data analysis and processing module includes the data-driven channel;

[0168] The fourth acquisition unit is used to apply knowledge constraints to the first medical entity based on the second medical entity in the medical knowledge graph through a knowledge constraint channel, and to obtain the probability of a second diagnostic result; wherein the second medical entity is a neighboring entity of the first medical entity; the data analysis and processing module includes the knowledge constraint channel;

[0169] The second output unit is used to output the medical data analysis results based on the probability of the first diagnostic result and the probability of the second diagnostic result through the result output channel; the data analysis and processing module includes the result output channel.

[0170] Optionally, in the aforementioned apparatus, the first output unit includes:

[0171] The first acquisition component is used to input the medical image features into the data-driven channel and acquire the knowledge context based on the first medical entity and the fusion weight; wherein, the fusion weight is the feature similarity between the medical image features and the first medical entity.

[0172] The second acquisition component is used to acquire a knowledge context vector based on the knowledge context and the medical image features;

[0173] The third acquisition component is used to acquire a knowledge sequence based on the knowledge context vector and the medical image features;

[0174] The fourth acquisition component is used to perform autoregressive modeling on the knowledge sequence to obtain the probability of the first diagnostic result.

[0175] Optionally, in the aforementioned apparatus, the fourth acquiring unit comprises:

[0176] A first determining component is used to input the first medical entity into the knowledge constraint channel and determine the second medical entity based on the first medical entity and the medical knowledge graph.

[0177] The fifth acquisition component is used to obtain the probability of the second diagnostic result by performing a linear transformation and aggregation on the node features of the second medical entity.

[0178] Optionally, in the aforementioned apparatus, the second output unit includes:

[0179] The second determining component is used to input the probability of the first diagnostic result and the probability of the second diagnostic result into the result output channel, and to add the product of the probability of the first diagnostic result and the first weight, and the product of the probability of the second diagnostic result and the second weight, to determine the medical data analysis result, wherein the sum of the first weight and the second weight is 1.

[0180] It should be noted that the apparatus provided in this embodiment of the invention can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0181] To achieve the above objectives, embodiments of the present invention provide a medical data analysis device, comprising: a processor, a memory, and a program or instructions stored in the memory and executable on the processor; wherein, when the processor executes the program or instructions, it implements the method for constructing a medical data analysis model as described above, or the medical data analysis method as described above.

[0182] To achieve the above objectives, embodiments of the present invention provide a readable storage medium having a program or instructions stored thereon, wherein the program or instructions, when executed by a processor, implement the method for constructing a medical data analysis model as described above, or the steps in the medical data analysis method as described above.

[0183] To achieve the above objectives, embodiments of the present invention provide a computer program product, comprising computer instructions that, when executed by a processor, implement the method for constructing a medical data analysis model as described above, or the steps of the medical data analysis method as described above.

[0184] In this embodiment of the invention, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different locations, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0185] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0186] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional Very Large Scale Integration (VLSI) circuits or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0187] The exemplary embodiments described above are illustrated with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of the invention. Therefore, the invention should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make the invention complete and convey its scope to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless explicitly stated otherwise, the singular forms “a,” “an,” and “the” are intended to include these multiple forms as well. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, elements, and / or groups thereof. Unless otherwise indicated, a range of values ​​is stated to include the upper and lower limits of the range and any subranges therebetween.

[0188] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for constructing a medical data analysis model, characterized in that, The medical data analysis model includes a medical knowledge graph, a multimodal data processing module, and a data analysis and processing module. The method includes: Based on medical data sources, a medical knowledge graph is constructed; the medical knowledge graph includes multiple medical entities and the semantic relationships between these multiple medical entities. The multimodal data processing module is constructed; the multimodal data processing module is used to perform data processing and feature extraction on multimodal medical image data to obtain medical image features; A data analysis and processing module is constructed; the data analysis and processing module performs semantic association on the medical image features and semantic relationship constraints between medical entities based on the medical knowledge graph, and obtains medical data analysis results.

2. The method according to claim 1, characterized in that, The construction of the medical knowledge graph includes: Knowledge extraction is performed on the medical data source to obtain multiple medical characteristics and the semantic relationships between the multiple medical characteristics; The medical entities in the medical knowledge graph are constructed based on the multiple medical characteristics, and the semantic relationships between the multiple medical entities in the medical knowledge graph are constructed based on the semantic relationships between the multiple medical characteristics.

3. The method according to claim 1, characterized in that, The data analysis and processing module includes: A data-driven channel is constructed; the data-driven channel is used to perform semantic association on the medical image features according to the medical knowledge graph and output the probability of a first diagnostic result; wherein, the probability of the first diagnostic result is obtained based on a first medical entity in the medical knowledge graph whose similarity to the medical image features is greater than a preset threshold; A knowledge constraint channel is constructed; the knowledge constraint channel is used to impose knowledge constraints on the first medical entity based on the second medical entity in the medical knowledge graph to obtain the probability of a second diagnostic result; wherein, the second medical entity is a neighboring entity of the first medical entity; A result output channel is constructed; the result output channel outputs the medical data analysis results based on the probability of the first diagnostic result and the probability of the second diagnostic result.

4. A medical data analysis method, characterized in that, The method, employing the medical data analysis model according to any one of claims 1 to 3, comprises: Multimodal medical image data is input into the multimodal data processing module of the medical data analysis model; Medical image features are extracted using the multimodal data processing module. The data analysis and processing module in the medical data analysis model performs semantic association on the medical image features and semantic relationship constraints between medical entities based on the medical knowledge graph in the medical data analysis model to obtain medical data analysis results; wherein, the medical knowledge graph includes multiple medical entities and semantic relationships between the multiple medical entities.

5. The method according to claim 4, characterized in that, Medical image features are extracted through the multimodal data processing module, including: Multimodal medical image data is input into the multimodal data processing module to preprocess the medical image data and obtain processed medical data; wherein, when the medical image data is of multiple types, the preprocessing includes alignment processing; The processed medical data is used to extract features using a deep learning algorithm to obtain the medical image features.

6. The method according to claim 4, characterized in that, The data analysis and processing module in the medical data analysis model performs semantic association on the medical image features and semantic relationship constraints between medical entities based on the medical knowledge graph in the medical data analysis model, and obtains medical data analysis results, including: The data-driven channel performs semantic association on the medical image features based on the medical knowledge graph, and outputs the probability of a first diagnostic result; wherein, the probability of the first diagnostic result is obtained based on a first medical entity in the medical knowledge graph whose similarity to the medical image features is greater than a preset threshold; the data analysis and processing module includes the data-driven channel; The probability of a second diagnostic result is obtained by applying knowledge constraints to the first medical entity based on the second medical entity in the medical knowledge graph through a knowledge constraint channel; wherein the second medical entity is a neighboring entity of the first medical entity; the data analysis and processing module includes the knowledge constraint channel; The medical data analysis results are output through the result output channel based on the probabilities of the first diagnostic result and the second diagnostic result; the data analysis and processing module includes the result output channel.

7. The method according to claim 6, characterized in that, The data-driven channel performs semantic association on the medical image features based on the medical knowledge graph, and outputs the probability of the first diagnostic result, including: The medical image features are input into the data-driven channel, and a knowledge context is obtained based on the first medical entity and the fusion weight; wherein, the fusion weight is the feature similarity between the medical image features and the first medical entity. Based on the knowledge context and the medical image features, obtain the knowledge context vector; Based on the knowledge context vector and the medical image features, a knowledge sequence is obtained; Autoregressive modeling is performed on the knowledge sequence to obtain the probability of the first diagnostic result.

8. The method according to claim 6, characterized in that, By applying knowledge constraints to the first medical entity based on the second medical entity in the medical knowledge graph through a knowledge constraint channel, the probability of obtaining a second diagnostic result is obtained, including: The first medical entity is input into the knowledge constraint channel, and the second medical entity is determined based on the first medical entity and the medical knowledge graph. The probability of the second diagnostic result is obtained by linearly transforming and aggregating the node features of the second medical entity.

9. The method according to claim 6, characterized in that, The medical data analysis results are output through the result output channel based on the probabilities of the first diagnostic result and the second diagnostic result, including: The probabilities of the first and second diagnostic results are input into the result output channel. The product of the probability of the first diagnostic result and the first weight, and the product of the probability of the second diagnostic result and the second weight are added together to determine the medical data analysis result, wherein the sum of the first weight and the second weight is 1.

10. A model building apparatus, characterized in that, include: The first construction module is used to construct the medical knowledge graph based on medical data sources; The medical knowledge graph includes multiple medical entities and the semantic relationships between these multiple medical entities; The second construction module is used to construct the multimodal data processing module; the multimodal data processing module is used to perform data processing and feature extraction on multimodal medical image data to obtain medical image features; The third construction module is used to construct a data analysis and processing module; the data analysis and processing module performs semantic association on the medical image features and semantic relationship constraints between medical entities based on the medical knowledge graph to obtain medical data analysis results.

11. A medical data analysis device, characterized in that, The medical data analysis model obtained by applying the model construction method according to any one of claims 1 to 3, the apparatus comprising: The first input module is used to input multimodal medical image data into the multimodal data processing module in the medical data analysis model; The first processing module is used to extract medical image features through the multimodal data processing module; The first acquisition module is used to obtain medical data analysis results by performing semantic association on the medical image features and semantic relationship constraints between medical entities based on the medical knowledge graph in the medical data analysis model through the data analysis and processing module in the medical data analysis model; wherein, the medical knowledge graph includes multiple medical entities and semantic relationships between the multiple medical entities.

12. A medical data analysis device, comprising: A processor, a memory, and a program or instructions stored in the memory and executable on the processor; characterized in that, when the processor executes the program or instructions, it implements the method for constructing a medical data analysis model as described in any one of claims 1-3, or the medical data analysis method as described in any one of claims 4-9.

13. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the method for constructing a medical data analysis model as described in any one of claims 1-3, or the steps in the medical data analysis method as described in any one of claims 4-9.

14. A computer program product, characterized in that, It includes computer instructions, which, when executed by a processor, implement the method for constructing a medical data analysis model as described in any one of claims 1-3, or the steps of the medical data analysis method as described in any one of claims 4-9.

Images