Medical image question and answer method and computer equipment

Abstract

The invention relates to the technical field of large models, and discloses a medical image question and answer method and computer equipment, and the method comprises the steps: firstly obtaining target multi-modal medical data, and inputting the target multi-modal medical data into a medical multi-modal large model; then performing feature extraction on the target multi-modal medical data through a feature extraction module to obtain image-text fusion feature data, performing entity recognition through an entity recognition module to obtain target medical entity data, and performing entity relationship extraction through a relationship extraction module to obtain target medical relationship data; constructing a multi-modal knowledge graph by using the extracted target medical entity data and the target medical relationship data; and finally, under the condition of receiving the multi-modal medical problem data, generating image analysis report data by using the model reasoning module and the multi-modal knowledge graph. Through a mode of combining the medical multi-modal large model and the multi-modal knowledge graph, highly-automatic medical image question answering is realized, so that the medical image question answering efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of large model technology, and in particular to a medical image question-and-answer method and computer equipment. Background Art

[0002] Medical imaging is the acquisition of information about the internal structure and function of the human body through various imaging technologies (such as X-ray, CT, MRI, ultrasound, etc.) for disease diagnosis, treatment planning and monitoring. Its importance lies in its ability to help doctors accurately identify lesions, assess conditions, predict disease progression, and provide key information in a non-invasive manner, greatly improving the efficiency and accuracy of diagnosis and treatment.

[0003] In the related art, medical image question answering relies on doctors to manually analyze images and provide diagnoses based on their experience and knowledge. Doctors need to spend a lot of time interpreting images and giving diagnostic opinions. Therefore, a new medical image question answering method is needed. Summary of the invention

[0004] The present application provides a medical imaging question and answer method, a computer device and a storage medium, which solve the technical problem in the related art that the efficiency of medical imaging question and answer needs to be improved, and achieves the technical effect of highly automated medical imaging question and answer.

[0005] In order to achieve the above objectives, the main technical solutions adopted in this application include:

[0006] In a first aspect, an embodiment of the present application provides a medical image question-answering method, the method comprising:

[0007] Acquire target multimodal medical data; wherein the target multimodal medical data includes medical image data and medical text data;

[0008] Inputting the target multimodal medical data into a medical multimodal macromodel; wherein the medical multimodal macromodel includes a feature extraction module, an entity recognition module, a relationship extraction module and a model reasoning module;

[0009] Performing feature extraction on the medical image data and the medical text data by the feature extraction module to obtain image-text fusion feature data;

[0010] Through the entity recognition module, entity recognition is performed based on the image-text fusion feature data to obtain target medical entity data;

[0011] Through the relationship extraction module, entity relationship extraction is performed based on the target medical entity data and the image-text fusion feature data to obtain target medical relationship data; wherein the target medical entity data and the target medical relationship data are used to construct a multimodal knowledge graph; the multimodal knowledge graph includes relationship data between disease entities and medical imaging entities;

[0012] When multimodal medical problem data is received, the model reasoning module and the multimodal knowledge graph are used to generate image analysis report data corresponding to the multimodal medical problem data.

[0013] Optionally, the entity recognition module includes a coding layer and a sequence annotation layer connected in sequence; the entity recognition module performs entity recognition based on the image-text fusion feature data to obtain target medical entity data, including:

[0014] Encoding the image-text fusion feature data through the encoding layer to obtain encoded feature data;

[0015] Through the sequence annotation layer, entity recognition is performed based on the coded feature data to obtain the target medical entity data.

[0016] Optionally, the entity recognition module further includes an attention layer, and the encoding layer, the attention layer and the sequence labeling layer are connected in sequence; the method further includes:

[0017] Performing semantic enhancement on the encoded feature data through the attention layer to obtain enhanced semantic feature data;

[0018] Accordingly, entity recognition is performed based on the coded feature data through the sequence annotation layer to obtain the target medical entity data, including:

[0019] The enhanced semantic feature data is subjected to entity recognition through the sequence annotation layer to obtain the target medical entity data.

[0020] Optionally, the entity relationship extraction is performed based on the target medical entity data and the image-text fusion feature data by the relationship extraction module to obtain the target medical relationship data, including:

[0021] Through the relationship extraction module, relationship identification is performed based on the target medical entity data, the image-text fusion feature data and the enhanced semantic feature data to obtain the target medical relationship data.

[0022] Optionally, the acquiring target multimodal medical data includes:

[0023] Acquiring initial multimodal medical data; wherein the initial multimodal medical data includes initial medical image data and initial medical text data;

[0024] Performing structured processing on the initial medical text data to obtain structured text data;

[0025] Performing standard processing on the initial medical image data to obtain standard image data;

[0026] The target multimodal medical data is constructed based on the structured text data and the standard image data.

[0027] Optionally, when receiving multimodal medical problem data, generating image analysis report data corresponding to the multimodal medical problem data using the model reasoning module and the multimodal knowledge graph includes:

[0028] Upon receiving the multimodal medical problem data, searching the multimodal knowledge graph based on the multimodal medical problem data to obtain target subgraph data;

[0029] The target subgraph data and the multimodal medical problem data are input into the model reasoning module for semantic enhancement reasoning to obtain the image analysis report data corresponding to the multimodal medical problem data.

[0030] Optionally, the step of searching the multimodal knowledge graph based on the multimodal medical problem data to obtain target subgraph data includes:

[0031] Retrieving a target node related to the multimodal medical problem data in the multimodal knowledge graph;

[0032] Retrieving edges and connection nodes directly related to the target node in the multimodal knowledge graph to obtain local subgraph data;

[0033] The local sub-graph data is weighted to obtain the weighted target sub-graph data.

[0034] Optionally, the method further comprises:

[0035] By using the medical multimodal large model, entity recognition and relationship extraction are performed based on the image analysis report data to obtain medical entity data and medical relationship data in the image analysis report data;

[0036] The multimodal knowledge graph is updated based on the medical entity data and medical relationship data in the image analysis report data.

[0037] Optionally, the model reasoning module is obtained by:

[0038] Loading a pre-trained model and freezing parameters of a text layer in the pre-trained model;

[0039] The pre-trained model is fine-tuned using a multimodal medical data sample set, and the parameters of the visual part layer in the pre-trained model are updated to obtain the model reasoning module.

[0040] In a second aspect, an embodiment of the present application provides a computer device, including:

[0041] A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method described in any of the above embodiments by executing the computer instructions.

[0042] In the embodiment of the present application, firstly, the target multimodal medical data is obtained, and the target multimodal medical data is input into the medical multimodal large model; then, the target multimodal medical data is feature extracted by the feature extraction module to obtain the image-text fusion feature data, the entity recognition module is used to perform entity recognition to obtain the target medical entity data, the entity relationship extraction module is used to perform entity relationship extraction to obtain the target medical relationship data, and the extracted target medical entity data and target medical relationship data are used to construct a multimodal knowledge graph; finally, when the multimodal medical question data is received, the model reasoning module and the multimodal knowledge graph are used to generate the image analysis report data. By combining the medical multimodal large model with the multimodal knowledge graph, a highly automated medical image question and answer is achieved, thereby improving the efficiency of medical image question and answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1a A flowchart of a medical imaging question-and-answer method provided in an embodiment of the present application;

[0045] Figure 1b A schematic diagram of the entity recognition and relationship extraction process provided in an embodiment of the present application;

[0046] Figure 1c A schematic diagram of the structure of a large medical multimodal model provided in an embodiment of the present application;

[0047] Figure 1dA schematic diagram of a multimodal knowledge graph provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the entity recognition and relationship extraction process provided in an embodiment of the present application;

[0049] Figure 3 A flowchart of a medical imaging question-and-answer method provided in an embodiment of the present application;

[0050] Figure 4a A flowchart of a medical imaging question-and-answer method provided in an embodiment of the present application;

[0051] Figure 4b A schematic diagram of a medical imaging question-and-answer process provided in an embodiment of the present application;

[0052] Figure 4c A schematic diagram of the module division of the medical multimodal large model provided in the embodiment of the present application;

[0053] Figure 5a A flowchart of a medical imaging question-and-answer method provided in an embodiment of the present application;

[0054] Figure 5b A schematic diagram of a medical imaging question-and-answer process provided in an embodiment of the present application;

[0055] Figure 6a A flowchart of a medical imaging question-and-answer method provided in an embodiment of the present application;

[0056] Figure 6b A schematic diagram of the updating process of the multimodal knowledge graph provided in the embodiment of the present application;

[0057] Figure 7 A flowchart of a medical imaging question-and-answer method provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0060] In the related art, medical image question answering relies on doctors to manually analyze images, or uses machine learning methods to rely on experts to manually extract image features, or uses deep learning models to rely on manually labeled data. The medical image question answering methods in the related art are highly dependent on manual intervention and are time-consuming. Therefore, the efficiency of the medical image question answering methods in the related art needs to be improved.

[0061] Based on this, this application proposes a new medical image question-answering method, which first obtains target multimodal medical data and inputs the target multimodal medical data into a medical multimodal large model; then, the target multimodal medical data is feature extracted by a feature extraction module to obtain graphic-text fusion feature data, entity recognition is performed by an entity recognition module to obtain target medical entity data, entity relationship extraction is performed by a relationship extraction module to obtain target medical relationship data, and a multimodal knowledge graph is constructed using the extracted target medical entity data and target medical relationship data; finally, when receiving multimodal medical question data, image analysis report data is generated using a model reasoning module and a multimodal knowledge graph. By combining a medical multimodal large model with a multimodal knowledge graph, highly automated medical image question-answering is achieved, thereby improving the efficiency of medical image question-answering.

[0062] For multimodal medical problem data, the medical multimodal big model can perform semantically enhanced reasoning based on the existing multimodal knowledge graph to generate more accurate image analysis report data. In addition, the model can also use the image analysis report data to further extract entities and relationships, and automatically update the multimodal knowledge graph, thereby continuously improving and enriching the content of the knowledge graph.

[0063] The medical image question and answer method in the embodiment of the present application significantly reduces manual intervention through high automation, thereby improving the efficiency and accuracy of medical image question and answer.

[0064] According to an embodiment of the present application, an embodiment of a medical image question and answer method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0065] For the steps of the medical image question-answering method provided in this embodiment of the present application, please refer to Figure 1a , the specific steps include:

[0066] S110. Acquire target multimodal medical data.

[0067] The target multimodal medical data includes medical image data and medical text data, which are obtained by preprocessing the initial multimodal medical data. Medical image data can be X-rays, CT, MRI scan images, etc. Medical text data can be medical reports, case descriptions, surgical records, etc.

[0068] Specifically, the acquisition of initial multimodal medical data, on the one hand, can obtain clinical data such as patients' pathological information, disease descriptions and diagnostic results through channels such as clinical data and electronic medical records; on the other hand, large-scale medical data can be obtained through public medical databases and data sets, such as MIMIC-III, TCIA, etc.; in addition, relevant research papers and medical imaging data can be obtained through literature retrieval and medical imaging databases as a supplement to clinical data. It should be noted that these initial medical data need to be preprocessed before they can become target multimodal medical data. Exemplarily, the preprocessing of medical imaging data can include steps such as size normalization, denoising, and enhancement. The preprocessing of medical text data can include steps such as data cleaning and supplementing missing values.

[0069] S120, inputting the target multimodal medical data into the medical multimodal large model.

[0070] See also Figure 1c The medical multimodal big model includes a feature extraction module, an entity recognition module, a relationship extraction module and a model reasoning module. Among them, the entity recognition module, the relationship extraction module and the model reasoning module receive the output of the feature extraction module respectively, and the relationship extraction module also receives the output of the entity recognition module. The medical multimodal big model can be a pre-trained big model fine-tuned using multimodal medical data (such as the InternVL2 big model, the LLAMA3.2 big model).

[0071] In other embodiments, the target multimodal medical data may first pass through a coreference resolution module before entering the medical multimodal macromodel. This is because there may be coreference in medical texts, such as different names for patients (such as "patients" or "patients") or abbreviations and full names of medical terms. By defining entities and using the macromodel in combination with contextual semantic information, the corresponding pronouns can be accurately linked to the entities they refer to, thereby identifying and processing expressions that refer to specific entities.

[0072] S130. Perform feature extraction on the medical image data and the medical text data through a feature extraction module to obtain image-text fusion feature data.

[0073] Specifically, feature extraction is performed on medical image data through a feature extraction module to obtain medical image features, and feature extraction is performed on medical text data through a feature extraction module to obtain medical text features, and image-text fusion feature data is determined based on the extracted medical image features and medical text features.

[0074] S140. Perform entity recognition based on the image-text fusion feature data through the entity recognition module to obtain target medical entity data.

[0075] The entity recognition module includes a sequentially connected encoding layer and a sequence annotation layer, see Figure 1b Specifically, the image-text fusion feature data is encoded through the encoding layer to obtain the encoded feature data. Through the sequence annotation layer, entity recognition is performed based on the encoded feature data to obtain the target medical entity data.

[0076] In this embodiment, the image feature vector is obtained by encoding through the visual encoder of the medical multimodal large model. These features usually include key information in the medical image data (such as lesion area, organ structure, etc.). The text feature vector representing the text semantics is obtained by encoding through the text encoder of the medical multimodal large model. These features usually include information such as disease name, symptom description, etc. Then the image feature vector and the text feature vector are spliced ​​or weighted averaged to obtain the encoded feature data.

[0077] In this embodiment, the sequence annotation layer is used to improve the accuracy of entity classification. The target medical entity data includes entities such as personal information, disease names, and disease examinations. After obtaining the encoded feature data, a linear classification layer can be added to the embedded features at each position as a sequence annotation layer to predict entity labels. Finally, the output layer can generate the target medical entity data based on the entity label information output by the sequence annotation layer, thereby providing a data basis for building a multimodal knowledge graph.

[0078] S150. Through the relationship extraction module, entity relationship extraction is performed based on the target medical entity data and the image-text fusion feature data to obtain the target medical relationship data.

[0079] The target medical relationship data may be the relationship between the target medical entity data. A multimodal knowledge graph may be constructed using the extracted target medical entity data and the target medical relationship data.

[0080] Specifically, we first need to define the possible relationship types between different entities, such as treatment, diagnosis, and cause, in order to extract relationships. Then, refer to Figure 1b,Graph Neural Network (GNN) can be integrated into the medical multimodal model to form a relationship extraction module. This module can extract entity relationships based on the defined relationships and the extracted target medical entity data on the basis of multimodal features to obtain the target medical relationship data. Finally, through GNN, the representation vector of each entity is iteratively updated, the characteristics of the entity itself and the information of its related entities are integrated, so as to enrich the representation information of the entity.

[0081] In this embodiment, the identified target medical entity data and the extracted target medical relationship data can be integrated to form a structured knowledge network and obtain a multimodal knowledge graph. Figure 1d . It should be noted that after constructing the multimodal knowledge graph, the redundant and repeated information in the multimodal knowledge graph can be removed through knowledge fusion technology, and it can be optimized and refined. At this point, the construction of the multimodal knowledge graph is completed in a highly automated way, providing an accurate data foundation for subsequent medical imaging question and answer. Therefore, under the premise of the existing multimodal knowledge graph and medical multimodal large model, the multimodal knowledge graph and the medical multimodal large model can be combined for medical imaging question and answer.

[0082] S160. When receiving multimodal medical problem data, use the model reasoning module and the multimodal knowledge graph to generate image analysis report data corresponding to the multimodal medical problem data.

[0083] Among them, multimodal medical problem data can include medical imaging data (such as X-rays, CT scans, magnetic resonance imaging, and ultrasound images, etc.) and medical text data (such as electronic medical records and clinical data, etc.).

[0084] Specifically, when multimodal medical problem data is received, the medical image data in the multimodal medical problem data can first be feature extracted through the visual encoder of the medical multimodal large model to obtain an image feature vector. At the same time, the medical text data in the multimodal medical problem data can be encoded into a text feature vector through the text encoder of the medical multimodal large model; then the image feature vector and the text feature vector are concatenated or weighted averaged to obtain the problem image-text fusion feature data; finally, the model reasoning module combines the multimodal knowledge graph to understand and reason the problem fusion feature data, and outputs the image analysis report data corresponding to the multimodal medical problem data.

[0085] In the above embodiment, the target multimodal medical data is first obtained and input into the medical multimodal large model; then, the target multimodal medical data is feature extracted by the feature extraction module to obtain the image-text fusion feature data, the entity recognition module is used to perform entity recognition to obtain the target medical entity data, the entity relationship extraction module is used to perform entity relationship extraction to obtain the target medical relationship data, and the extracted target medical entity data and target medical relationship data are used to construct a multimodal knowledge graph; finally, when the multimodal medical question data is received, the model reasoning module and the multimodal knowledge graph are used to generate the image analysis report data. By combining the medical multimodal large model with the multimodal knowledge graph, highly automated medical image question and answer is achieved, thereby improving the efficiency of medical image question and answer.

[0086] See also Figure 2 In some embodiments, the entity recognition module further includes an attention layer, and the encoding layer, the attention layer and the sequence annotation layer are connected in sequence; the medical image question-answering method may further include: semantically enhancing the encoded feature data through the attention layer to obtain enhanced semantic feature data.

[0087] Correspondingly, the sequence annotation layer performs entity recognition based on the encoded feature data to obtain target medical entity data, including: semantically enhancing the encoded feature data through the attention layer to obtain enhanced semantic feature data.

[0088] Through the relationship extraction module, entity relationship extraction is performed based on the target medical entity data and the coded feature data to obtain the target medical relationship data, including: through the relationship extraction module, relationship identification is performed based on the target medical entity data, the coded feature data and the enhanced semantic feature data to obtain the target medical relationship data.

[0089] Specifically, medical image data can provide visual diagnostic information, while medical text data can provide more detailed background information. After the target multimodal medical data is input into the medical multimodal large model, the feature extraction module first extracts features from the medical image data and medical text data to obtain image-text fusion feature data; then the image-text fusion feature data is encoded by the encoding layer to obtain encoded feature data; finally, the encoded feature data enters the attention layer, which enhances the semantic information of the two modalities through the attention mechanism to generate enhanced semantic feature data. This mechanism helps the model focus on the most relevant parts, so as to better extract and utilize valuable information.

[0090] After obtaining the enhanced semantic feature data, a linear classification layer can be added to the embedded features at each position as a sequence annotation layer to predict the entity label. Finally, the output layer can generate the target medical entity data based on the entity label information output by the sequence annotation layer. Since the enhanced semantic feature data can provide more contextual detail information for the medical multimodal large model, the model can better understand the actual meaning of each word or phrase. When performing entity recognition through the sequence annotation layer, this enhancement helps to improve the precision, classification accuracy and boundary positioning of the entity, thereby reducing entity recognition errors.

[0091] The graph neural network (GNN) is integrated into the medical multimodal model to form a relationship extraction module. The target medical entity data, encoded feature data and enhanced semantic feature data are input into the GNN. The GNN can enhance the information interaction between different modalities through the attention mechanism. Then, based on the multimodal features, the explicit relationship between the target medical entities is extracted according to the defined relationship and the extracted target medical entity data; in addition, the enhanced semantic feature data can be used to extract the implicit relationship between the target medical entities. The explicit relationship and the implicit relationship together constitute the complete target medical relationship data.

[0092] Compared with the related art of extracting entities and relationships through deep learning or directly using a general large language model, the above embodiment adds a sequence annotation layer to the medical multimodal large model and combines it with the attention mechanism to make entity recognition more accurate, better capture complex semantic relationships, and mine potential implicit relationships.

[0093] See also Figure 3 In some embodiments, the step of acquiring target multimodal medical data includes:

[0094] S310, obtaining initial multimodal medical data.

[0095] S320: Structuring the initial medical text data to obtain structured text data.

[0096] S330 , performing standard processing on the initial medical image data to obtain standard image data.

[0097] S340, construct target multimodal medical data based on structured text data and standard image data.

[0098] The initial multimodal medical data includes initial medical image data and initial medical text data, and the target multimodal medical data includes structured text data and standard image data.

[0099] Specifically, initial multimodal medical data can be obtained from a variety of different sources. On the one hand, clinical data such as patient pathological information, disease description, and diagnosis results can be obtained through channels such as clinical data and electronic medical records; on the other hand, large-scale medical data can be obtained through public medical databases and data sets, such as MIMIC-III, TCIA, etc.; in addition, relevant research papers and medical imaging data can be obtained through literature retrieval and medical imaging databases as a supplement to clinical data.

[0100] The structural processing of initial medical text data includes cleaning and standardization.

[0101] In some embodiments, the cleaning process of the initial medical text data includes: first, using statistical methods or rules to detect and process outliers in the data to ensure data quality; second, using interpolation to fill missing values ​​in the data to avoid the impact of data loss on analysis results; finally, identifying and removing duplicate items in the data to avoid deviations in analysis results caused by data duplication.

[0102] In some embodiments, the standardized processing of initial medical text data includes: for patient information, natural language processing technology can be used to align and integrate personal information, disease descriptions, diagnostic information, etc.; for numerical data, normalization processing is performed to unify the numerical ranges between different features to avoid feature weight deviations in subsequent model fine-tuning; at the same time, data from different data sources are converted into a unified format, and the coding specifications and units are unified for subsequent processing and analysis.

[0103] After being cleaned and standardized, the initial medical text data becomes structured text data.

[0104] In some embodiments, the standard processing of the initial medical image data includes adjusting its size, brightness, contrast and other information, and using Gaussian filtering to remove possible noise in the image. After the processing is completed, standard image data is obtained.

[0105] In the above embodiment, the initial multimodal medical data is first obtained; secondly, the initial medical text data is structured to obtain structured text data, and the initial medical image data is standardly processed to obtain standard image data; finally, the target multimodal medical data is constructed based on the structured text data and the standard image data. On this basis, the target multimodal medical data can be used to fine-tune the pre-trained model, thereby improving the application ability of the model in the medical field. At the same time, it also provides a good data foundation for the construction of the knowledge graph.

[0106] See also Figure 4aIn some embodiments, when receiving multimodal medical problem data, generating image analysis report data corresponding to the multimodal medical problem data using the medical multimodal big model and the multimodal knowledge graph includes:

[0107] S410. When the multimodal medical problem data is received, a search is performed in the multimodal knowledge graph based on the multimodal medical problem data to obtain target subgraph data.

[0108] S420. Input the target subgraph data and the multimodal medical problem data into the model reasoning module for semantic enhancement reasoning to obtain the image analysis report data corresponding to the multimodal medical problem data.

[0109] Among them, multimodal medical problem data can include medical imaging data (such as X-rays, CT scans, magnetic resonance imaging, and ultrasound images, etc.) and medical text data (such as electronic medical records and clinical data, etc.). For example, you can input a lung X-ray and the text "What disease is this?"

[0110] The target subgraph data may be a subgraph consisting of target medical entity data and target medical relationship data related to the multimodal medical problem data retrieved by the medical multimodal big model.

[0111] In some embodiments, see Figure 4b ,The medical multimodal big model retrieves the relevant target ,medical entity data nodes based on the input multimodal medical problem data, ,which may include information such as disease name, symptoms, treatment methods, etc. These target ,medical entity data and the target medical relationship data between them ,(such as the association between disease and symptoms, the relationship between treatment methods and ,diseases, etc.) together constitute the target subgraph data.

[0112] After the target subgraph data and multimodal medical problem data are input into the model reasoning module, they first enter the encoding layer, where the visual encoder of the medical multimodal large model can be used to extract features from the medical image data to obtain image feature vectors, capturing the key structures and lesion information in the medical image data; at the same time, the medical text data is encoded into text feature vectors through the text encoder of the medical multimodal large model to capture the target medical entity data and target medical relationship data in the text. In addition, feature extraction is performed on the target subgraph data in the encoding layer to obtain a node representation vector, including the semantic information of the nodes and edges in the graph. Finally, the image feature vector, text feature vector and node representation vector are concatenated or weighted averaged to obtain the problem image-text fusion feature data. Furthermore, the problem image-text fusion feature data enters the attention layer, which can selectively focus on the most relevant part of the input data to the current task through the attention mechanism, and automatically learn the association and interaction between the various modalities. For example, a specific area in the medical image data may be associated with a symptom in the medical text data or a disease node in the target subgraph data. Through the cross-modal attention mechanism, the medical multimodal large model can learn and strengthen these semantic associations, so that different modal information can be effectively integrated in the reasoning process. Finally, the output layer outputs the image analysis report data corresponding to the multimodal medical problem data based on comprehensive reasoning, such as abnormal areas, lesion classification, and diagnostic suggestions in medical image data.

[0113] In some embodiments, see Figure 4c The medical multimodal big model 400 includes a feature extraction module 402, an entity recognition module 404, a relationship extraction module 406 and a model reasoning module 408. The medical multimodal big model 400 can extract features of the target multimodal medical data 401 through the feature extraction module 402, and perform entity recognition through the entity recognition module 404 based on the extracted features; and perform relationship extraction through the relationship extraction module 406 based on the extracted features and the recognized entities; and construct a multimodal knowledge graph 403 based on the obtained entities and relationships. In addition, the medical multimodal big model 400 can also perform reasoning based on the features extracted by the feature extraction module 402 and the multimodal knowledge graph 403 through the model reasoning module 408 to obtain the image analysis report data 405.

[0114] See also Figure 5a In some embodiments, searching in a multimodal knowledge graph based on multimodal medical problem data to obtain target subgraph data comprises:

[0115] S510, retrieving target nodes related to multimodal medical problem data in the multimodal knowledge graph.

[0116] S520. Retrieve edges and connection nodes directly related to the target node in the multimodal knowledge graph to obtain local subgraph data.

[0117] S530: Perform weight evaluation on the local subgraph data to obtain weighted target subgraph data.

[0118] The target node can be a related node retrieved from the multimodal knowledge graph based on the keywords or concepts in the multimodal medical problem data, which usually represents the core entity or concept of the multimodal medical problem. The local subgraph data can be data composed of the target node, the edges directly related to the target node, and the connecting nodes.

[0119] Specifically, see Figure 5b The medical multimodal big model retrieves and generates relevant target nodes in the multimodal knowledge graph based on the input multimodal medical problem data through Graph RAG (Graph Retrieval Augmented Generation). For example, for a certain disease, the retrieval may be the disease name, symptoms, treatment methods, etc., and the target node may be the disease name.

[0120] Furthermore, after obtaining the target node, Graph RAG combines the information retrieval capability of the graph structure and continues to query the edges (relationships) and other connected nodes directly related to the target node based on the similarity between the nodes and edges in the graph. The retrieved edges and connected nodes directly related to the target node and the target node together constitute the local subgraph data.

[0121] Furthermore, after obtaining the local subgraph data, Graph RAG can evaluate the weights of nodes and edges in the local subgraph by introducing technologies such as attention mechanisms, graph embedding methods, and relationship extraction. The basis for the evaluation may include factors such as the centrality of the node, the credibility of the edge, and the influence of the node. Through these evaluations, the weighted target subgraph data is finally obtained, which more accurately reflects which information or relationships are critical and important in the multimodal medical problem data.

[0122] Finally, the target subgraph data is combined with the original multimodal medical problem data to form an enhanced context, which is input into the model reasoning module, providing rich background knowledge for the generation process of the multimodal medical big model. With the enhanced context, the big model can consider more pathological semantic factors and multi-level semantic relationships when generating output, thereby enhancing the understanding of complex medical situations and generating more accurate and relevant image analysis report data.

[0123] In the above embodiment, first, the target node related to the multimodal medical problem data is retrieved in the multimodal knowledge graph; secondly, the edges and connection nodes directly related to the target node are retrieved in the multimodal knowledge graph to obtain local subgraph data; finally, the local subgraph data is weighted and evaluated to obtain weighted target subgraph data. By retrieving the local subgraph data, the hidden semantic relationships in the multimodal medical problem data can be revealed, providing more in-depth and extended information. On this basis, by weighting the local subgraph data, the importance of various types of information to semantic enhancement is determined, so that the multimodal medical large model can effectively perform medical image analysis and answer medical-related questions more accurately. Compared with the related technology in which the knowledge graph is used as a static knowledge base and the associated information in the knowledge graph cannot be used, the multimodal knowledge graph realizes the reverse support and optimization of the reasoning ability of the large model, and improves the depth and accuracy of reasoning.

[0124] See also Figure 6a In some embodiments, the medical image question answering method further includes:

[0125] S610. Entity recognition and relationship extraction are performed based on the image analysis report data through the medical multimodal big model to obtain medical entity data and medical relationship data in the image analysis report data.

[0126] S620. Update the multimodal knowledge graph based on the medical entity data and medical relationship data in the image analysis report data.

[0127] Among them, the image analysis report data can be generated by the multimodal medical big model based on the input multimodal medical problem data.

[0128] In this embodiment, after the multimodal medical big model generates image analysis report data based on the input multimodal medical problem data, it performs entity recognition and relationship extraction on the image analysis report data.

[0129] Specifically, the image analysis report data can be multimodal data, including text reports and image reports. First, the feature extraction module extracts features from the medical image data and medical text data to obtain report text and image fusion feature data. Secondly, the report text and image fusion feature data is input into the encoding layer of the entity recognition module to obtain report encoding feature data.

[0130] Furthermore, the report encoding feature data is input into the attention layer, where the attention mechanism enhances the semantic information of the two modalities of text and image to generate report enhanced semantic feature data. This mechanism helps the model focus on the most relevant parts, thereby better extracting and utilizing valuable information.

[0131] Furthermore, after obtaining the report enhanced semantic feature data, a linear classification layer can be added to the embedded features at each position as a sequence annotation layer to predict entity labels. Finally, the output layer can generate medical entity data based on the entity label information output by the sequence annotation layer.

[0132] Finally, the report encoding feature data, report enhanced semantic feature data and medical entity data are input into the graph neural network (GNN). GNN can enhance the information interaction between different modalities through the attention mechanism. Then, based on the report encoding feature data, the explicit relationship between medical entity data is extracted according to the defined relationship and the extracted medical entity data; in addition, the report enhanced semantic feature data is used to extract the implicit relationship between medical entity data. The explicit relationship and implicit relationship together constitute the complete medical relationship data.

[0133] Based on the obtained medical entity data and medical relationship data, the multimodal medical big model creates new nodes in the multimodal knowledge graph, assigns a unique identifier to each medical entity data, and adds its related attribute information (such as name, description, attributes, etc.) to the multimodal knowledge graph. At the same time, combined with the medical relationship data extracted from the image analysis report data, a new edge is established to connect it between the newly generated medical entity data node and the existing entity node, and ensure that the type and meaning of the new relationship are clear and unambiguous, so as to facilitate subsequent query and application in the multimodal knowledge graph.

[0134] In some embodiments, see Figure 6b ,The steps to update the multimodal knowledge graph through the target multimodal medical data are as follows:

[0135] 1) Fine-tune the pre-trained multimodal large model using target multimodal medical data to obtain a medical multimodal large model;

[0136] 2) Extract entities and relationships from target multimodal medical data through a large medical multimodal model and construct a multimodal knowledge graph;

[0137] 3) Select certain data from the target multimodal medical data (such as information about a patient, medical images from two examinations before and after), and input the selected data and questions (such as "Did the symptoms improve during the second examination") as multimodal medical question data into the medical multimodal big model;

[0138] 4) The medical multimodal large model uses Graph RAG technology to retrieve weighted target subgraph data in the multimodal knowledge graph, and uses this data to perform semantic enhancement on the multimodal medical problem data.

[0139] 5) After inference, the medical multimodal large model outputs the image analysis report data, performs entity recognition and relationship extraction on the image analysis report data, obtains medical entity data and medical relationship data, and updates the multimodal knowledge graph based on this data.

[0140] In the above embodiment, firstly, entity recognition and relationship extraction are performed based on the image analysis report data through the medical multimodal large model to obtain the medical entity data and medical relationship data in the image analysis report data; then the multimodal knowledge graph is updated based on the medical entity data and medical relationship data in the image analysis report data. Compared with the need to manually add new nodes in the related technology, which makes it more difficult to update the knowledge graph, this embodiment can automatically extract new entities and their relationships through entity recognition and relationship extraction in combination with the image analysis report data generated by the model after receiving the multimodal medical problem data input, and map these new entities and relationships to the existing multimodal knowledge graph, thereby realizing automatic update.

[0141] See also Figure 7 In some embodiments, the model reasoning module is obtained by:

[0142] S710, loading a pre-trained model, and freezing parameters of a text layer in the pre-trained model;

[0143] S720. Fine-tune the pre-trained model using the multimodal medical data sample set, update the parameters of the visual part layer in the pre-trained model, and obtain the model inference module.

[0144] The pre-trained model can be a large multimodal model such as InternVL2, LLAMA3.2, etc. The multimodal medical data sample set can be a subset of the target multimodal medical data, such as 80% of the multimodal medical data, and the other 20% of the data can be used as a validation set.

[0145] Multimodal pre-trained models have been very powerful in basic tasks (such as image recognition, language understanding, etc.), but they usually lack specific knowledge in the medical field. For example, standard visual-language models may understand the basic relationship between images and text, but may not be able to effectively handle complex domain knowledge such as medical terminology, disease classification, and symptoms. By fine-tuning on medical multimodal datasets, multimodal pre-trained models can specifically learn language and visual patterns in the medical field. For example, multimodal pre-trained models will learn the subtle connections between medical images and medical texts, and can provide more precise answers in specific medical question-and-answer scenarios.

[0146] In some implementations, the specific steps of obtaining the model reasoning module are as follows:

[0147] 1) Load the pre-trained model and its weights;

[0148] 2) Setting hyperparameters such as learning rate and number of iterations to fully retain the knowledge learned by the pre-trained model during pre-training and to enable further adjustment on new tasks;

[0149] 3) Select an appropriate loss function to measure the performance of the model on the target task;

[0150] 4) Freeze the parameters of the text layer in the pre-trained model and only update the parameters of the visual layer in the pre-trained model;

[0151] 5) Use multimodal medical data sample sets to fine-tune the model reasoning module.

[0152] It should be noted that in order to save computing resources, you can choose to use the LoRA (Low-Rank Adaptation) fine-tuning method. By introducing a low-rank matrix, LoRA can simulate the effect of full fine-tuning while using less computing power, thereby achieving efficient fine-tuning and achieving ideal performance improvement.

[0153] In the above embodiment, the pre-trained model is first loaded, and the parameters of the text part layer in the pre-trained model are frozen; secondly, the pre-trained model is fine-tuned using the multimodal medical data sample set, and the parameters of the visual part layer in the pre-trained model are updated to obtain the model reasoning module. By freezing the text part layer parameters and fine-tuning the visual part, and using the multimodal medical data sample set, the obtained model reasoning module can better adapt to medical tasks and improve the ability to integrate visual and text information. This not only improves the accuracy of medical reasoning, but also enhances the recognition accuracy of entities and relationships in the medical field.

[0154] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0155] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0156] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0157] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0158] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0159] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0160] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0161] The embodiment of the present application also provides a computer-readable storage medium. The above method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0162] The embodiment of the present application provides a computer program product, which includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of any embodiment of the present application.

[0163] Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

[0164] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0165] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0166] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0167] The present application is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0168] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0169] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0170] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0171] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0172] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

[0173] Although the embodiments of the present application have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A medical image question-answering method, characterized in that: The method comprises: Acquire target multimodal medical data; wherein the target multimodal medical data includes medical image data and medical text data; Inputting the target multimodal medical data into a medical multimodal macromodel; wherein the medical multimodal macromodel includes a feature extraction module, an entity recognition module, a relationship extraction module and a model reasoning module; Performing feature extraction on the medical image data and the medical text data by the feature extraction module to obtain image-text fusion feature data; Through the entity recognition module, entity recognition is performed based on the image-text fusion feature data to obtain target medical entity data; Through the relationship extraction module, entity relationship extraction is performed based on the target medical entity data and the image-text fusion feature data to obtain target medical relationship data; wherein the target medical entity data and the target medical relationship data are used to construct a multimodal knowledge graph; the multimodal knowledge graph includes relationship data between disease entities and medical imaging entities; When multimodal medical problem data is received, the model reasoning module and the multimodal knowledge graph are used to generate image analysis report data corresponding to the multimodal medical problem data.

2. The method according to claim 1, characterized in that The entity recognition module includes a coding layer and a sequence annotation layer connected in sequence; through the entity recognition module, entity recognition is performed based on the image-text fusion feature data to obtain target medical entity data, including: Encoding the image-text fusion feature data through the encoding layer to obtain encoded feature data; Through the sequence annotation layer, entity recognition is performed based on the coded feature data to obtain the target medical entity data.

3. The method according to claim 2, characterized in that The entity recognition module further includes an attention layer, and the encoding layer, the attention layer and the sequence labeling layer are sequentially connected; the method further includes: Performing semantic enhancement on the encoded feature data through the attention layer to obtain enhanced semantic feature data; Accordingly, entity recognition is performed based on the coded feature data through the sequence annotation layer to obtain the target medical entity data, including: The enhanced semantic feature data is subjected to entity recognition through the sequence annotation layer to obtain the target medical entity data.

4. The method according to claim 3, characterized in that The entity relationship extraction module is used to extract the target medical entity data and the image-text fusion feature data to obtain the target medical relationship data, including: Through the relationship extraction module, relationship identification is performed based on the target medical entity data, the image-text fusion feature data and the enhanced semantic feature data to obtain the target medical relationship data.

5. The method according to claim 1, characterized in that The acquiring of target multimodal medical data includes: Acquiring initial multimodal medical data; wherein the initial multimodal medical data includes initial medical image data and initial medical text data; Performing structured processing on the initial medical text data to obtain structured text data; Performing standard processing on the initial medical image data to obtain standard image data; The target multimodal medical data is constructed based on the structured text data and the standard image data.

6. The method according to claim 1, characterized in that When receiving the multimodal medical problem data, generating image analysis report data corresponding to the multimodal medical problem data by using the model reasoning module and the multimodal knowledge graph includes: Upon receiving the multimodal medical problem data, searching the multimodal knowledge graph based on the multimodal medical problem data to obtain target subgraph data; The target subgraph data and the multimodal medical problem data are input into the model reasoning module for semantic enhancement reasoning to obtain the image analysis report data corresponding to the multimodal medical problem data.

7. The method according to claim 6, characterized in that The step of searching the multimodal knowledge graph based on the multimodal medical problem data to obtain target subgraph data includes: Retrieving a target node related to the multimodal medical problem data in the multimodal knowledge graph; Retrieving edges and connection nodes directly related to the target node in the multimodal knowledge graph to obtain local subgraph data; The local sub-graph data is weighted to obtain the weighted target sub-graph data.

8. The method according to claim 6, characterized in that The method further comprises: By using the medical multimodal large model, entity recognition and relationship extraction are performed based on the image analysis report data to obtain medical entity data and medical relationship data in the image analysis report data; The multimodal knowledge graph is updated based on the medical entity data and medical relationship data in the image analysis report data.

9. The method according to any one of claims 1 to 8, characterized in that: The model reasoning module is obtained by: Loading a pre-trained model and freezing parameters of a text layer in the pre-trained model; The pre-trained model is fine-tuned using a multimodal medical data sample set, and the parameters of the visual part layer in the pre-trained model are updated to obtain the model reasoning module.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 9 by executing the computer instructions.

Images