Apparatus and storage medium for multimodal based disease recognition

Abstract

The application discloses a device and a storage medium for disease recognition based on multi-modal. The device comprises a processor and a memory, and implements the following operations: constructing a knowledge graph for acute abdomen recognition; dividing the knowledge graph into subgraphs to obtain an initial subgraph set; calculating the initial correlation of clinical information and each subgraph in the initial subgraph set, and extracting a target subgraph set with the maximum correlation from the initial subgraph set in combination with a subgraph knowledge confidence measure; inputting the clinical information and the target subgraph set into a language encoder for text encoding to obtain text features; performing image encoding on image data based on the image data to extract image features; and decoding based on the text features and the image features by using a multi-modal decoder and a subgraph knowledge verification model to obtain a recognition result of the acute abdomen recognition. The scheme can provide accurate and reliable recognition results and decision support for the clinic.

Description

Technical Field

[0001] This application generally relates to the field of medical artificial intelligence technology. More specifically, this application relates to a device and computer-readable storage medium for multimodal disease recognition. Background Technology

[0002] Acute abdomen is a general term for abdominal diseases characterized by acute abdominal pain, encompassing various conditions such as acute appendicitis, pancreatitis, intestinal obstruction, and gastrointestinal perforation. Its etiology is complex, and its progression is rapid. Accurate and rapid identification is directly related to treatment planning, patient prognosis, and mortality control, making it a critical issue that urgently needs to be addressed in emergency and surgical clinical practice. Accurate identification of acute abdomen helps doctors quickly differentiate between different causes, avoid misdiagnosis and missed diagnosis, gain valuable time for critically ill patients, and significantly improve the efficiency and quality of clinical diagnosis and treatment.

[0003] Currently, technologies related to intelligent identification of acute abdomen largely rely on single medical record data or imaging features, failing to achieve deep synergy between clinical information, medical guidelines, cutting-edge literature, and imaging data. In terms of knowledge graph applications, traditional subgraph partitioning is based solely on the topological structure of graph networks, making it difficult to match the clinical thinking patterns involved in the diagnosis and treatment of acute abdomen. Furthermore, subgraph selection relies on only a single relevance indicator, resulting in insufficient accuracy in subgraph retrieval and an inability to provide personalized medical knowledge support for individual patients. In addition, existing technologies often involve simple splicing at the feature fusion level and directly output results, limiting the comprehensiveness and accuracy of the identification results and making it difficult to meet the needs of complex clinical scenarios.

[0004] In view of this, this application provides a scheme for the identification of acute abdominal pain, in order to solve the problems of data subgraph segmentation being out of touch with clinical practice, insufficient reliability of subgraph screening, and inadequate feature fusion, so as to provide accurate and comprehensive acute abdominal pain identification results and decision support for clinical practice. Summary of the Invention

[0005] In order to at least address one or more of the technical problems mentioned above, this application proposes a multimodal disease identification scheme in several aspects.

[0006] In a first aspect, this application provides a multimodal disease identification apparatus, comprising: a processor; and a memory storing computer instructions for multimodal disease identification, wherein when the computer instructions are executed by the processor, the following operations are performed: constructing a knowledge graph for acute abdominal pain identification based on clinical information and a target literature set; dividing the knowledge graph into subgraphs to obtain an initial subgraph set; calculating the initial relevance between the clinical information and each subgraph in the initial subgraph set; extracting a target subgraph set with the highest relevance from the initial subgraph set by combining the initial relevance and a subgraph knowledge confidence metric; inputting the clinical information and the target subgraph set into a language encoder for text encoding to obtain text features; performing image encoding using an image encoder based on image data to extract image features; and decoding using a multimodal decoder and a subgraph knowledge verification model based on the text features and the image features to obtain an identification result for acute abdominal pain identification.

[0007] In some embodiments, the apparatus further performs the following operations: setting different partitioning hyperparameters to split and obtain multiple initial subgraphs; merging initial subgraphs whose difference values ​​between different initial subgraphs are less than a difference threshold to optimize the set of initial subgraphs.

[0008] In some embodiments, the apparatus further performs the following operations to calculate the initial relevance: extracting key information based on the clinical information to construct key entity nodes; calculating the intersection ratio between the key entity nodes and nodes of each subgraph in each initial subgraph set; calculating the semantic similarity between the clinical information and coherent text of each subgraph in each initial subgraph set; and calculating the initial relevance based on the corresponding intersection ratio and corresponding semantic similarity in each initial subgraph set.

[0009] In some embodiments, the apparatus further performs the following operations to obtain the subgraph knowledge confidence metric: setting source weights for each subgraph in the initial subgraph set based on the knowledge graph; calculating confidence values ​​of the association strength between nodes within each subgraph; and multiplying the source weights and the confidence values ​​to obtain the knowledge confidence weight of each subgraph, thereby obtaining the subgraph knowledge confidence metric.

[0010] In some embodiments, the apparatus further performs the following operations to extract the target sub-graph set with the highest relevance: calculates the weighted value of the initial relevance and the knowledge confidence weight of each sub-graph to obtain a comprehensive relevance score; and extracts the target sub-graph set with the highest relevance from the initial sub-graph set based on the comprehensive relevance score.

[0011] In some embodiments, the apparatus further performs the following operations: calculating the pathological path correlation degree between the key entity node and the node of each subgraph in each initial subgraph set; and calculating the initial correlation based on the corresponding intersection ratio, corresponding semantic similarity, and corresponding pathological path correlation degree in each initial subgraph set.

[0012] In some embodiments, the device further performs the following operations: constructing a clinical feature vector, the clinical feature vector including one or more of symptom complexity, number of complications, and length of medical history; and determining the fusion weights of the corresponding intersection ratio, corresponding semantic similarity, and corresponding pathological path correlation degree under each initial sub-map based on the clinical feature vector, so as to calculate the initial correlation.

[0013] In some embodiments, the apparatus further performs the following operations: annotating key region masks in the image data according to the target sub-atlas; and performing image encoding using an image encoder to extract image features based on the image data and the key region masks.

[0014] In some embodiments, the device further performs the following operations to obtain an acute abdominal pain identification result: based on the text features and the image features, a multimodal decoder is used to decode to obtain an initial identification result for acute abdominal pain identification; the initial identification result is input into a subgraph knowledge verification model for verification to obtain a matching score; in response to the matching score being higher than a first preset threshold, the initial identification result is output; in response to the matching score being lower than the first preset threshold, a feature quantity corresponding to a similarity higher than a second preset threshold is selected from the initial identification result; based on the text features, the image features, and the feature quantity, a second decoding is performed using a multimodal decoder to obtain an identification result, wherein the first preset threshold is greater than the second preset threshold.

[0015] In a second aspect, this application provides a computer-readable storage medium having stored thereon computer program instructions for multimodal disease identification, which, when executed by one or more processors, cause to perform the operations performed by the apparatus described in one or more embodiments of the first aspect.

[0016] Using the multimodal disease identification scheme provided above, this application's embodiments construct a knowledge graph by fully integrating clinical information and target literature sets, and achieve clinically appropriate knowledge decomposition by combining multi-dimensional subgraph partitioning. By calculating initial relevance and combining it with subgraph knowledge confidence metrics, a two-dimensional precise selection of target subgraph sets is achieved. Then, text and image features are extracted by a multimodal encoder, fused by a multimodal decoder, and corrected for biases by combining a subgraph knowledge verification model. This effectively solves the problems of subgraph partitioning being detached from clinical practice, insufficient reliability of subgraph selection, and lack of verification of identification results, achieving precise and reliable identification of acute abdominal pain, and providing clinicians with scientifically sound and practical decision support. Attached Figure Description

[0017] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this application are illustrated by way of example and not limitation, and the same or corresponding reference numerals denote the same or corresponding parts, wherein: Figure 1 This is an exemplary structural block diagram illustrating a multimodal disease identification apparatus 100 according to an embodiment of this application; Figure 2 This is an exemplary flowchart illustrating operation 200 implemented by a device for multimodal disease identification according to an embodiment of this application; Figure 3 This is an exemplary flowchart illustrating the overall operation of the device for multimodal disease identification according to an embodiment of this application; Figure 4 An exemplary structural block diagram of an electronic device 400 according to an embodiment of this application is shown. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0019] It should be understood that the terms "comprising" and "including" as used in the specification and claims of this application indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0020] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0021] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0022] The specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0023] Figure 1 This is an exemplary structural block diagram illustrating a multimodal disease identification apparatus 100 according to an embodiment of this application. Figure 1 As shown, the device 100 may include a processor 110 and a memory 120. The processor 110 may include, for example, a general-purpose processor (“CPU”) or a dedicated graphics processor (“GPU”), and the memory 120 stores program instructions executable on the processor. In some embodiments, the memory 120 may include, but is not limited to, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), and enhanced dynamic random access memory (EDRAM).

[0024] Furthermore, the aforementioned memory 120 can store program instructions for multimodal disease recognition. When the aforementioned program instructions are executed by the processor 110, the device 100 performs the following operations: constructing a knowledge graph for acute abdominal pain recognition based on clinical information and a target literature set; dividing the knowledge graph into subgraphs to obtain an initial subgraph set; calculating the initial relevance between the clinical information and each subgraph in the initial subgraph set; extracting the target subgraph set with the highest relevance from the initial subgraph set by combining the initial relevance and the subgraph knowledge confidence metric; inputting the clinical information and the target subgraph set into a language encoder for text encoding to obtain text features; using an image encoder to perform image encoding based on image data to extract image features; and using a multimodal decoder and a subgraph knowledge verification model to decode based on text features and image features to obtain the recognition result for acute abdominal pain recognition. The following will combine... Figure 2 The operation implemented by the apparatus 100 in the embodiments of this application is described in detail.

[0025] Figure 2 This is an exemplary flowchart illustrating operation 200 implemented by a multimodal disease identification apparatus according to an embodiment of this application. Figure 2 As shown, in step S201, a knowledge graph for acute abdominal pain identification is constructed based on clinical information and the target literature set. It can be understood that the knowledge graph is a graph network with medical entities as nodes and relationships between entities as edges, integrating clinical practice patterns and authoritative medical knowledge. The core data structure of the knowledge graph is that nodes represent medical concepts, and edges represent semantic relationships between concepts.

[0026] In some implementation scenarios, clinical information can be extracted from multi-source medical data platforms such as electronic medical record systems and laboratory information systems. Specifically, clinical information can encompass the patient's medical history, such as present illness, past medical history, and family history; symptoms and signs, such as the location, nature, duration, accompanying symptoms, degree of abdominal muscle tension, and rebound tenderness of abdominal pain; laboratory test indicators, such as complete blood count, amylase, bilirubin, and inflammatory factors; and medication history and allergy history. The data format can be standardized as structured JSON. For unstructured text such as medical records, natural language processing (NLP) technology can be used to convert it into structured data, ensuring data standardization and operability.

[0027] The aforementioned target literature set can include authoritative sources such as clinical guidelines, cutting-edge academic literature, treatment protocols, and expert consensus related to acute abdomen. Target literature is screened using keywords such as "acute abdomen," "diagnosis and treatment of acute appendicitis," and "differential diagnosis of pancreatitis" to ensure that the content is highly relevant to the identification of acute abdomen, thus providing authoritative medical knowledge support for the knowledge graph.

[0028] In the implementation scenario, the clinical information and target literature set can first undergo data preprocessing, such as format processing. Next, a large language model (e.g., GPT-4 or LlaMA model) is used to extract the relationships between medical entities, constructing triples containing head entities, relationships, and tail entities. For example, extracting "Patient A, persistent upper abdominal pain, serum amylase increased 3-fold" from clinical information can be transformed into the triple [persistent upper abdominal pain, accompanying indicators, elevated serum amylase]. Similarly, extracting "patients with acute pancreatitis often experience elevated serum amylase" from target literature can be transformed into the triple [acute pancreatitis, typical indicators, elevated serum amylase].

[0029] All triples are mapped to a standardized medical coding system (such as UMLS) to ensure consistent representation of medical concepts from different sources. Finally, medical entities are used as nodes in a graph network, with relationships between entities serving as edge attributes, such as "accompanying indicators," "typical symptoms," and "pathological associations." By integrating all standardized triples, a complete knowledge graph for acute abdominal pain identification is constructed. This graph includes both real-world case associations from clinical practice and theoretical knowledge from authoritative literature, achieving a deep integration of clinical data and medical knowledge.

[0030] Next, in step S202, the knowledge graph is divided into subgraphs to obtain an initial subgraph set.

[0031] Specifically, the Leiden algorithm can be used to initially partition the knowledge graph into subgraphs. Based on the modular optimization criterion, the Leiden algorithm divides the graph network into multiple tightly connected subgraphs through an iterative optimization process, ensuring that the correlation between nodes within each subgraph is higher than the correlation between subgraphs.

[0032] Next, prompts can be set for each subgraph in each sub-image set to guide the conversion of the subgraph into coherent text. The prompt design focuses on the needs of acute abdomen recognition; for example, please summarize the core knowledge related to the diagnosis of acute abdomen based on the following medical triples: [sub-image] Figure 3 [List of tuples]. The triple list for each subgraph is processed using a large language model to generate corresponding coherent text. For example, the coherent text for the subgraph "acute appendicitis" is: "The typical symptoms of acute appendicitis are migratory right lower quadrant abdominal pain, often accompanied by right lower quadrant tenderness and rebound tenderness. Laboratory tests show elevated white blood cell count, and imaging examinations reveal thickened appendix and surrounding exudate." This process transforms structured subgraph data into natural language text that is easy to semantically retrieve, laying the foundation for subsequent relevance calculations.

[0033] In some embodiments, the apparatus further performs the following operations: splitting the graph into multiple initial subgraphs using different partitioning hyperparameters; merging initial subgraphs whose differences are less than a difference threshold to optimize the initial subgraph set. In some implementations, the aforementioned partitioning hyperparameters can be, for example, a random seed, a resolution parameter, or an iteration stopping condition. Splitting the graph into multiple initial subgraphs using different partitioning hyperparameters involves multiple partitioning operations. For example, running the algorithm multiple times with different random seeds yields multiple initial subgraphs; running it with different resolution parameters yields initial subgraphs of different granularities. For initial subgraphs whose differences are less than a difference threshold, merging them using methods such as consensus clustering or ensemble methods yields a better initial subgraph set. The difference threshold can be set based on the specific scenario to minimize the differences between different initial subgraphs. This ensures that the retained subgraphs are both free of redundancy and reliable, thus optimizing the quality of the initial subgraph set.

[0034] In other embodiments, two types of coherent text can be generated for each subgraph. One type focuses on key core needs, such as "clinical diagnosis and treatment of acute abdomen," ensuring the text is highly relevant to the recognition requirements. The other type is global, such as "clinical prediction," expanding the knowledge coverage of the text and avoiding omission of potential related knowledge. As an example, for the subgraph related to "gallbladder stones," the first type of coherent text is: "Gallbladder stones can cause biliary colic. If a stone breaks off and obstructs the pancreatic duct, it can lead to acute biliary pancreatitis. Clinically, abdominal ultrasound and amylase tests are needed for differentiation." The second type of coherent text is: "Patients with gallbladder stones who are also obese and have high blood lipids have an increased risk of developing acute pancreatitis. It is necessary to pay attention to changes in the patient's metabolic indicators and abdominal signs." The two types of text complement each other, improving the comprehensiveness of the subgraph's knowledge.

[0035] In some implementation scenarios, the initial subgraph can be further processed to extract multi-dimensional subgraphs, forming a three-dimensional subgraph set. The first-dimensional subgraph set corresponds to key subgraphs. For example, subgraphs covering general differential diagnostic knowledge of acute abdomen and tightly interconnected cross-domain relationships, as well as subgraphs focusing on specific medical problems or sub-fields. That is, subgraphs focusing on key knowledge for identifying acute abdomen, such as those including typical symptom-cause correlations, key examination-diagnosis correspondences, and early warnings of common complications, such as the subgraph related to "migratory right lower quadrant pain - right lower quadrant tenderness - acute appendicitis".

[0036] The second-dimensional subgraph corresponds to a local subgraph of a single organ. In the implementation scenario, N organs with a high incidence of acute abdominal pain (such as the stomach, intestines, liver, gallbladder, pancreas, appendix, etc.) can be defined. The n-hop (e.g., 3-hop) neighbor nodes directly connected to each organ can be extracted from the global knowledge graph to form a local subgraph centered on that organ. For example, a local subgraph centered on the pancreas includes nodes and relationships related to the pancreas, amylase, abdominal pain, biliary obstruction, and acute pancreatitis.

[0037] The third-dimensional subgraph set corresponds to the common local subgraphs of strongly interacting organ pairs. In some embodiments, strongly interacting organ pairs may include, but are not limited to, one or more of the following: gallbladder and pancreas, stomach and duodenum, small intestine and colon, liver and bile duct, appendix and cecum. These organ pairs are anatomically adjacent and physiologically synergistic, representing a common combination in the cross-organ pathogenesis of acute abdomen. The common local subgraphs of each organ pair are extracted; for example, the common local subgraph of the gallbladder and pancreas pair includes nodes and relationships related to the gallbladder, gallstones, bile reflux, pancreas, and acute biliary pancreatitis. Knowledge decomposition for clinical adaptation is achieved through multi-dimensional subgraph partitioning.

[0038] Further, in step S203, the initial correlation between clinical information and each subplot in the initial subplot set is calculated.

[0039] In some embodiments, the above-described apparatus may further perform the following operations to calculate the initial relevance: extracting key information based on clinical information to construct key entity nodes; calculating the intersection ratio between the key entity nodes and nodes of each subgraph in each initial subgraph set; calculating the semantic similarity between the clinical information and the coherent text of each subgraph in each initial subgraph set; and calculating the initial relevance based on the corresponding intersection ratio and corresponding semantic similarity in each initial subgraph set.

[0040] In some implementation scenarios, core medical entities can be extracted from patient clinical information using, for example, large language models. These entities include those corresponding to key information such as symptoms, signs, examination indicators, and medical history. For instance, if a patient's clinical information is "female, 45 years old, persistent upper abdominal pain for 8 hours, accompanied by nausea and vomiting, serum amylase elevated 3 times, history of gallstones," the extracted set of key entity nodes would be {persistent upper abdominal pain, nausea and vomiting, elevated serum amylase, history of gallstones}.

[0041] In other implementations, the intersection ratio between key entity nodes and subgraph nodes can be calculated using `ratio = len(entities_g ∩ entities_patient) / len(entities_patient)`, where `entities_patient` represents subgraph nodes, `entities_g` represents key entity nodes, and `len` represents the number of entities. In still other implementations, patient clinical information and coherent subgraph text can be transformed into high-dimensional semantic vectors using models such as BERT, and their semantic similarity can be calculated using a cosine similarity formula. The cosine similarity formula is: `similarity_to_summary = cosine_similarity(V_patient, V_summary) = (V_patient...` The text similarity is calculated using the formula V_summary / (||V_patient|| × ||V_summary||). Here, V_patient is the clinical information vector, and V_summary is the coherent text vector. Represents the vector dot product. Let L2 norm represent the vector. For coherent text of two classes, the similarity to the clinical information can be calculated separately for each class, and then the total similarity can be calculated by weighted summation.

[0042] In some embodiments, the above-described apparatus may further perform the following operations: calculate the semantic difference between the first type of coherent text and the second type of coherent text; and remove texts with a semantic difference less than a preset value from the first type of coherent text and the second type of coherent text, respectively, to optimize the first and second coherent texts. It can be understood that the semantic difference is an indicator that quantifies the degree of semantic overlap between the first type of coherent text (direct risk) and the second type of coherent text (global management). The smaller the difference, the more redundant information exists between the two types of texts.

[0043] In some implementation scenarios, the semantic difference between the first type of coherent text V_summary1 and the second type of coherent text V_summary2 can be calculated based on 1-cosine_similarity(V_summary1, V_summary2). During calculation, each coherent text can be broken down into sentences, and the semantic difference between each sentence and all sentences in the other text can be calculated. Texts with a semantic difference less than a preset value (e.g., 0.3) are discarded as redundant text segments. The sentences after removing redundancy are then reassembled into corresponding coherent texts according to their original semantic logic to optimize the first and second types of coherent texts. Based on this, the core differentiated information of the two texts can be accurately preserved while reducing information duplication, improving the efficiency and accuracy of subsequent language encoding. Combining the intersection ratio and semantic similarity, the initial correlation of the intersection ratio and similarity can be calculated using similarity_total=(ratio+similarity_to_summary) / 2.

[0044] In some embodiments, the above-described apparatus further performs the following operations: calculating the pathological path correlation degree between key entity nodes and nodes in each subgraph of each initial subgraph set, and calculating initial relevance based on the corresponding intersection ratio, corresponding semantic similarity, and corresponding pathological path correlation degree in each initial subgraph set. That is, pathological path correlation degree is introduced as a supplementary relevance indicator. The pathological path correlation degree is quantified by calculating the shortest pathological path length between key entity nodes and subgraph nodes; the shorter the path length, the higher the correlation degree.

[0045] As an example, the shortest pathological path between the patient's key entity "History of Gallstones" and the subgraph node "Acute Pancreatitis" is Gallstones → Bile Reflux → Pancreatic Duct Obstruction → Acute Pancreatitis, with a path length of 3. The corresponding pathological path relevance is 1 / 3 ≈ 0.33. The reciprocal of the path length ranges from 0 to 1. In this scenario, the initial relevance can be calculated using similarity_total = (ratio + similarity + path_relevance) / 3. The path_relevanc value represents the pathological path relevance, further improving the accuracy of the relevance calculation and ensuring that the selected subgraphs highly match the patient's pathological mechanisms.

[0046] In some embodiments, the above-described apparatus further performs the following operations: constructing a clinical feature vector, which includes one or more of symptom complexity, number of complications, and length of medical history; determining the fusion weights of the corresponding intersection ratio, corresponding semantic similarity, and corresponding pathological path correlation degree under each initial sub-map based on the clinical feature vector, to calculate the initial relevance. That is, the fusion weights of each relevance indicator are dynamically adjusted through the clinical feature vector. In some implementation scenarios, the clinical feature vector can be processed using, for example, a logistic regression model to output the fusion weights of each relevance indicator.

[0047] In the exemplary scenario, assuming a patient's symptom complexity is 0.8 (high complexity), the number of complications is 1, and the medical history is 5 years, the feature vector is constructed as [0.8, 1, 5]. Through logistic regression, when the symptom complexity is high, the weight of semantic similarity is increased to 0.4, the weight of intersection ratio is 0.3, and the weight of pathological path correlation is 0.3. When the medical history is long, the weight of pathological path correlation is increased to 0.4, ensuring that the correlation calculation can adapt to the clinical characteristics of different patients, further improving the personalized adaptation of the target sub-map.

[0048] After obtaining the initial relevance, in step S204, the target subgraph set with the highest relevance is extracted from the initial subgraph set by combining the initial relevance and the subgraph knowledge confidence metric. In some embodiments, the above apparatus may further perform the following operations to obtain the subgraph knowledge confidence metric: setting source weights for each subgraph in the initial subgraph set based on the knowledge graph; calculating the confidence value of the association strength between nodes in each subgraph; multiplying the source weights and the confidence values ​​to obtain the knowledge confidence weight of each subgraph, thereby obtaining the subgraph knowledge confidence metric.

[0049] Specifically, weights can be assigned to clinical data, clinical guidelines, and cutting-edge literature within the knowledge graph, with a total weight of 1. Clinical data, derived from a large number of real-world case studies, closely reflects clinical practice and can therefore have the highest weight, such as 0.6. Clinical guidelines, with their strong evidence-based basis, can have a slightly lower weight, such as 0.3. Cutting-edge literature, requiring validation through clinical practice, can have the lowest weight, such as 0.1. In this scenario, if 70% of the triplets in a subgraph come from clinical data, 20% from guidelines, and 10% from literature, then its source weight (source_weight) could be 0.7 × 0.6 + 0.2 × 0.3 + 0.1 × 0.1 = 0.49.

[0050] Next, the strength of the association between nodes within the subgraph can be quantified using, for example, the Pearson correlation coefficient. Combined with adjustments for co-occurrence frequency, a confidence value for the association strength can be obtained. Specifically, the confidence value can be calculated as: confidence = (pearson_corr + 1) / 2 × log10(co_occurrence + 1), where pearson_corr is the Pearson correlation coefficient between nodes (ranging from -1 to 1), and co_occurrence is the co-occurrence frequency of the association, such as the number of times the triple appears in clinical data. In this scenario, the Pearson correlation coefficient for the core association within the subgraph, acute pancreatitis-elevated serum amylase, is 0.8. If the co-occurrence frequency is 1200, then confidence = (0.8+1) / 2×log10(1200+1) ≈ 0.9×3.08 = 2.77. Normalizing this to the 0-1 interval yields normalized_confidence = 0.85. Multiplying the source weight by the normalized association strength confidence value gives the knowledge confidence weight: confidence_weight = source_weight × normalized_confidence. A higher value indicates higher confidence in the subgraph knowledge.

[0051] In some embodiments, the above-described apparatus may further perform the following operations to extract the target sub-graph set with the highest relevance: calculating the weighted value of the initial relevance and the knowledge confidence weight of each sub-graph to obtain a comprehensive relevance score; and extracting the target sub-graph set with the highest relevance from the initial sub-graph set based on the comprehensive relevance score. Specifically, the initial relevance is weighted and summed with the knowledge confidence weight to obtain a comprehensive relevance score. The top-N (e.g., top-5) most relevant sub-graphs are selected according to the comprehensive relevance scores from high to low to form the target sub-graph set. For example, a patient's five highly relevant sub-graphs are "differentiation of acute biliary pancreatitis," "complications of gallstones," "diseases related to elevated serum amylase," "differential diagnosis of upper abdominal pain," and "pathological association between pancreas and gallbladder." These sub-graphs accurately match the patient's clinical characteristics, providing personalized knowledge support for subsequent identification.

[0052] In step S205, clinical information and the target sub-graph set are input into a language encoder for text encoding to obtain text features. In some implementations, the language encoder can be, for example, a BERT model. As mentioned earlier, clinical information can be converted into natural language using structured JSON, and the coherent text of each target sub-graph is concatenated into knowledge text. The Transformer architecture in the BERT model can capture the semantic relationships within the text context to obtain text features. Based on this, by transforming unstructured and semi-structured clinical text and medical knowledge text into high-dimensional semantic features, the individual features of patient clinical information and the medical knowledge features in the target sub-graph set are fully integrated, providing accurate text feature support for subsequent multimodal fusion.

[0053] Further, in step S206, image encoding is performed using an image encoder based on the image data to extract image features.

[0054] In some embodiments, the image data may include abdominal CT images, ultrasound images, MRI images, etc., in the DICOM standard format, containing metadata such as patient ID, scan time, and image parameters. Before image encoding, the image data can be preprocessed. For example, automatic thresholding can be used to preserve the region of interest (ROI) in the abdomen and remove irrelevant tissues (such as back muscles and subcutaneous fat). Bilinear interpolation can be used to scale the images to a uniform size (e.g., 256×256 pixels) to ensure consistent image size. Global normalization can be performed to eliminate imaging differences caused by different scanning devices and parameters, improving the comparability of the image data.

[0055] In some implementation scenarios, the image encoder can be, for example, a Vision Transformer model, which can be adapted to capture the correlation between global and local features in medical images. In this scenario, the Vision Transformer model first transforms image patches into high-dimensional vectors through linear projection, and preserves the spatial location information of the patches (such as the relative position of aneurysms and surrounding blood vessels) by adding sinusoidal positional encoding. Then, through multi-head self-attention computation, it captures the dependencies between patches, such as the differences between the aneurysm edge and normal brain tissue, and outputs image features. Based on this, the original image can be transformed into a high-dimensional feature vector, preserving the core information (morphology, location, density) of cerebrovascular lesions, providing accurate image feature support for subsequent multimodal fusion.

[0056] In some embodiments, the above-described apparatus further performs the following operations: annotating key region masks in image data according to the target sub-atlas; and performing image encoding using an image encoder based on the image data and the key region masks to extract image features.

[0057] For example, the target sub-image set includes sub-images related to the pancreas and gallbladder. By locating the corresponding organ regions in the image using entity nodes (such as pancreas and gallbladder) in these sub-images, a key region mask is generated. This masked region represents the location of the pancreas and gallbladder, while other regions are unmasked. The image data and the key region mask are input into the image encoder. The model uses an attention mechanism to focus on feature extraction from the masked regions, strengthening the feature representation of lesion areas and weakening interference from irrelevant areas, further improving the accuracy of image features. For instance, after labeling the pancreas region mask, the model significantly improves the accuracy of extracting features such as pancreatic enlargement and abnormal density, which helps in the accurate identification of acute pancreatitis.

[0058] Finally, in step S207, based on text features and image features, a multimodal decoder and a subgraph knowledge verification model are used for decoding to obtain the recognition result of acute abdomen identification. In some embodiments, the above-described apparatus further performs the following operations to obtain the recognition result of acute abdomen identification: based on text features and the image features, a multimodal decoder is used for decoding to obtain an initial recognition result of acute abdomen identification; the initial recognition result is input into a subgraph knowledge verification model for verification to obtain a matching score; in response to a matching score higher than a first preset threshold, the initial recognition result is output; in response to a matching score lower than the first preset threshold, a feature quantity corresponding to a similarity higher than a second preset threshold is selected from the initial recognition result; based on text features, image features, and feature quantity, a second decoding is performed using a multimodal decoder to obtain the recognition result, wherein the first preset threshold is greater than the second preset threshold.

[0059] In some implementation scenarios, a multimodal decoder can be, for example, a cross-attention fusion layer with eight attention heads and two fully connected layers, with the output layer using sigmoid activation. The cross-modal attention fusion layer captures the cross-modal association between text features and image features, while the fully connected layer handles non-linear transformation of features and outputs the recognition result. First, text feature vectors and image feature vectors are input into the cross-modal attention fusion layer. Within the layer, attention weights are calculated between text features and image features to capture their semantic association. For example, the association between elevated blood amylase in text and pancreatic enlargement in image. The two features are then weighted and fused to form a cross-modal fused feature. For example, the attention weight for elevated blood amylase in text features is 0.8, and the attention weight for pancreatic enlargement in image features is 0.9; this weighted fusion strengthens the expression of these two associated features.

[0060] Cross-modal fusion features are input into a fully connected layer. After two layers of nonlinear transformation, the recognition results are output through a Sigmoid activation function. The recognition results are structured outputs, including the etiological diagnosis of acute abdominal pain, such as acute biliary pancreatitis, acute appendicitis, intestinal obstruction, etc.; severity grading, such as mild, moderate, and severe; ranking of suspected diseases, such as sorting by probability from high to low; and key evidence, such as a diagnosis of acute biliary pancreatitis based on persistent upper abdominal pain, a 3-fold increase in serum amylase, a history of gallstones, and CT scan showing pancreatic enlargement, providing clinicians with clear and intuitive decision-making references.

[0061] For subgraph knowledge verification models, the first step is to integrate the coherent text of the target subgraph set with a list of triples to form a structured verification knowledge base containing core knowledge items. For example, "A typical indicator of acute pancreatitis is elevated serum amylase," and "Pancreatitis caused by gallstones is often accompanied by bile reflux." Using a model such as BERT, the diagnostic conclusions and key evidence of the initial identification results are transformed into semantic vectors. Cosine similarity is calculated between these vectors and the semantic vectors of the target verification items (diagnostic evidence for major diseases of acute abdominal pain) in the verification knowledge base, and the maximum value is taken as the matching score. For example, based on `match_score=max(cosine_similarity(Embedding(result),Embedding(knowledge)))`, where `Embedding(result)` is the semantic vector of the initial identification result, and `Embedding(knowledge)` is the semantic vector of a single target verification item in the knowledge base.

[0062] Furthermore, the matching score is compared with a first preset threshold (e.g., 0.8). If the matching score is higher than the preset threshold, the initial recognition result is directly output as the final recognition result. If the matching score is lower than the first preset threshold, features with a cosine similarity higher than a second preset threshold (e.g., 0.3) between the initial recognition result and the diagnostic criteria for major acute abdominal diseases in the verification knowledge base can be selected. These features, along with the aforementioned text features and image features, are then input into the multimodal decoder for decoding, and the output result is used as the final recognition result.

[0063] In some implementation scenarios, if the matching score is lower than the first preset threshold, the target weights of cross-modal fusion can be adjusted, such as increasing the weights of features related to the core knowledge of the subgraph, and then re-inputting the data into the multimodal decoder for decoding until the matching score is higher than the first preset threshold. For example, if a patient's initial identification result is acute gastritis with a matching score of 0.62, after adjusting the weights and re-decoding, the identification result is updated to acute pancreatitis with a matching score of 0.78, which meets the threshold requirement, and this result is output.

[0064] In some embodiments, the above-described apparatus further performs the following operations: based on text features, image features, and a target sub-map, a multimodal decoder is used to decode and obtain a recognition result for acute abdominal pain. Specifically, in some embodiments, a multimodal decoder is used to perform cross-modal attention calculation on text features and image features to obtain the corresponding association weights between text features, image features, and the target sub-map; the text features and image features are weighted and fused based on the corresponding association weights to obtain fused features; the fused features are decoded to obtain the recognition result for acute abdominal pain.

[0065] As an example, the features of the subgraph related to acute biliary pancreatitis in the target subgraph set have a correlation weight of 0.85 with the text feature of gallstone history and a correlation weight of 0.9 with the image feature of pancreatic enlargement. Based on these correlation weights, the text features and image features are weighted and fused to obtain fused features. Subsequently, the fused features are decoded using a multimodal decoder and a subgraph knowledge verification model to output the recognition results, ensuring that the recognition results can fully utilize the medical knowledge in the target subgraph set, further improving the accuracy and reliability of the recognition.

[0066] In other embodiments, the multimodal decoder may also introduce a residual connection mechanism to residually connect the original text features and image features with the cross-modal fusion features, avoiding information dilution during the fusion process and preserving the core information of the original features. For example, residual fusion feature = cross-modal fusion feature + 0.3 × original text feature + 0.3 × original image feature. By using residual connections to strengthen key information in the original features (such as key symptoms in the text or typical lesions in the image), the stability and accuracy of the recognition results are further improved.

[0067] As described above, this application's embodiments construct a knowledge graph by integrating clinical information and target literature sets. It combines multi-dimensional subgraph partitioning with clinical diagnostic thinking, calculating initial relevance based on intersection ratio and semantic similarity. This is then combined with a subgraph knowledge confidence metric composed of source weights and association strength confidence values ​​to achieve precise, two-dimensional screening of highly relevant and reliable target subgraph sets. Furthermore, multimodal fusion extracts text and image features, and a subgraph knowledge verification model verifies and corrects deviations in the initial identification results. This effectively solves the problems of subgraph partitioning being detached from clinical practice, insufficient reliability in subgraph screening, and lack of verification of identification results. It achieves precise and reliable identification of acute abdominal pain, providing clinical decision support that is both scientific and practical.

[0068] Figure 3 This is an exemplary flowchart illustrating the overall operation of the device for multimodal disease identification according to an embodiment of this application. Figure 3As shown, in steps S301 and S302, the target literature set and clinical information are obtained. In step S303, a knowledge graph for the identification of acute abdomen is constructed based on the clinical information and the target literature set. Based on this knowledge graph, in step S304, the knowledge graph is initially divided into subgraphs, and in step S305, an initial subgraph set is obtained.

[0069] Further, in step S306, the initial relevance between clinical information and each subgraph in the initial subgraph set is calculated. On the other hand, in step S307, a subgraph knowledge confidence metric can also be extracted. And in step S308, the target subgraph set with the highest relevance is extracted from the initial subgraph set by combining the initial relevance and the subgraph knowledge confidence metric. In some embodiments, the pathological path correlation degree between key entity nodes and nodes in each subgraph of each initial subgraph set can also be calculated, for example, in step S309. The initial relevance is calculated by comprehensively considering the corresponding intersection ratio, corresponding semantic similarity, and corresponding pathological path correlation degree. Additionally, by constructing a clinical feature vector in step S310, the fusion weights of the corresponding intersection ratio, corresponding semantic similarity, and corresponding pathological path correlation degree under each initial subgraph set can be determined to calculate the initial relevance.

[0070] In step S311, clinical information and the target sub-map are input into a language encoder for text encoding, and in step S312, text features are obtained. In step S313, image data is acquired, and in step S314, image encoding is performed using an image encoder based on the image data. In step S315, image features are extracted. In step S316, based on the text features and image features, a multimodal decoder can be used for decoding to obtain an initial recognition result. Further, in step S317, a sub-map knowledge verification model is used for verification, so that in step S318, the recognition result for acute abdomen is obtained.

[0071] Furthermore, in some implementation scenarios, a target sub-map can be introduced to obtain the corresponding association weights between text features and image features and the target sub-map; based on the corresponding association weights, the text features and image features are weighted and fused to obtain fused features, which are then decoded to obtain the recognition result for acute abdominal pain identification. For more details on each step, please refer to the above. Figure 2 The descriptions made will not be repeated here.

[0072] Figure 4 An exemplary structural block diagram of an electronic device 400 according to an embodiment of this application is shown. It will be understood that the device implementing the solution of this application may be a single device (e.g., a computing device) or a multifunctional device including various peripheral devices.

[0073] like Figure 4As shown, the electronic device of this application may include a central processing unit (“CPU”) 411, which may be a general-purpose CPU, a dedicated CPU, or other information processing and program execution unit. Furthermore, the electronic device 400 may also include a mass storage device 412 and a read-only memory (“ROM”) 413. The mass storage device 412 may be configured to store various types of data, including various clinical information, knowledge graphs, image data, algorithm data, intermediate results, and various programs required to run the electronic device 400. The ROM 413 may be configured to store power-on self-test (POST) of the electronic device 400, initialization of various functional modules in the system, drivers for the system's basic input / output, and data and instructions required to boot the operating system.

[0074] Optionally, the electronic device 400 may also include other hardware platforms or components, such as the tensor processing unit (“TPU”) 414, graphics processing unit (“GPU”) 415, field-programmable gate array (“FPGA”) 416, and machine learning unit (“MLU”) 417 shown. It is understood that although various hardware platforms or components are shown in the electronic device 400, they are merely exemplary and not limiting, and those skilled in the art can add or remove appropriate hardware as needed. For example, the electronic device 400 may include only a CPU, associated storage devices, and interface devices to implement the operations performed by the multimodal disease recognition device of this application.

[0075] In some embodiments, to facilitate data transmission and interaction with external networks, the electronic device 400 of this application further includes a communication interface 418, through which it can connect to a local area network / wireless local area network (“LAN / WLAN”) 405, and further through the LAN / WLAN to connect to a local server 406 or to the Internet (“Internet”) 407. Alternatively or additionally, the electronic device 400 of this application can also directly connect to the Internet or cellular network via the communication interface 418 based on wireless communication technology, such as wireless communication technology based on 3G (“3G”), 4G (“4G”), or 5G (“5G”). In some application scenarios, the electronic device 400 of this application can also access the server 408 and database 409 of an external network as needed to obtain various known algorithms, data, and modules, and can remotely store various data, such as various data or instructions for presenting clinical information, knowledge graphs, image data, etc.

[0076] Peripherals of the electronic device 400 may include a display device 402, an input device 403, and a data transmission interface 404. In one embodiment, the display device 402 may include, for example, one or more speakers and / or one or more visual displays, configured to provide voice prompts and / or display images and videos for operations performed by the device based on multimodal disease recognition. The input device 403 may include, for example, a keyboard, mouse, microphone, gesture capture camera, and other input buttons or controls, configured to receive audio data input and / or user commands. The data transmission interface 404 may include, for example, a serial interface, parallel interface, or Universal Serial Bus interface (“USB”), Small Computer System Interface (“SCSI”), Serial ATA, FireWire (“FireWire”), PCI Express, and High Definition Multimedia Interface (“HDMI”), configured for data transmission and interaction with other devices or systems. According to the present application, the data transmission interface 404 can receive acquired image data from CT or MRI equipment and transmit image data or various other types of data or results to the electronic device 400.

[0077] The CPU 411, mass storage 412, ROM 413, TPU 414, GPU 415, FPGA 416, MLU 417, and communication interface 418 of the electronic device 400 of this application can be interconnected via bus 419, and can interact with peripheral devices through this bus. In one embodiment, the CPU 411 can control other hardware components and peripheral devices in the electronic device 400 through bus 419.

[0078] The above combination Figure 4 An electronic device that can be used to execute this application is described. It should be understood that the device structure or architecture described herein is merely exemplary, and the implementation method and implementation entity of this application are not limited thereto, but can be changed without departing from the spirit of this application.

[0079] Based on the foregoing description in conjunction with the accompanying drawings, those skilled in the art will understand that the embodiments of this application can also be implemented by software programs. Therefore, this application also provides a computer-readable storage medium storing computer-readable instructions for multimodal disease identification. When executed by one or more processors, these computer-readable instructions can be used to implement the embodiments of this application in conjunction with the accompanying drawings. Figures 1-3 The operation performed by the device for multimodal disease identification described.

[0080] It should be noted that although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0081] It should be understood that when the terms "first," "second," "third," and "fourth," etc., are used in the claims, specification, and drawings of this application, they are used only to distinguish different objects and not to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this application indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0082] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0083] While numerous embodiments of this application have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will arise for those skilled in the art without departing from the spirit and intent of this application. It should be understood that various alternatives to the embodiments of this application described herein may be employed in the practice of this application. The appended claims are intended to define the scope of protection of this application and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A device for multimodal disease identification, comprising: processor; as well as A memory storing computer instructions for multimodal disease identification, which, when executed by a processor, cause the following operations to be performed: Constructing a knowledge graph for the identification of acute abdomen based on clinical information and target literature set; The knowledge graph is divided into subgraphs to obtain an initial subgraph set; Calculate the initial correlation between the clinical information and each subplot in the initial subplot set; By combining the initial relevance and the subgraph knowledge confidence metric, the target subgraph set with the highest relevance is extracted from the initial subgraph set; The clinical information and the target sub-map are input into a language encoder for text encoding to obtain text features; Based on image data, an image encoder is used to encode the images and extract image features. Based on the text features and image features, a multimodal decoder and a subgraph knowledge verification model are used for decoding to obtain the recognition result for acute abdomen. The device further performs the following operations to obtain the identification result of acute abdomen: Based on the text features and the image features, a multimodal decoder is used to decode and obtain the initial recognition result for acute abdominal pain. The initial recognition result is input into the subgraph knowledge verification model for verification to obtain a matching score; In response to the matching score being higher than a first preset threshold, an initial recognition result is output; In response to the matching score being lower than the first preset threshold, a feature quantity corresponding to the similarity to the target verification item being higher than the second preset threshold is selected from the initial identification results; Based on the text features, the image features, and the feature quantity, a multimodal decoder is used for secondary decoding to obtain the recognition result, wherein the first preset threshold is greater than the second preset threshold.

2. The apparatus of claim 1, wherein the apparatus further performs the following operations: Different partitioning hyperparameters are set to split the graph into multiple initial subgraphs; Initial subgraphs whose difference values ​​are less than a difference threshold are merged to optimize the initial subgraph set.

3. The apparatus of claim 1, wherein the apparatus further performs the following operations to calculate the initial correlation: Based on the aforementioned clinical information, key information is extracted to construct key entity nodes; Calculate the intersection ratio between the key entity node and the nodes of each subgraph in each initial subgraph set; Calculate the semantic similarity between the clinical information and the coherent text of each subgraph in each initial subgraph set; The initial correlation is calculated based on the corresponding intersection ratio and semantic similarity under each initial sub-graphet.

4. The apparatus of claim 1, wherein the apparatus further performs the following operations to obtain the subgraph knowledge confidence metric: Based on the knowledge graph, source weights are set for each subgraph in the initial subgraph set; Calculate the confidence value of the association strength between nodes within each subgraph; The source weights and the confidence values ​​are multiplied to obtain the knowledge confidence weights of each subgraph, thereby obtaining the knowledge confidence metric of the subgraph.

5. The apparatus of claim 4, wherein the apparatus further performs the following operations to extract the target sub-map set with the highest relevance: Calculate the weighted value of the initial relevance and the knowledge confidence weight of each subgraph to obtain the comprehensive relevance score; Based on the comprehensive relevance score, the target sub-map set with the highest relevance is extracted from the initial sub-map set.

6. The apparatus of claim 3, wherein the apparatus further performs the following operations: Calculate the pathological path correlation degree between the key entity node and the node of each subgraph in each initial subgraph set; The initial correlation is calculated based on the corresponding intersection ratio, semantic similarity, and pathological path correlation degree under each initial sub-map set.

7. The apparatus of claim 6, wherein the apparatus further performs the following operations: Construct a clinical feature vector, which includes one or more of the following: symptom complexity, number of complications, and length of medical history; Based on the clinical feature vectors, the fusion weights of the corresponding intersection ratio, corresponding semantic similarity, and corresponding pathological path correlation under each initial sub-map set are determined to calculate the initial correlation.

8. The apparatus of claim 1, wherein the apparatus further performs the following operations: The key region mask is marked in the image data according to the target sub-atlas; Based on the image data and the key region mask, an image encoder is used to encode the image and extract image features.

9. A computer-readable storage medium having stored thereon computer program instructions for multimodal disease identification, which, when executed by one or more processors, cause the operation performed by the apparatus according to any one of claims 1-8 to be implemented.

Images