A method and system for intelligent processing of medical knowledge in multi-modal communication

Abstract

The present application relates to the technical field of intelligent processing of medical knowledge, and particularly relates to a medical knowledge intelligent processing method and system based on multi-modal communication, comprising the following steps: obtaining voice description information, text record information, medical image information and physiological monitoring data, and processing them into medical data units of multiple time windows; extracting medical features in each time window and mapping them to medical knowledge nodes in a medical knowledge intermediate layer to generate semantic activation states; performing consistency analysis based on semantic constraint relationships and generating structured conflict information; adjusting the credibility weight based on the disease hypothesis state and eliminating the structured conflict information; determining the missing medical evidence type, generating information supplement demand and feeding back the execution iteration processing. The present application divides the multi-modal medical information into time windows and maps them to the medical knowledge intermediate layer to form semantic activation states, performs cross-time consistency analysis, structured conflict elimination and missing medical evidence supplement processing based on semantic constraint relationships.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical knowledge processing technology, and in particular to a multimodal communication method and system for intelligent medical knowledge processing. Background Technology

[0002] With the development of smart healthcare, telemedicine, and continuous health monitoring technologies, medical information processing is gradually evolving from single-data type analysis to multimodal fusion analysis. Current medical information processing technologies typically collect patient voice descriptions, medical records, medical imaging data, and physiological monitoring data, and extract medical features through natural language processing, medical image recognition, and physiological signal analysis to achieve disease-assisted analysis, clinical status assessment, and medical information communication. In continuous healthcare scenarios, patient states typically change dynamically over time, and multimodal medical information generated at different time points exhibits significant temporal correlations. Therefore, how to uniformly organize, correlate, and reason about multimodal medical information generated over continuous time has become an important research direction in the field of intelligent medical information processing.

[0003] Traditional multimodal medical information processing mostly adopts a single-data independent reasoning approach. Due to the lack of cross-time continuous semantic constraints and conflict feedback mechanisms, medical semantics become inconsistent in the time dimension. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides a multimodal communication-based intelligent medical knowledge processing method and system, aiming to improve the problem that traditional multimodal medical information processing mostly adopts a single-data independent reasoning method, which easily leads to inconsistencies in medical semantics in the time dimension.

[0005] In a first aspect, the present invention provides the following technical solution: a method for intelligent processing of medical knowledge through multimodal communication, comprising the following steps:

[0006] S1. Acquire multimodal medical information and process it into medical data units with multiple time windows, wherein the multimodal medical information includes voice description information, text record information, medical image information and physiological monitoring data;

[0007] S2. Extract features from each modality of medical information within each time window, map the extracted medical features to the corresponding medical knowledge nodes in the preset medical knowledge intermediary layer, and generate semantic activation states corresponding to the medical knowledge nodes; wherein, the medical knowledge intermediary layer is preset with medical knowledge nodes and semantic constraint relationships between medical knowledge nodes.

[0008] S3. Based on the semantic constraint relationship, perform consistency analysis on the semantic activation state across time windows. When the semantic activation state does not satisfy the corresponding semantic constraint relationship, generate structured conflict information.

[0009] S4. Determine the disease hypothesis state associated with the medical knowledge nodes involved in the structured conflict information, adjust the credibility weights corresponding to each modality of medical information based on the disease hypothesis state, and resolve the structured conflict information based on the semantic constraint relationship.

[0010] S5. When it is detected that there is still unresolved structured conflict information after processing in step S4, the missing medical evidence type is determined based on the structured conflict information and the semantic constraint relationship, an information supplementation requirement indicating the missing medical evidence type is generated, and the supplemented multimodal medical information is input to steps S1 to S4 for processing.

[0011] By adopting the above technical solution, multimodal medical information is divided into time windows and mapped to the medical knowledge intermediary layer to form a semantic activation state. Then, based on semantic constraint relationships, cross-time consistency analysis, structured conflict resolution, and closed-loop supplementation of missing medical evidence are performed. This improves the problem that traditional multimodal medical information processing mostly adopts a single data independent reasoning method, which lacks cross-time continuous semantic constraints and conflict feedback mechanisms, resulting in inconsistencies in medical semantics in the time dimension.

[0012] The present invention has the following beneficial effects:

[0013] 1. In this invention, multimodal medical information is divided into time windows and mapped to the medical knowledge intermediary layer to form a semantic activation state. Then, based on semantic constraint relationships, cross-time consistency analysis, structured conflict resolution, and closed-loop supplementation of missing medical evidence are performed. This improves the problem that traditional multimodal medical information processing mostly adopts a single data independent reasoning method. Due to the lack of cross-time continuous semantic constraints and conflict feedback mechanisms, medical semantics are inconsistent in the time dimension.

[0014] 2. In this invention, by performing constraint matching on the semantic activation state across time windows and generating structured conflict information, and then dynamically adjusting the credibility weights corresponding to each modality of medical information based on the disease hypothesis state, the problem of losing key medical evidence information is improved by the fact that traditional multimodal medical information processing mostly adopts the method of directly discarding conflict information or static fusion, and lacks a conflict resolution mechanism oriented towards the medical knowledge relationship.

[0015] 3. In this invention, the missing medical evidence type is determined based on the unresolved structured conflict information and semantic constraint relationship, and information supplementation requirements are generated. Then, the supplemented multimodal medical information is returned to the previous processing flow for reprocessing. This improves the problem that traditional medical information processing mostly adopts a passive data update method, which lacks an iterative supplementation mechanism for missing medical evidence, resulting in incomplete medical data. Attached Figure Description

[0016] Figure 1 This is a flowchart of a multimodal communication-based intelligent medical knowledge processing method proposed in this invention;

[0017] Figure 2 This is a schematic diagram of the multimodal medical information time window construction and processing flow in this invention;

[0018] Figure 3 This is a schematic diagram of the process of medical feature extraction, medical knowledge node mapping and semantic activation state generation in this invention;

[0019] Figure 4 This is a schematic diagram of the cross-time semantic consistency analysis and structured conflict information generation process in this invention;

[0020] Figure 5 This is a schematic diagram of the credibility weight adjustment and structured conflict resolution process based on the disease hypothesis state in this invention;

[0021] Figure 6 This is a schematic diagram of the process for determining the type of missing medical evidence and generating information supplementation requirements in this invention;

[0022] Figure 7 This is an architecture diagram of a multimodal communication-based intelligent medical knowledge processing system proposed in this invention. Detailed Implementation

[0023] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Example 1: In the first embodiment of the present invention, the present invention provides a multimodal communication-based intelligent processing method for medical knowledge, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0025] S1. Acquire multimodal medical information and process it into medical data units with multiple time windows. The multimodal medical information includes voice description information, text record information, medical image information, and physiological monitoring data.

[0026] Furthermore, in S1, the steps of acquiring multimodal medical information and processing it into medical data units for multiple time windows include:

[0027] Acquire voice description information, text recording information, medical imaging information, and physiological monitoring data;

[0028] Time-stamped processing is performed on voice description information, text record information, medical imaging information, and physiological monitoring data to form time-stamped multimodal medical information;

[0029] The time-stamped multimodal medical information is arranged in chronological order according to the time stamp to generate a multimodal medical information sequence;

[0030] The multimodal medical information sequence is divided into time intervals based on time order, and the multimodal medical information in each time interval is aggregated to generate medical data units corresponding to multiple time windows.

[0031] Specifically, in continuous medical information acquisition scenarios, by uniformly aligning and windowing voice descriptions, text records, medical imaging data, and physiological monitoring data, the temporal organization of heterogeneous medical information over a continuous time period is achieved, providing structured input for subsequent medical feature extraction and cross-temporal semantic consistency analysis. In practice, the continuous input sequences of voice descriptions, text records, medical imaging data, and physiological monitoring data are first acquired, and each input piece is assigned a collection time identifier. Let the first... The collection time corresponding to each input information is ,in The data is obtained through the timestamp recorded by the system upon reception; in this scheme, the timestamp precision is preferably at the millisecond level. Subsequently, a multimodal medical information sequence is constructed in ascending chronological order. ;in Indicates the first One medical information record, This represents the total number of information acquired within a continuous acquisition period; to implement time window division, the window length is set to [value]. The window sliding step size is ,in Indicates the time span covered by a single time window. This indicates the start time interval between adjacent windows. In this scheme, the preferred window length is... The preferred window sliding step size is... Based on the window start time Construct the first The time interval of each time window And based on the judgment conditions Medical information that meets the time-based attribution criteria will be assigned to the corresponding time window; for the first... Each time window, its medical data unit is represented as: ;in This represents the aggregated multimodal medical data set within a time window. To ensure a consistent representation of different modal information within the same time window, a modality classification operation is further performed on the multimodal medical information within the time window to form a subset of speech descriptions. Text record subset Medical imaging subset and a subset of physiological monitoring data Satisfying the relationship In this scheme, when specific modality data is missing within a certain time window, the corresponding modality null identifier is retained to maintain the consistency of the time window data structure. After the above processing, multiple medical data unit sequences corresponding to the time windows are output. ;in This represents the total number of time windows obtained from the division; this medical data unit sequence, together with the initial confidence weights generated based on modal integrity rate, time synchronization rate and sampling stability, serves as the input for the next stage of medical feature extraction processing.

[0032] like Figure 1 and Figure 3 As shown, S2 extracts features from each modality of medical information within each time window, maps the extracted medical features to the corresponding medical knowledge nodes in the preset medical knowledge intermediary layer, and generates semantic activation states corresponding to the medical knowledge nodes; wherein, the medical knowledge intermediary layer presets medical knowledge nodes and semantic constraint relationships between medical knowledge nodes.

[0033] Furthermore, in S2, the steps for feature extraction of each modality of medical information within each time window include:

[0034] Perform medical semantic analysis on voice description information and text record information to extract medical entity words and medical terminology;

[0035] Medical image features are extracted from medical image information to generate image structure feature representations;

[0036] Time series analysis was performed on physiological monitoring data to extract characteristic sequences of physiological index changes.

[0037] Specifically, after forming medical data units corresponding to multiple time windows, heterogeneous feature parsing processing is performed on the voice description information, text record information, medical image information, and physiological monitoring data in each time window to generate a unified set of medical features, which is then used as the input for mapping medical knowledge nodes. In specific implementation, for the first... Medical data units corresponding to each time window First, extract a subset of speech description information. Subset of text record information The speech description information is processed to form a text sequence. ;in This indicates the number of text entries after speech-to-text transcription, followed by the text sequence. Subset of text record information Merging to form a medical text collection Each text in the medical text collection undergoes word segmentation and semantic parsing to obtain a term set. ;in This represents the number of terms obtained from parsing; the weight value of medical entities is calculated based on word frequency statistics and medical terminology matching rules, using the following formula: ;in Indicates the first The weight value of each term. Indicates the first In this scheme, the frequency of occurrence of a term in a medical text set is retained, satisfying the following criteria: The terms are used as medical entity words, and the output is a set of medical entity words. For subsets of medical image information The image is divided into several regions, and the grayscale texture features of each region are extracted to construct an image structure feature vector. ;in The dimension of the image features is represented by the preferred value in this scheme. By calculating regional gradient statistics to form feature components in each dimension, the image structure feature representation is output; for a subset of physiological monitoring data Assuming continuous sampling forms a physiological sequence ;in Indicates the number of sampling points. Indicates the first The rate of change sequence is calculated from the differences between adjacent samples using the following formula: ;in Indicates the first The rate of change at each sampling point In this scheme, the sampling time interval is indicated. Preferred Further calculation of the statistical characteristics of the rate of change forms a sequence of physiological indicator changes. Ultimately, the set of medical entity terms will be... Image structure feature vector and physiological indicator change characteristic sequence To organize in a unified manner and form a set of medical characteristics within a time window. ;in, Indicates the first The comprehensive medical feature output results corresponding to each time window are used as input for subsequent medical knowledge intermediary layer mapping processing. The medical knowledge intermediary layer is generated by encoding a structured medical knowledge graph. The medical knowledge nodes are encoded and stored using a triplet structure of medical concept identifiers, semantic attribute vectors, and associated constraint edges. This is used to retrieve the corresponding medical knowledge nodes in the medical knowledge intermediary layer and further generate semantic activation states, thereby supporting cross-time window semantic consistency analysis.

[0038] Furthermore, in S2, the step of mapping the extracted medical features to the corresponding medical knowledge nodes in the preset medical knowledge intermediary layer includes:

[0039] Retrieve candidate medical knowledge nodes in the medical knowledge intermediary layer based on medical features;

[0040] Perform semantic matching processing on medical features and candidate medical knowledge nodes to determine the target medical knowledge node;

[0041] Establish a corresponding mapping relationship between medical features and target medical knowledge nodes.

[0042] Specifically, the process of mapping medical features to corresponding medical knowledge nodes in the medical knowledge intermediary layer first involves mapping the medical feature sets M of each time window. k Unfold to form a medical feature sequence to be mapped, denoted as: F k ={f (k,1) ,f (k,2) ,…,f (k,nk)}; where f (k,i) Indicates the first Each medical feature originates from medical entity words obtained through parsing speech description information, medical terminology expressions obtained through parsing text record information, image structure feature representations generated from medical imaging information, and physiological indicator change feature sequences formed from physiological monitoring data; the set of medical knowledge nodes in the medical knowledge intermediary layer is denoted as... ;in Indicates the first Each medical knowledge node is formed by importing a pre-constructed disease knowledge graph and establishing semantic constraints between nodes based on clinical diagnosis and treatment knowledge; for sequence F k any medical feature f in (k,i)In the medical knowledge intermediary layer, candidate node retrieval is performed. Candidate matching values ​​are generated by calculating the relevance between medical feature vectors and the semantic vectors of each medical knowledge node. The relevance calculation method is expressed as follows: ;in Indicates medical characteristics f (k,i) The feature vectors are obtained by encoding in the preceding feature extraction stage. In this scheme, the preferred vector dimension is 256. Representing medical knowledge nodes The semantic representation vector is obtained by encoding the knowledge graph in the preset medical knowledge intermediary layer. In this scheme, the vector dimension is preferably also set to 256. Indicates medical characteristics Medical knowledge nodes The semantic matching score is calculated; then, the maximum value is filtered out from all matching scores to determine the target medical knowledge node. The filtering rule is expressed as follows: ;in Indications and medical characteristics The corresponding target medical knowledge node; a mapping relationship is established when the maximum matching score meets the preset matching threshold condition, and the judgment rule is expressed as follows: ;in The semantic matching threshold is preferably set to 0.78 in this scheme; after the threshold condition is met, a mapping set between medical features and target medical knowledge nodes is generated, denoted as... After the mapping set is generated, it serves as the input data for the next stage of semantic activation state generation. By performing activation updates on the state identifiers of the target medical knowledge nodes, a propagable semantic activation state across time windows is formed. This state is then used for subsequent cross-time consistency analysis and structured conflict identification and processing, thereby aligning multimodal medical features to a unified medical knowledge semantic space.

[0043] Furthermore, in S2, the steps for generating the semantic activation state corresponding to the medical knowledge node include:

[0044] Set node status identifiers for medical knowledge nodes;

[0045] The node status identifier is updated based on the mapping relationship between medical features and medical knowledge nodes;

[0046] Semantic activation states are generated based on the update results of node state identifiers.

[0047] Specifically, for the semantic activation state generation process, the first step is to receive the set of mapping relationships output from the medical feature mapping stage as input. This set of mapping relationships is represented as... ;in Indicates the first The medical features are derived from voice descriptions, text records, medical imaging data, and physiological monitoring data through feature extraction. Indications and medical characteristics Matching medical knowledge nodes; establishing a set of node status identifiers for all medical knowledge nodes in the medical knowledge intermediary layer, represented as... ;in Indicates the first The state value of each medical knowledge node is initially uniformly assigned a value. In this scheme, the initial state value is preferably set to zero to represent an inactive state. Subsequently, the node state identifier is updated according to the mapping relationship set. For medical knowledge nodes with medical feature mapping inputs, activation accumulation processing is performed. When there are no medical features mapped to the target medical knowledge node in the current time window, only time decay updates are retained. The update rule is expressed as follows: ;in Indicates the number of times within the current time window The state value of each medical knowledge node. This indicates the status value of the corresponding medical knowledge node within the previous time window. This represents the time decay coefficient, which is preferably set to 0.85 in this scheme. It is used to characterize the proportion of state inheritance across time windows. Indicates medical characteristics The feature contribution weights are generated by normalizing the semantic matching scores output from the medical feature matching stage. In this scheme, the preferred weights are those that satisfy the following conditions: After updating the node state value, the update result is compared with the preset activation threshold to generate a semantic activation state. The determination rule is expressed as follows: ;in Indicates the first The semantic activation status corresponds to each medical knowledge node, with a value of 1 indicating activation and a value of 0 indicating inactivation. The activation threshold is represented, and in this scheme, it is preferably set to 0.6; the semantic activation states of all medical knowledge nodes together constitute the semantic activation state set of the current time window, represented as... After the semantic activation state set is generated, it serves as input data for the cross-time consistency analysis stage. It is used to form a cross-time alignment sequence with other time windows and to perform node-by-node constraint matching based on the semantic constraint relationship between medical knowledge nodes in order to identify medical state conflicts between different time windows and generate structured conflict information.

[0048] like Figure 1 and Figure 4 As shown, S3, based on semantic constraint relationships, performs consistency analysis on semantic activation states across time windows. When a semantic activation state does not satisfy the corresponding semantic constraint relationship, structured conflict information is generated.

[0049] Furthermore, in S3, the steps for performing consistency analysis on semantic activation states across time windows include:

[0050] Extract the semantic activation state sets corresponding to different time windows;

[0051] The semantic activation state set is time-aligned according to the time window order to form a cross-time alignment sequence;

[0052] Based on the semantic constraint relationship between medical knowledge nodes in the medical knowledge mediation layer, node-by-node constraint matching is performed on the semantic activation state in the cross-time alignment sequence.

[0053] The satisfaction status between semantic activation states is determined based on the constraint relationship matching results.

[0054] Specifically, for the cross-time window semantic activation state consistency analysis process, it first receives multiple time window semantic activation state sets output from the semantic activation state generation stage as input, and represents the semantic activation states corresponding to all time windows as follows: ;in Indicates the first The set of semantic activation states corresponding to each time window This represents the total number of time windows. In this scheme, the preferred time window length is set to a 5-minute interval during continuous monitoring. The elements in the semantic activation state set are represented as... ;in Indicates the first Within the first time window The activation state of each medical knowledge node is obtained by comparing the node's state value with the activation threshold during the semantic activation state generation stage. First, time alignment processing is performed on the semantic activation state sets for each time window, forming a cross-time aligned sequence according to the time identifier, represented as follows: ;in Indicates the first The state evolution sequence of each medical knowledge node over the entire time window; subsequently, the set of semantic constraint relationships between medical knowledge nodes is extracted from the medical knowledge mediation layer, represented as... ;in Representing medical knowledge nodes Medical knowledge nodes The semantic constraints between these relationships, constructed based on a medical knowledge graph, are used to characterize disease state evolution, symptom co-occurrence, and physiological indicator correlations. For any semantic constraint relationship... For time-aligned sequences, node-by-node constraint matching is performed, and the consistency value of state transitions between adjacent time windows is used for determination. The consistency value is calculated as follows: ;in Indicates the first The time window to the The constraint consistency value of the medical knowledge node status across time windows; when the average consistency value of all time windows meets the threshold condition, the semantic constraint relationship is determined to be satisfied, and the determination rule is expressed as follows: ;in The consistency threshold is preferably set to 0.75 in this scheme. When the mean is lower than this threshold, a cross-temporal semantic inconsistency is marked between the corresponding medical knowledge nodes, and a constraint mismatch identifier is generated, represented as... All constraint mismatch identifiers together form the consistency analysis result set, represented as: The consistency analysis results are output to the next processing stage to locate the mismatched combinations of medical knowledge nodes and their corresponding time windows, further generate structured conflict information, and serve as the data basis for subsequent disease hypothesis state derivation and credibility weight adjustment.

[0055] Furthermore, in S3, the steps for generating structured conflict information include:

[0056] Identify medical knowledge nodes that do not satisfy semantic constraints in the consistency analysis;

[0057] Determine the time window combination relationship corresponding to medical knowledge nodes that do not satisfy semantic constraints;

[0058] Generate conflict type identifiers based on semantic constraint relationships;

[0059] Structured conflict information is generated based on medical knowledge nodes, time window combinations, and conflict type identifiers.

[0060] Specifically, for the structured conflict information generation process, the consistency analysis result set output from the consistency analysis phase is first received as input, and all constraint mismatch results are represented as... ;in Indicates the first The time window to the Medical knowledge nodes between time windows Medical knowledge nodes The semantic constraint matching result is represented by a value of 1, indicating a constraint mismatch, and a value of 0, indicating that the semantic constraint relationship is satisfied. The medical knowledge node set is represented as follows. First, a mismatch screening process is performed on the consistency analysis result set to extract all results that meet the criteria. The medical knowledge node pairs form a set of mismatched nodes, represented as Subsequently, based on the set of mismatched nodes, the time window combination relationship is extracted, and the start and end time windows of the mismatch are used to form a time window combination identifier, represented as... ;in Representing medical knowledge nodes Medical knowledge nodes The corresponding time window combination relationship; further, the semantic constraint relationship set in the medical knowledge mediation layer is invoked to perform conflict category identification for each group of mismatched nodes. The conflict type determination function is expressed as follows: ;in Representing medical knowledge nodes Medical knowledge nodes semantic constraint relations, Indicates the first Medical knowledge nodes within a time window semantic activation state, Indicates the first Medical knowledge nodes within a time window semantic activation state, The conflict type mapping rules are defined based on a pre-defined logical lookup table of semantic constraints. In this scheme, they are preferably categorized into three types: state missing, state reversal, and state transition, with corresponding values ​​of 1, 2, and 3, respectively. After determining the conflict type, the conflict intensity value is calculated, expressed as follows: ;in The constraint consistency values ​​between time window t and time window t+1, calculated during the consistency analysis phase. This represents the conflict intensity value, which is used in this scheme when... A conflict is identified as a valid conflict at that time; finally, the medical knowledge nodes, time window combinations, conflict type identifiers, and conflict intensity values ​​are combined to generate structured conflict information, represented as follows: Structured conflict information serves as input data for the subsequent disease hypothesis state derivation stage. By analyzing the medical knowledge nodes and conflict types involved in the conflict, a candidate set of disease hypothesis states is established, which is then used for credibility weight adjustment and conflict resolution.

[0061] like Figure 1 and Figure 5 As shown, S4: Determine the disease hypothesis state associated with the medical knowledge nodes involved in the structured conflict information, adjust the credibility weights corresponding to each modality of medical information based on the disease hypothesis state, and resolve the structured conflict information based on semantic constraint relationships.

[0062] Furthermore, in S4, the steps for adjusting the credibility weights corresponding to each modality of medical information based on the disease hypothesis state include:

[0063] The disease hypothesis state is determined based on the medical knowledge nodes involved in the structured conflict information;

[0064] Based on the disease hypothesis state, extract the medical evidence types and medical constraint relationships corresponding to the disease hypothesis state from the medical knowledge intermediary layer;

[0065] Based on the types of medical evidence, determine the modal evidence association relationships corresponding to voice description information, text record information, medical imaging information, and physiological monitoring data;

[0066] The initial credibility weights of voice description information, text record information, medical image information, and physiological monitoring data are redistributed based on medical constraints.

[0067] Specifically, in this implementation, structured conflict information is used as input data. This structured conflict information is output from the consistency analysis stage and includes a set of conflicting medical knowledge nodes, a combination relationship of conflict time windows, and a conflict type identifier. First, a disease candidate set is constructed based on the set of medical knowledge nodes in the structured conflict information, and then the association score between each disease candidate and the conflicting medical knowledge nodes is calculated. The association score is expressed using the following formula. ;in, Represents the association score of disease candidates. This indicates the number of medical knowledge nodes contained in the structured conflict information. Indicates the first Medical knowledge nodes and disease candidates The strength of the association between them is derived from the disease association knowledge graph in the medical knowledge intermediary layer. The importance coefficient of medical knowledge nodes is preferred to be within the range specified in this scheme. Based on the disease candidate with the highest association score, the disease hypothesis state used to explain the structured conflict information is determined. Subsequently, based on the disease hypothesis state, the corresponding set of medical evidence types is retrieved in the medical knowledge mediation layer, and a modal evidence association matrix is ​​established. The elements of the association matrix are calculated using the following relational formula. ;in, Indicates the first Modal medical information and the first The correlation weights between medical evidence, This represents the association strength value recorded in the knowledge intermediary layer, where l represents the summation index of medical evidence categories. This represents the total number of medical evidence types. This value is obtained through statistical relationships in a knowledge graph, and is preferably set to [value missing] in this scheme. Then, the initial credibility weights generated during the initial processing stage are read from the voice description information, text record information, medical imaging information, and physiological monitoring data. These weights are then combined with the modal evidence association matrix to perform a redistribution calculation. The updated credibility weights are obtained as follows: ;in, Indicates the updated number Modal credibility weights This represents the initial credibility weight, derived from the data quality assessment results during the multimodal medical information collection phase. Indicates the first The constraint contribution coefficient of medical evidence is preferably selected from the following range in this scheme: , The value represents the number of modal medical information categories, which is 4 in this scheme; h represents the modal category summation index; the initial confidence weight is calculated based on the modal integrity rate, time synchronization rate, and sampling stability normalization. ;in Modal integrity rate, For time synchronization rate, To ensure sampling stability, an updated set of credibility weights is obtained after redistribution. This result serves as the input for the structured conflict information resolution stage. Based on the updated credibility weights, the support scores of each conflicting medical knowledge node are recalculated, retaining the activation states of nodes with higher support scores and suppressing the activation states of nodes with support scores below a preset threshold. Subsequently, semantic constraint relationship matching is re-executed. When the conflict intensity value is lower than the conflict resolution threshold, the conflict resolution threshold is denoted as ζ, which is preferably set to 0.15 in this scheme. The corresponding structured conflict information is marked as resolved, thus providing a weight basis for subsequent structured conflict information resolution based on semantic constraint relationships. This enables the disease hypothesis state to participate in cross-modal evidence credibility adjustment, realizing dynamic credibility reconstruction of multimodal medical information in conflict scenarios.

[0068] like Figure 1 and Figure 6 As shown, S5, when unresolved structured conflict information is detected after processing in step S4, the missing medical evidence type is determined based on the relationship between the structured conflict information and semantic constraints, information supplementation requirements indicating the missing medical evidence type are generated, and the supplemented multimodal medical information is input into steps S1 to S4 for processing.

[0069] Furthermore, in S5, the steps for generating supplementary information requirements indicating the type of missing medical evidence include:

[0070] Parse the unresolved structured conflict information and extract the corresponding medical knowledge nodes and semantic constraint relationships;

[0071] Based on medical knowledge nodes and semantic constraint relationships, reverse constraint deduction is performed on medical knowledge nodes that do not meet the constraint conditions to determine the type of missing medical evidence.

[0072] The types of missing medical evidence are classified and processed according to the evidence classification system of the medical knowledge intermediary layer;

[0073] Based on the classification results, information supplementation requirements are generated corresponding to the types of missing medical evidence.

[0074] Specifically, unresolved structured conflict information is used as input data. This structured conflict information is output from the structured conflict resolution stage and includes a set of conflicting medical knowledge nodes, a conflict time window sequence, and a conflict type identifier. Simultaneously, a pre-defined set of semantic constraint relationships in the medical knowledge intermediary layer is read. First, the unresolved structured conflict information is parsed to extract the conflicting medical knowledge nodes and their corresponding semantic constraint relationships, and a conflict constraint association matrix is ​​constructed. The elements of the association matrix are determined using the following formula. ;in, Indicates the first The first medical knowledge node and the first The association identifier between semantic constraint relationships, This represents the association determination function, which takes a value of 1 when the medical knowledge node satisfies the semantic constraint relationship scope, and a value of 0 otherwise. The conflicting medical knowledge nodes are derived from the results of structured conflict information parsing. This represents the first layer in the medical knowledge intermediary layer. The semantic constraint relationships are defined; the evidence requirement rules are pre-stored in the medical knowledge intermediary layer and are constructed from the statistical relationships between medical knowledge nodes and supplementary evidence in historical cases. Then, by retrieving the evidence requirement rules corresponding to missing premise nodes in the semantic constraint relationships, a reverse backtracking is performed to calculate the support score for conflict resolution for each candidate medical evidence. The support score is obtained in the following manner. ;in, Indicates the type of candidate medical evidence Support rating, Indicates the number of conflicting medical knowledge nodes. Indicates the number of semantic constraint relations. Indicates the type of candidate medical evidence for the first The first medical knowledge node and the first The contribution value of the semantic constraint relationship is used to supplement the evidence supplementation mapping relationship. This is established by pre-statistically analyzing the contribution ratio of candidate medical evidence in historical cases to the satisfaction of the corresponding semantic constraint relationship. In this scheme, the preferred value range is [value range missing]. Candidate medical evidence types whose scores meet preset screening criteria are identified as missing medical evidence types. In this scheme, the candidate medical evidence type with the highest support score is preferentially selected, and the contribution value for supplementation is obtained from the preset evidence supplementation rule table in the medical knowledge intermediary layer. Then, the missing medical evidence types are classified and coded according to the evidence classification system in the medical knowledge intermediary layer. The classification result is expressed by the following expression. ;in, This indicates the classification number corresponding to the type of missing medical evidence. This indicates that the candidate medical evidence type belongs to the first category. The matching strength of evidence categories is obtained from the evidence classification mapping relationship in the knowledge mediation layer. In this scheme, the evidence categories are preferably divided into four categories: voice supplementation, text supplementation, image supplementation, and physiological monitoring supplementation. After the classification is completed, an information supplementation requirement description vector is generated based on the classification results. The generation rules are determined as follows. ;in, This indicates the need for supplementary information corresponding to the missing medical evidence in category e. This indicates the priority of supplementation, which is determined based on the ranking of support scores. The corresponding time window positioning information is derived from the conflict time window sequence in the structured conflict information. The generated information supplementation requirements are used as output results to guide the subsequent collection of corresponding types of multimodal medical information. The supplemented multimodal medical information re-enters the preprocessing flow to perform time window division, medical feature extraction, semantic activation analysis and structured conflict resolution, thereby forming a closed-loop iterative data completion path.

[0075] Example 2: In the Intensive Care Unit (ICU) of a tertiary hospital, a critically ill postoperative patient was under continuous monitoring. Their medical information included physiological monitoring data from bedside monitors, nursing records, doctor's rounds audio recordings, and bedside imaging results. Due to differences in acquisition time, expression methods, and reliability among different modalities of medical information, inconsistencies across time periods are prone to occur. For example, early audio recordings might indicate chest tightness, while concurrent monitoring indicators showed no obvious abnormalities, but subsequent imaging revealed abnormal signs in the chest cavity. This could lead to difficulties in timely identification of early signs of latent complications. To address these issues, a multimodal communication-based intelligent medical knowledge processing system provided by this invention was adopted, the structure of which is as follows: Figure 7 As shown. The specific implementation process of this system is as follows:

[0076] First, the time window construction module acquires voice description information, text record information, medical image information and physiological monitoring data generated during the continuous monitoring of patients. Then, according to the preset time sliding window, the continuous data stream is divided into medical data units corresponding to multiple time windows, thereby establishing a time synchronization communication relationship between multimodal medical information, ensuring that subsequent analysis has a unified time granularity, and improving the ability to continuously organize medical information across time.

[0077] Then, the semantic activation module extracts features from the multimodal information within each time window. Specifically, it extracts medical entities from speech and text records, trends from physiological monitoring data, and lesion structure features from medical images. The extracted medical features are then mapped to corresponding medical knowledge nodes in a pre-defined medical knowledge intermediary layer, generating semantic activation states. Through this process, previously heterogeneous and isolated medical data are uniformly anchored to the same medical knowledge space, achieving a transformation from low-level data communication to high-level semantic communication. This allows different modalities to align and cross-verify within a unified knowledge framework, thereby improving the consistency of semantic understanding of disease conditions.

[0078] Next, the conflict resolution module performs consistency analysis on the semantic activation states across time windows based on the semantic constraints in the medical knowledge intermediary layer. When the system detects a temporal logical mismatch between early symptom complaints and subsequent imaging signs and physiological indicators, it automatically generates structured conflict information, recording the medical knowledge nodes involved, the time window location, and the conflict type. Through this step, the originally scattered abnormal information is transformed into a traceable structured medical evidence chain, thereby enabling rapid identification of hidden medical abnormalities.

[0079] Subsequently, the weight resolution module determines the corresponding disease hypothesis state based on structured conflict information, such as generating candidate disease states like "progressive hemothorax" or "pulmonary embolism," and dynamically adjusts the credibility weights of voice description information, text record information, medical imaging information, and physiological monitoring data according to the disease hypothesis. Then, semantic constraint relationship matching is re-executed to suppress conflict nodes with low credibility support and retain nodes with high credibility support, thereby achieving dynamic coordination and conflict resolution among multimodal evidence and improving the reliability of medical reasoning results.

[0080] Finally, when the supplementary evidence feedback module detects unresolved structured conflict information after weighted resolution, it indicates that the current evidence chain is incomplete. At this point, the module parses the unmet medical knowledge constraints, identifies the type of missing medical evidence, and automatically generates targeted information supplementation requests, such as prompting for supplementary pulmonary artery CTA examinations or laboratory test data. This request is sent to the clinical terminal, guiding medical staff to complete targeted supplementary data collection. The supplemented multimodal medical information is then re-input into the time window construction module and processed by the weighted resolution module, forming a closed-loop optimization process. This process enables human-machine collaborative communication between the system and medical staff, allowing the system to proactively identify information gaps and initiate supplementary evidence requests, thereby effectively improving the completeness of medical information and supporting the early and accurate identification of latent complications.

[0081] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent processing of medical knowledge using multimodal communication, characterized in that, Includes the following steps: S1. Acquire multimodal medical information and process it into medical data units with multiple time windows, wherein the multimodal medical information includes voice description information, text record information, medical image information and physiological monitoring data; S2. Extract features from each modality of medical information within each time window, map the extracted medical features to the corresponding medical knowledge nodes in the preset medical knowledge intermediary layer, and generate semantic activation states corresponding to the medical knowledge nodes; wherein, the medical knowledge intermediary layer is preset with medical knowledge nodes and semantic constraint relationships between medical knowledge nodes. S3. Based on the semantic constraint relationship, perform consistency analysis on the semantic activation state across time windows. When the semantic activation state does not satisfy the corresponding semantic constraint relationship, generate structured conflict information. S4. Determine the disease hypothesis state associated with the medical knowledge nodes involved in the structured conflict information, adjust the credibility weights corresponding to each modality of medical information based on the disease hypothesis state, and resolve the structured conflict information based on the semantic constraint relationship. S5. When it is detected that there is still unresolved structured conflict information after processing in step S4, the missing medical evidence type is determined based on the structured conflict information and the semantic constraint relationship, an information supplementation requirement indicating the missing medical evidence type is generated, and the supplemented multimodal medical information is input to steps S1 to S4 for processing.

2. The medical knowledge intelligent processing method based on multimodal communication according to claim 1, characterized in that, In S1, the step of acquiring multimodal medical information and processing it into medical data units of multiple time windows includes: Acquire voice description information, text recording information, medical imaging information, and physiological monitoring data; Time-stamped processing is performed on voice description information, text record information, medical imaging information, and physiological monitoring data to form time-stamped multimodal medical information; The time-stamped multimodal medical information is arranged in chronological order according to the time stamp to generate a multimodal medical information sequence; The multimodal medical information sequence is divided into time intervals based on time order, and the multimodal medical information in each time interval is aggregated to generate medical data units corresponding to multiple time windows.

3. The medical knowledge intelligent processing method based on multimodal communication according to claim 1, characterized in that, In S2, the step of extracting features from each modality of medical information within each time window includes: Perform medical semantic analysis on voice description information and text record information to extract medical entity words and medical terminology; Medical image features are extracted from medical image information to generate image structure feature representations; Time series analysis was performed on physiological monitoring data to extract characteristic sequences of physiological index changes.

4. The medical knowledge intelligent processing method using multimodal communication according to claim 1, characterized in that, In S2, the step of mapping the extracted medical features to the corresponding medical knowledge nodes in the preset medical knowledge intermediary layer includes: Retrieve candidate medical knowledge nodes in the medical knowledge intermediary layer based on medical features; Perform semantic matching processing on medical features and candidate medical knowledge nodes to determine the target medical knowledge node; Establish a corresponding mapping relationship between medical features and target medical knowledge nodes.

5. The medical knowledge intelligent processing method using multimodal communication according to claim 1, characterized in that, In S2, the step of generating the semantic activation state corresponding to the medical knowledge node includes: Set node status identifiers for medical knowledge nodes; The node status identifier is updated based on the mapping relationship between medical features and medical knowledge nodes; Semantic activation states are generated based on the update results of node state identifiers.

6. The medical knowledge intelligent processing method based on multimodal communication according to claim 1, characterized in that, In S3, the step of performing consistency analysis on the semantic activation states across time windows includes: Extract the semantic activation state sets corresponding to different time windows; The semantic activation state set is time-aligned according to the time window order to form a cross-time alignment sequence; Based on the semantic constraint relationship between medical knowledge nodes in the medical knowledge mediation layer, node-by-node constraint matching is performed on the semantic activation state in the cross-time alignment sequence. The satisfaction status between semantic activation states is determined based on the constraint relationship matching results.

7. The medical knowledge intelligent processing method based on multimodal communication according to claim 1, characterized in that, In S3, the step of generating structured conflict information includes: Identify medical knowledge nodes that do not satisfy the semantic constraints in the consistency analysis; Determine the time window combination relationship corresponding to medical knowledge nodes that do not satisfy the semantic constraint relationship; Generate conflict type identifiers based on the semantic constraint relationships; Structured conflict information is generated based on medical knowledge nodes, time window combinations, and conflict type identifiers.

8. The medical knowledge intelligent processing method using multimodal communication according to claim 1, characterized in that, In S4, the step of adjusting the credibility weights corresponding to each modality of medical information based on the disease hypothesis state includes: The disease hypothesis state is determined based on the medical knowledge nodes involved in the structured conflict information; Based on the disease hypothesis state, extract the medical evidence types and medical constraint relationships corresponding to the disease hypothesis state from the medical knowledge intermediary layer; Based on the types of medical evidence, determine the modal evidence association relationships corresponding to voice description information, text record information, medical imaging information, and physiological monitoring data; The initial credibility weights of voice description information, text record information, medical image information, and physiological monitoring data are redistributed based on medical constraints.

9. The medical knowledge intelligent processing method based on multimodal communication according to claim 1, characterized in that, In S5, the step of generating supplementary information indicating the type of missing medical evidence includes: Parse the unresolved structured conflict information and extract the corresponding medical knowledge nodes and semantic constraint relationships; Based on medical knowledge nodes and semantic constraint relationships, reverse constraint deduction is performed on medical knowledge nodes that do not meet the constraint conditions to determine the type of missing medical evidence. The types of missing medical evidence are classified and processed according to the evidence classification system of the medical knowledge intermediary layer; Based on the classification results, information supplementation requirements are generated corresponding to the types of missing medical evidence.

10. A multimodal communication-based intelligent medical knowledge processing system, characterized in that, A medical knowledge intelligent processing method for multimodal communication according to any one of claims 1-9, the system comprising: The time window construction module is used to acquire multimodal medical information and process it into medical data units with multiple time windows. The multimodal medical information includes voice description information, text record information, medical image information and physiological monitoring data. The semantic activation module is used to extract features from medical information of each modality within each time window, map the extracted medical features to the corresponding medical knowledge nodes in the preset medical knowledge intermediary layer, and generate the semantic activation state corresponding to the medical knowledge nodes; wherein, the medical knowledge intermediary layer is preset with medical knowledge nodes and semantic constraint relationships between medical knowledge nodes. The conflict resolution module is used to perform consistency analysis on the semantic activation state across time windows based on the semantic constraint relationship. When the semantic activation state does not satisfy the corresponding semantic constraint relationship, structured conflict information is generated. The weight resolution module is used to determine the disease hypothesis state associated with the medical knowledge nodes involved in the structured conflict information, adjust the credibility weights corresponding to each modality of medical information based on the disease hypothesis state, and resolve the structured conflict information based on the semantic constraint relationship. The supplementary evidence feedback module is used to determine the type of missing medical evidence based on the relationship between the structured conflict information and the semantic constraints when unresolved structured conflict information is detected after processing by the weight resolution module. It generates information supplementation requirements indicating the type of missing medical evidence and inputs the supplemented multimodal medical information into the time window construction module for processing by the weight resolution module.

Images