Neural symbol hybrid reasoning-based multi-modal clinical scientific research data processing method and system

Abstract

The invention discloses a multi-modal clinical scientific research data processing method and system based on neural symbol hybrid reasoning, and belongs to the field of medical informatization and artificial intelligence. The method comprises the steps of receiving a natural language analysis instruction; analyzing the instruction into a structured formal problem description data object through a language understanding unit by utilizing a nerve-symbol hybrid inference engine, and selecting a data analysis algorithm according to an expert rule base through a symbol inference unit; then, generating a directed acyclic graph analysis flow based on the selected algorithm; and finally, according to the directed acyclic graph analysis flow, processing the multi-modal clinical scientific research data stored in the database, and generating an analysis report containing quantitative indexes. The technical problems that in the prior art, a clinical scientific research data analysis process is split, efficiency is low, and a semantic gap exists in man-machine interaction are solved, automation and intelligence of scientific research analysis are achieved, and the efficiency, depth and scientificity of data processing are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multimodal clinical research data processing method and system based on neural symbolic hybrid reasoning. Background Technology

[0002] With the development of precision medicine, the demand for comprehensive analysis of multimodal data in clinical research is becoming increasingly urgent. Multimodal data includes various types such as images, medical record texts, physiological signals, genomics, and rating scales. Currently, the clinical research field generally faces the problem of data silos. Clinical data is physically dispersed across different medical institutions and information systems, and logically exhibits heterogeneity in format, structure, and semantics. This heterogeneity makes cross-modal correlation analysis difficult, resulting in a large amount of potential medical knowledge remaining untapped.

[0003] Clinical researchers typically export data manually and then perform offline analysis using external statistical software such as SPSS, R, and Python. This analytical process is time-consuming and error-prone, requiring strong programming and statistical skills from researchers, limiting the depth and breadth of data analysis. Furthermore, the analysis process is disconnected from data management, hindering real-time, collaborative exploratory analysis. Existing software interfaces struggle to understand researchers' complex analytical intentions, requiring researchers to translate their research ideas into cumbersome software steps, resulting in low human-computer interaction efficiency. In addition, traditional role-based access control (RBAC) is insufficient to meet the needs of research projects for granular data authorization. This disconnect between technical access control and research ethics poses security and compliance risks to multi-center collaborative data research.

[0004] Therefore, there is an urgent need for a multimodal clinical research data processing method that can effectively solve the above problems and improve the efficiency and quality of clinical research.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a multimodal clinical research data processing method and system based on neural symbolic hybrid reasoning, in order to solve the problems in the existing technology of clinical research when conducting comprehensive analysis of multimodal data, such as difficulty in effectively integrating heterogeneous data, inefficient data analysis process, semantic gaps in human-computer interaction, and disconnect between data security and ethical norms, so as to achieve efficient and intelligent analysis and processing of multimodal clinical research data.

[0007] This invention provides a multimodal clinical research data processing method based on neural symbolic hybrid reasoning, comprising the following steps executed by a computer:

[0008] Receive natural language analysis commands from user terminals for multimodal clinical research data;

[0009] Using a neural-symbolic hybrid reasoning engine, the natural language analysis instructions are first parsed into structured formal problem description data objects by the language understanding unit. Then, the symbolic reasoning unit selects one or more data analysis algorithms from a pre-set algorithm library based on the pre-set expert rule base and the formal problem description data objects.

[0010] Based on one or more selected data analysis algorithms and formal problem description data objects, a directed acyclic graph analysis flow is generated that defines the data processing steps and the dependencies between them.

[0011] Based on the directed acyclic graph analysis flow, multimodal clinical research data containing at least two types of technically meaningful data stored in the database are extracted and processed, and a research data analysis report containing quantitative analysis indicators is generated.

[0012] In some alternative embodiments, the formal problem description data object includes at least one analytical intent and at least one parameter entity.

[0013] In some optional embodiments, before selecting a data analysis algorithm using the symbolic reasoning unit, the method further includes:

[0014] By using the semantic entity grounding module, the parameter entities in the formal problem description data object are mapped to specific data fields or data codes in the database.

[0015] In some optional embodiments, the semantic entity grounding module queries and maps parameter entities through a pre-built medical ontology knowledge graph.

[0016] In some optional embodiments, it also includes:

[0017] During the mapping process, when a parameter entity has multiple mapping results or no mapping result, the mapping target of the parameter entity is determined by combining the user's historical operation data or the context information of the current scientific research project.

[0018] In some optional embodiments, the database is a time-series graph hybrid database; the step of extracting multimodal clinical research data based on directed acyclic graph analysis flow specifically includes:

[0019] By querying the federated optimizer, the data extraction request is decomposed into a graph traversal sub-request for the graph database part of the time-series graph hybrid database and a time-series scan sub-request for the time-series database part, and the data returned after executing the sub-requests is merged.

[0020] In some alternative embodiments, multimodal clinical research data includes at least two of the following: physiological signal data, imaging feature data, rating scale data, or genomic data.

[0021] In some optional embodiments, the method further includes, before receiving natural language analysis instructions:

[0022] Through upstream data processing pipelines, multimodal clinical research data from different sources are uniformly converted into a clinical event data structure that includes timestamps, event types, and event content.

[0023] In some alternative embodiments, the upstream data processing pipeline further includes:

[0024] Using natural language processing models, standardized medical entity labels are automatically extracted and labeled from text-based clinical event data.

[0025] In some optional embodiments, the method further includes:

[0026] High-frequency medical entity tags extracted from upstream data processing pipelines that cannot be mapped in existing medical ontology knowledge graphs are used as candidate entities; and

[0027] The system receives user confirmation of candidate entities through a human-computer collaborative verification interface and automatically updates the medical ontology knowledge graph based on the confirmation.

[0028] In some optional embodiments, prior to extracting and processing multimodal clinical research data based on directed acyclic graph analysis flow, the method further includes:

[0029] The access control module verifies the authorization of data processing requests and their associated scientific research purpose identifiers according to a preset authorization policy.

[0030] In some optional embodiments, the method further includes:

[0031] Data processing requests, research objective identifiers, and summary information of research data analysis reports are recorded in an audit log based on distributed ledger technology.

[0032] This invention provides a multimodal clinical research data processing system based on neural symbolic hybrid reasoning, comprising:

[0033] The instruction receiving module is configured to receive natural language analysis instructions for multimodal clinical research data input from the user terminal.

[0034] The neural symbolic hybrid reasoning module is connected to the instruction receiving module and is configured to first parse the natural language analysis instructions into structured formal problem description data objects through the language understanding unit, and then select one or more data analysis algorithms from the preset algorithm library through the symbolic reasoning unit based on the preset expert rule base and the formal problem description data objects.

[0035] The analysis flow generation module, connected to the neural symbolic hybrid reasoning module, is configured to generate a directed acyclic graph analysis flow that defines the data processing steps and the dependencies between them, based on one or more selected data analysis algorithms and formal problem description data objects.

[0036] The data processing and report generation module, connected to the analysis flow generation module, is configured to extract and process multimodal clinical research data stored in the database, which contains at least two types of data with technical significance, based on the directed acyclic graph analysis flow, and generate a research data analysis report containing quantitative analysis indicators.

[0037] In some alternative embodiments, the system further includes:

[0038] The semantic entity grounding module is configured to map the parameter entities contained in the formal problem description data object to specific data fields or data codes in the database before the symbolic reasoning unit of the neural symbolic hybrid reasoning module selects a data analysis algorithm.

[0039] In some optional embodiments, the database is a time-series graph hybrid database; the data processing and report generation module includes:

[0040] The query federated optimizer is configured to decompose data extraction requests into graph traversal sub-requests for the graph database portion of the time-series graph hybrid database and time-series scan sub-requests for the time-series database portion, and then merge the data returned after executing the sub-requests.

[0041] In some alternative embodiments, the system further includes:

[0042] The access control module is configured to authorize and verify data processing requests and their accompanying research purpose identifiers according to a preset authorization policy before the data processing and report generation module extracts and processes multimodal clinical research data.

[0043] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention.

[0044] The multimodal clinical research data processing method and system based on neural symbolic hybrid reasoning of the present invention has the following beneficial effects:

[0045] This invention provides a multimodal clinical research data processing method based on neural symbolic hybrid reasoning. It transforms natural language commands into structured data objects through a neural symbolic hybrid reasoning engine, selects appropriate algorithms, and automatically generates a directed acyclic graph analysis process for extracting, processing, and analyzing multimodal data. This method can reduce manual processes that previously took weeks or even months to minutes, significantly improving research efficiency. The system can perform cross-modal data correlation analysis to reveal deep medical mechanisms. This method employs natural language understanding, providing data analysis capabilities for clinical experts without programming backgrounds. Furthermore, the neural symbolic hybrid reasoning mechanism ensures the scientific rigor of the selected analysis methods and the accuracy of the computational results. An access control module safeguards research ethics and data security. Attached Figure Description

[0046] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0047] Figure 1 This is a flowchart of a multimodal clinical research data processing method based on neural symbolic hybrid reasoning according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of the structure of a multimodal clinical research data processing system based on neural symbolic hybrid reasoning according to an embodiment of the present invention. Detailed Implementation

[0049] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0050] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0051] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined. Therefore, the actual execution order may change depending on the specific circumstances.

[0052] Clinical research data processing involves multiple fields, including natural language understanding, knowledge representation, reasoning and decision-making, and data mining. Natural language understanding enables computers to parse and understand human language, thus facilitating human-computer interaction; knowledge representation formalizes medical knowledge for computer reasoning and application; reasoning and decision-making simulates the thought processes of human experts, making judgments and choices based on existing knowledge and data; and data mining discovers potential patterns and correlations from massive amounts of data. The synergistic effect of these technologies automates the process from natural language commands to research data analysis reports, freeing researchers from tedious data processing tasks and allowing them to focus on medical research.

[0053] like Figure 1 As shown, this embodiment of the invention provides a multimodal clinical research data processing method based on neural symbolic hybrid reasoning. This method is executed by a computer and aims to process multimodal clinical research data to assist researchers in analysis and research.

[0054] S100: Receives natural language analysis commands for multimodal clinical research data input from the user terminal. These commands are received via a web interface or application programming interface (API), and users can input commands using a keyboard, voice, or other methods. For example, the user inputs the command: "Compare the changes in mean systolic blood pressure in hypertensive patients treated with drug A and drug B."

[0055] S200 processes received instructions using a neural-symbolic hybrid reasoning engine. This engine includes a language understanding unit and a symbolic reasoning unit.

[0056] The language understanding unit is based on a pre-trained natural language processing model, such as BERT or the GPT series, which has been fine-tuned on medical datasets to improve its ability to recognize medical terminology and research intent. The language understanding unit parses the user's input natural language analysis instructions into a structured formal problem description data object, represented in formats such as JSON or XML. In this example, the formal problem description data object might contain the following information:

[0057] {

[0058] "Analysis Intent": "Intergroup Comparison"

[0059] Target audience: Patients with hypertension

[0060] "Grouping": ["Group A: Drug Treatment Group", "Group B: Drug Treatment Group"],

[0061] "Observational Indicator": "Mean Systolic Blood Pressure"

[0062] "Time range": "During treatment"

[0063] }

[0064] Optionally, the language understanding unit can also use a rule-based approach for natural language parsing, for example, by predefining a set of word and grammar rules and converting natural language instructions into formal problem description data objects according to these rules.

[0065] The symbolic reasoning unit receives a formalized problem description data object and selects one or more data analysis algorithms from a pre-defined algorithm library based on a pre-set expert rule base and the formalized problem description data object. The expert rule base, constructed by medical statistics experts, contains a series of rules for selecting appropriate data analysis algorithms based on different analytical intentions and data types. For example, if the analytical intention is "between-group comparison" and the observed indicator is a "continuous variable," then a t-test or analysis of variance is selected; if the analytical intention is "correlation analysis" and the data type is a "categorical variable," then a chi-square test is selected. In this example, since the analytical intention is "between-group comparison" and the observed indicator "mean systolic blood pressure" is a continuous variable, the symbolic reasoning unit selects the t-test as the data analysis algorithm. The algorithm library can contain various commonly used data analysis algorithms, such as t-tests, analysis of variance, linear regression, logistic regression, survival analysis, etc., which can be implemented using Python, R, or other programming languages.

[0066] S300. Based on one or more selected data analysis algorithms and the formal problem description data object, generate a directed acyclic graph (DAG) analysis flow that defines the data processing steps and the dependencies between them. Each node in the DAG represents a data processing step, such as data filtering, data transformation, data aggregation, statistical calculation, etc., and edges represent the dependencies between these steps. The analysis flow generation module automatically generates the DAG analysis flow based on the formal problem description data object and the selected data analysis algorithm. In this example, the generated DAG analysis flow may contain the following nodes:

[0067] Node 1: Screening data for patients with hypertension.

[0068] Node 2: Divide hypertensive patients into drug treatment group A and drug treatment group B.

[0069] Node 3: Calculate the mean systolic blood pressure for drug treatment group A and drug treatment group B.

[0070] Node 4: Use a t-test to compare whether there is a statistically significant difference in the mean systolic blood pressure between the two groups.

[0071] Node 5: Generate an analysis report containing the results of the t-test, such as t-value, p-value, and confidence interval.

[0072] The dependencies between nodes are as follows: Node 1 is the input of Node 2, Node 2 is the input of Node 3, Node 3 is the input of Node 4, and Node 4 is the input of Node 5.

[0073] Alternatively, the directed acyclic graph analysis flow can also be represented using other data structures, such as tree structures or linear structures.

[0074] S400. Based on the directed acyclic graph analysis flow, extract and process the multimodal clinical research data stored in the database, and generate a research data analysis report containing quantitative analysis indicators. The multimodal clinical research data stored in the database contains at least two types of data with technical significance, such as patient medical records, imaging data, genomic data, and physiological signal data. In this example, the database may contain the following data: patient ID, name, age, gender, diagnosis, treatment plan, and systolic blood pressure measurement records.

[0075] The data processing module executes the data processing steps of each node sequentially according to the definition of a directed acyclic graph (DAG) analysis flow. For example, it first executes node 1 to filter out all patient data diagnosed with hypertension; then it executes node 2 to divide the hypertensive patients into drug treatment group A and drug treatment group B; next, it executes node 3 to calculate the mean systolic blood pressure of drug treatment group A and drug treatment group B; then it executes node 4 to use a t-test to compare whether there is a statistically significant difference in the mean systolic blood pressure between the two groups; finally, it executes node 5 to generate an analysis report, including the results of the t-test, such as t-values, p-values, and confidence intervals. The analysis report can be displayed in HTML, PDF, or other formats. During data processing, the data extraction module uses SQL queries to extract data from relational databases or uses APIs to extract data from non-relational databases. The data processing module uses Python, R, or other programming languages ​​to implement various data processing algorithms, such as data filtering, data transformation, data aggregation, and statistical calculations.

[0076] Through the above steps, the method provided by this invention can automatically extract and process multimodal clinical research data from a database based on user-input natural language analysis commands, and generate a research data analysis report containing quantitative analysis indicators, thereby helping researchers conduct data analysis and research more efficiently. This method uses a neural-symbolic hybrid inference engine to convert natural language commands into structured, formalized problem description data objects, and selects appropriate data analysis algorithms based on an expert rule base, ensuring the scientific rigor and accuracy of the data analysis. By using directed acyclic graph analysis flow, the complex data analysis process is decomposed into a series of simple data processing steps, making the data analysis process clearer and more controllable.

[0077] In some embodiments, the structured formal problem description data object generated by the multimodal clinical research data processing method includes at least one analytical intent and at least one parameter entity, thereby providing a basis for selecting a suitable data analysis algorithm for subsequent symbolic reasoning units.

[0078] The analysis intent indicates the type of analysis the researchers wish to perform, while the parameter entity represents the specific object or variable involved in the analysis. In this embodiment, the analysis intent is represented as a string, describing the analytical operation the researchers wish to perform, such as inter-group comparison, correlation analysis, survival analysis, etc. The parameter entity is represented as a key-value pair, where the key represents the parameter type and the value represents the parameter's specific content. For example, for the natural language analysis instruction "Compare the changes in VAS scores within one year after patients with lumbar spinal stenosis receive minimally invasive decompression surgery and conservative physical therapy, respectively," the formal problem description data object can be represented in JSON format, and its analysis intent is set to "longitudinal group comparison," i.e., `"analysisIntent": "longitudinal_group_comparison"`. The parameter entity includes information such as the study population, group, time window, and evaluation indicators. The study population is set to patients with lumbar spinal stenosis, i.e., `"population": {"disease": "lumbar spinal stenosis"}`. The grouping information was set as follows: patients receiving minimally invasive decompression surgery and patients receiving conservative physical therapy, i.e., `"groups": [{"treatment": "minimally invasive decompression surgery"}, {"treatment": "conservative physical therapy"}]`. The time window was set to within one year after treatment, i.e., `"timeWindow": {"duration": "1 year", "unit": "post_treatment"}`. The evaluation indicator was set as the VAS score, i.e., `"metrics": [{"scale": "VAS score"}]`.

[0079] The language understanding unit parses natural language instructions into formal problem description data objects in the following ways: First, the natural language instructions are segmented and tagged with parts of speech; then, a pre-trained language model, such as a proprietary large language model, is used to semantically encode each word; next, named entity recognition technology is used to identify entities in the instructions, such as disease names, treatment methods, and evaluation indicators; finally, the identified entities are filled into the formal problem description data objects according to a predefined template.

[0080] Optionally, the parameter entity can contain multiple values. For example, the evaluation metrics can include VAS scores and ODI scores, i.e., `"metrics": [{"scale": "VAS score"}, {"scale": "ODI score"}]`.

[0081] After receiving the formalized problem description data object, the symbolic reasoning unit selects an appropriate data analysis algorithm based on a pre-defined expert rule base. This expert rule base, constructed by medical statistics experts, contains a series of rules used to map analytical intent and parameter entities to specific data analysis algorithms. For example, if the analytical intent is a longitudinal grouped comparison, the evaluation index is a continuous variable, and there are two comparison groups, then a linear mixed-effects model is selected.

[0082] Optionally, the formal problem description data object can be represented in other formats, such as XML or YAML. The expert rule base can also be represented in other forms, such as decision trees or rule engines.

[0083] By including analytical intent and parameter entities, formal problem description data objects can clearly express researchers' analytical intent and provide a basis for the selection of subsequent data analysis algorithms, enabling the system to automatically complete data processing and analysis, thereby improving the efficiency and depth of clinical research.

[0084] In some embodiments, before selecting a data analysis algorithm using the symbolic reasoning unit, the system performs a semantic entity grounding operation, mapping the parameter entities in the formal problem description data object to specific data fields or data codes in the database. Specifically, the semantic entity grounding module receives a structured formal problem description data object from the neural-symbolic hybrid reasoning engine. For example, the object contains a parameter entity of the form `{"disease": "lumbar spinal stenosis"}`. The task of the semantic entity grounding module is to map the string "lumbar spinal stenosis" to a specific field or code in the database that represents the disease. This module first invokes its deterministic mapping submodule. The deterministic mapping submodule is connected to a pre-built medical ontology knowledge graph. This knowledge graph can be stored and managed using the Resource Description Framework (RDF) or other suitable knowledge representation methods. The knowledge graph stores a large number of medical concepts, entities, and the relationships between them. Each medical concept or entity is associated with one or more unique identifiers, such as ICD-10 codes, MeSH terminology, etc. The deterministic mapping submodule performs an exact match query in the knowledge graph using the string "lumbar spinal stenosis". If a matching entry exists in the knowledge graph, such as a medical concept named "lumbar spinal stenosis" with the corresponding ICD-10 code M48.0, the deterministic mapping submodule maps the parameter entity to the ICD-10 code M48.0. This mapping result will be used for subsequent data queries and analysis. Specifically, subsequent database queries will use the ICD-10 code M48.0 to filter for patients with lumbar spinal stenosis.

[0085] Optionally, if the deterministic mapping submodule cannot find a perfectly matching entry—for example, if there is no entry in the knowledge graph directly named "lumbar spinal stenosis" but there are synonyms or near-synonyms, such as "lumbar spinal stenosis"—then the deterministic mapping submodule can use a fuzzy matching algorithm, such as an edit distance algorithm or word vector-based similarity calculation, to identify potential matches. Then, the deterministic mapping submodule returns all matches with a similarity exceeding a preset threshold, which are then manually confirmed and selected.

[0086] Optionally, the deterministic mapping submodule can also be combined with natural language processing techniques, such as named entity recognition and relation extraction, to automatically learn representations of medical entities from clinical text data and add them to the knowledge graph, thereby improving the coverage and accuracy of the knowledge graph.

[0087] Furthermore, building upon deterministic mapping, if a parameter entity has multiple possible mapping results—for example, if the user input is "compare Dr. Li's LSS patients..."—and there may be multiple "Dr. Li"s in the system, or the parameter entity may have multiple possible interpretations (e.g., "LSS" could represent lumbar spinal stenosis or other diseases), then the system invokes a probabilistic disambiguation module. This module utilizes the user's historical operation data or the contextual information of the current research project to determine the mapping target of the parameter entity. For instance, the module might obtain the current user's profile, including their department, title, and research direction. If the user is a neurosurgeon and has recently frequently viewed data related to "Dr. Li Ming," the module would infer with high confidence that "Dr. Li" refers to "Li Ming." Alternatively, the module might analyze the description and objectives of the current research project. If the project is about lumbar spine diseases, the module would infer with high confidence that "LSS" represents lumbar spinal stenosis. The probabilistic disambiguation module can use Bayesian networks or other probabilistic graphical models to calculate the probabilities of different mapping targets and select the mapping target with the highest probability as the final result. This mapping result will overwrite the previous deterministic mapping results, thereby improving the accuracy of the mapping.

[0088] Optionally, the probabilistic disambiguation submodule can also interact with the user, for example, by displaying a dialog box allowing the user to select the correct mapping target. The user's selection will be used to update the user profile and the contextual information of the research project, thereby improving the accuracy of subsequent disambiguation.

[0089] Through the semantic entity grounding process described above, the parameter entities in the formal problem description data object are accurately mapped to specific data fields or data codes in the database. For example, `{"disease": "lumbar spinal stenosis"}` is mapped to `{"disease_icd10": "M48.0"}`, ensuring that subsequent data queries and analyses can be performed based on a unified semantic understanding, avoiding data errors and analytical biases caused by semantic ambiguity. This process guarantees the accuracy and reliability of the data processing workflow and improves the quality of research results.

[0090] In some embodiments, to accurately map parameter entities in a formal problem description data object to specific data fields or codes in a database, the semantic entity grounding module utilizes a pre-built medical ontology knowledge graph for querying and mapping. This medical ontology knowledge graph is a structured knowledge base that stores various concepts, entities, and relationships within the medical field. The knowledge graph can be stored and represented in various forms, such as relational databases, graph databases, or RDF-based triplet storage. Optionally, the knowledge graph can be based on existing medical ontologies, such as SNOMED CT, ICD, MeSH, etc., and customized and extended according to the actual needs of clinical research. For example, SNOMED CT provides a comprehensive set of medical terms that can be used to standardize the representation of concepts such as diseases, symptoms, examinations, and treatments. Graph databases such as Neo4j can be used to store complex relationships between medical entities, such as the association between diseases and genes, or the association between drugs and adverse reactions. The workflow of the semantic entity grounding module is as follows:

[0091] 1. Receiving Parameter Entities: The Semantic Entity Grounding module receives parameter entities from the formal problem description data object passed from the neural-symbolic hybrid inference engine. For example, the received parameter entity might be the string "diabetes".

[0092] 2. Querying the Medical Ontology Knowledge Graph: The semantic entity grounding module uses the received parameter entities as query terms to perform queries within the medical ontology knowledge graph. The query process can employ various algorithms, such as string matching, fuzzy matching, and semantic similarity calculation.

[0093] String matching: Directly compare the parameter entity with the entity name in the knowledge graph to find an entity that matches exactly. For example, if the parameter entity is "diabetes", then search for an entity named "diabetes" in the knowledge graph.

[0094] Fuzzy matching: Allows for differences between the parameter entity and the entity name in the knowledge graph, such as spelling errors or abbreviations. For example, algorithms such as Levenshtein distance or Jaro-Winkler distance can be used to calculate the similarity between strings, and a threshold can be set to determine whether a match is made.

[0095] Semantic similarity calculation: Based on entity relationships in the knowledge graph, the semantic similarity between a parameter entity and entities in the knowledge graph is calculated. For example, if the parameter entity is "type 2 diabetes," and the knowledge graph contains the entity "diabetes," and "type 2 diabetes" is a type of "diabetes," then a high semantic similarity can be considered between the two. Semantic similarity can be calculated using natural language processing techniques such as Word2Vec and GloVe. These techniques learn from large amounts of text data, mapping words to a vector space, thereby calculating the similarity between words.

[0096] 3. Returning Mapping Results: If a matching entity is found, the semantic entity grounding module will return the entity's unique identifier in the knowledge graph, such as SNOMED CT encoding or ICD encoding. If multiple matching entities are found, disambiguation processing is required, such as combining user history or contextual information from the current research project to select the most appropriate entity. If no matching entity is found, manual intervention may be necessary, such as having domain experts label and map the parameter entities.

[0097] Optionally, the semantic entity grounding module can also utilize machine learning models, such as named entity recognition models or entity linking models, to automatically identify and map parameter entities. Named entity recognition models can be used to identify medical entities in text, such as diseases, drugs, and genes. Entity linking models can link identified entities to corresponding entities in a knowledge graph. For example, when the received parameter entity is "lumbar spinal stenosis," the semantic entity grounding module queries the medical ontology knowledge graph, which returns its corresponding ICD-10 code M48.0. This code will then be used for database queries to improve query accuracy and efficiency.

[0098] By leveraging a pre-built medical ontology knowledge graph, this semantic entity grounding module can accurately map parameter entities in natural language instructions to specific data fields or codes in the database, thus laying the foundation for subsequent data extraction and analysis. The introduction of the medical ontology knowledge graph enables the system to understand the meaning of medical terms and perform standardization, avoiding data errors caused by inconsistencies in terminology.

[0099] In some embodiments, when a parameter entity has multiple mapping results or no mapping result, the mapping target of the parameter entity is determined by combining the user's historical operation data or the context information of the current research project. The specific implementation process is as follows:

[0100] First, the system receives a formalized problem description data object from the neural-symbolic hybrid reasoning engine, for example, {"disease": "Dr. Li's LSS patient"}. The parameter entity "Dr. Li's LSS patient" needs to be mapped to a specific patient group in the database. The semantic entity grounding module attempts to query the medical ontology knowledge graph. Assuming the knowledge graph contains the ICD-10 code M48.0 for "lumbar spinal stenosis (LSS)," but the system contains multiple doctors with the surname Li, such as Li Ming (employee ID: 1024, department: neurosurgery) and Li Hua (employee ID: 2048, department: orthopedics), determining which doctor "Dr. Li" refers to results in multiple mappings. To address this issue, a probabilistic disambiguation submodule is activated, combining user historical operation data and the contextual information of the current research project to determine the mapping target of the parameter entity. One implementation is that this submodule queries the current user's historical operation logs. If a user (e.g., a neurosurgeon) has frequently accessed Dr. Li Ming's (Employee ID: 1024, Department: Neurosurgery) patient data within the past month, and the current research project, "A Study on the Prognosis of Minimally Invasive Surgery in LSS Patients," is also initiated by the Neurosurgery Department, then this submodule will calculate a confidence score. For example, Dr. Li Ming's confidence score might be 0.9, and Dr. Li Hua's might be 0.1. The confidence score can be calculated based on a weighted average of factors such as user access frequency, department matching degree, and the department that initiated the project. Another implementation method is to consider the contextual information of the current research project. Research project metadata typically includes information such as the project leader, participants, and research direction. If Dr. Li Ming is the project leader, or if the project's research direction is neurosurgical diseases, then Dr. Li Ming's confidence score can be increased.

[0101] The system then selects the mapping target with the highest confidence level, namely Dr. Li Ming (Employee ID: 1024), as the final mapping result. Therefore, the parameter entity "Dr. Li's LSS patient" is finally mapped to: {ICD-10: "M48.0", Doctor ID: "1024"}.

[0102] Optionally, if the confidence level of the mapping result with the highest confidence is below a certain threshold, such as 0.7, the system can display a prompt box allowing the user to manually select the correct mapping target. The prompt box will list all possible Dr. Lis and display their departments and titles for the user's reference.

[0103] By combining historical user operation data or contextual information from the current research project, the mapping target of parameter entities can be determined more accurately, resolving the issue of multiple mapping results or no mapping result for parameter entities. This method can avoid data analysis bias caused by mapping errors, improving the accuracy and reliability of research results.

[0104] In other implementations, when no matching entry exists in the medical ontology knowledge graph for the parameter entity "Dr. Li's LSS patient," the system attempts fuzzy matching. For example, the system first identifies "Dr. Li" and "LSS patient" as two independent entities, and then searches the database for data that simultaneously meets both conditions. If matching data is found, the system asks the user whether to use that data as a mapping target. If the user confirms, the system adds the data to the medical ontology knowledge graph for later use, thus enabling dynamic updates to the medical ontology knowledge graph.

[0105] In some embodiments, the database employs a time-series-graph hybrid database architecture, with data extraction implemented through a query federated optimizer to improve query efficiency and flexibility. Time-series databases, such as InfluxDB, store clinical event data with timestamp attributes, such as patients' physiological signals, medication records, and laboratory test results. Graph databases, such as Neo4j, store entities such as patients, diseases, drugs, genes, and their relationships, forming a knowledge graph. This hybrid architecture fully leverages the advantages of both databases to meet the data query needs of complex clinical research scenarios. The role of the query federated optimizer is to receive data extraction requests from upper-level modules and decompose them into sub-requests that can be executed on the time-series database and the graph database, respectively. The decomposition is based on the characteristics of the data storage and the type of query. For example, a query request might be "find patients with lumbar spinal stenosis who have undergone minimally invasive laminectomy and obtain their VAS score data one year post-surgery." The query federated optimizer would first identify that "finding patients with lumbar spinal stenosis who have undergone minimally invasive laminectomy" is a task involving entity relationship queries, suitable for execution on a graph database. Therefore, it translates this part of the request into a query language supported by graph databases, such as Cypher. The generated query might be "MATCH(p:Patient)-[:HAS_DIAGNOSIS]->(d:Diagnosis {name: 'Lumbar Spinal Stenosis'})-[:HAS_PROCEDURE]->(pr:Procedure {name: 'Minimally Invasive Lamina Decompression'}) RETURN p.patient_id". This statement is executed in the graph database and returns a list of patient IDs that meet the criteria. Meanwhile, the query federated optimizer recognizes that "retrieving VAS score data one year post-surgery" is a query task involving time-series data, suitable for execution in a time-series database. It translates this part of the request into a query language supported by time-series databases, such as Flux or SQL. The generated query might be "SELECT value FROM vas_scores WHERE patient_id IN (list of patient IDs) AND time>= surgery time AND time = surgery time + 1 year". This statement is executed in the time-series database and returns VAS score data that meet the criteria. Subsequently, the query federated optimizer joins and merges the query results obtained from the graph database and the time-series database. For example, it associates a list of patient IDs with their corresponding VAS scores to form a complete dataset. The join can be based on patient IDs. Finally, the query federated optimizer returns the merged data to the upper-level modules for subsequent data processing and analysis.

[0106] Optionally, the query federated optimizer can employ either a rule-based optimization strategy or a cost-based optimization strategy to select a query decomposition and execution plan. A rule-based optimization strategy predefines a set of rules and selects appropriate rules for decomposition and optimization based on the characteristics of the query. A cost-based optimization strategy, on the other hand, evaluates the costs of different query decomposition and execution plans and selects the plan with the lowest cost. Costs can include factors such as query execution time and resource consumption.

[0107] Optionally, the time-series database and the graph database can be deployed on different physical nodes, and the query federated optimizer needs to communicate with them via a network connection. To improve query efficiency, the query federated optimizer can employ techniques such as data prefetching and query parallelization to reduce network transmission latency.

[0108] Through the aforementioned technical means, this embodiment achieves efficient querying of a time-series-graph hybrid database. The query federated optimizer intelligently decomposes complex query requests into sub-requests suitable for execution on different types of databases and merges the results, thereby fully utilizing the advantages of time-series and graph databases, improving the efficiency and flexibility of data extraction, and providing strong support for subsequent clinical research data analysis.

[0109] In some embodiments, multimodal clinical research data includes at least two data types selected from physiological signal data, imaging feature data, rating scale data, and genomic data.

[0110] Regarding physiological signal data, time-series data of patients' electromyography (EMG) signals can be acquired using electromyography (EMG) equipment. Specifically, surface EMG electrodes can be placed on the skin surface of the target muscle groups in the patient. For example, when studying patients with lumbar spinal stenosis, electrodes are placed on the erector spinae and multifidus muscles in the lower back to record the patient's muscle electrical activity in different postures or movement states. After amplification, filtering, and analog-to-digital conversion, the EMG signals are stored in the system in time-series format. The sampling frequency can be set to 1000Hz to ensure signal accuracy. The time-series data includes timestamps and corresponding voltage values, with the voltage values ​​representing the intensity of muscle electrical activity. In other implementations, physiological signal data can also be electroencephalogram (EEG) data, electrocardiogram (ECG) data, respiratory rate data, etc. These data are acquired using EEG, ECG, and respiratory monitoring equipment, respectively, and stored in the system in a similar time-series format.

[0111] Imaging feature data can be obtained by processing magnetic resonance imaging (MRI) images. Specifically, medical image segmentation algorithms, such as the U-Net network based on deep learning, can be used to automatically segment lumbar spine MRI images to obtain the cross-sectional area of ​​the spinal canal. The segmented images can then undergo further morphological analysis to extract parameters such as the anteroposterior diameter and lateral diameter of the spinal canal. These parameters are stored numerically in the system and correlated with the patient's MRI images. For example, in studying lumbar spinal stenosis, the cross-sectional area of ​​the spinal canal at the L4 / L5 intervertebral disc level can be extracted as an indicator of the degree of stenosis. Optionally, imaging feature data can also be obtained from CT images, X-ray images, ultrasound images, etc., from which feature parameters such as tumor size, bone density, and blood vessel diameter can be extracted.

[0112] Regarding rating scale data, subjective patient ratings can be obtained through structured questionnaires. Specifically, scales such as the Visual Analogue Scale (VAS) and the Oswestry Disability Index (ODI) can be used to assess the degree of pain and functional impairment in patients. The scale data is stored in the system in time-series format, recording patient ratings at different time points. The time interval for scoring can be set according to research needs; for example, patient ratings can be recorded before treatment, and at 1 month, 3 months, 6 months, and 12 months after treatment. For instance, when studying the treatment effect of lumbar spinal stenosis, VAS and ODI scores can be recorded before and after minimally invasive decompression surgery or conservative treatment. Optionally, rating scales such as the Self-Rating Anxiety Scale (SAS) and the Self-Rating Depression Scale (SDS) can also be used to assess the patient's mental and psychological state.

[0113] Regarding genomic data, a patient's genomic information can be obtained through gene sequencing technology. Specifically, whole-exome sequencing (WES) or whole-genome sequencing (WGS) technologies can be used to sequence the patient's genome. The sequencing results are stored in the system as VCF files, containing information on the patient's single nucleotide polymorphisms (SNPs). Each SNP locus includes its chromosomal location, reference genotype, and mutant genotype. For example, when studying the genetic factors of lumbar spinal stenosis, SNP loci of genes related to cartilage metabolism, such as the COL1A1 and COL2A1 genes, can be analyzed.

[0114] Optionally, genomic data can also include gene expression data, copy number variation data, etc. The four data types mentioned above can be used individually or in combination. For example, combining imaging feature data with rating scale data can analyze the relationship between the degree of spinal stenosis and the patient's subjective pain level. For instance, correlation analysis between spinal canal cross-sectional area and VAS score can be performed to explore the association between imaging features and the patient's subjective experience. Another example is combining genomic data with physiological signal data to analyze the relationship between specific genotypes and muscle electrical activity. Comprehensive analysis of multimodal data allows for a more comprehensive and in-depth understanding of the mechanisms of disease development and progression, providing more comprehensive data support for clinical research.

[0115] This embodiment fully discloses the technical solution by detailing the specific implementation methods of four data types: physiological signal data, imaging feature data, rating scale data, and genomic data, as well as their applications in clinical research. Furthermore, this embodiment illustrates how to combine and analyze these data types to gain a more comprehensive and in-depth understanding of the mechanisms of disease development, providing more comprehensive data support for clinical research and addressing the issue of data diversity in clinical research.

[0116] In some embodiments, before the system receives natural language analysis instructions from users, it is necessary to preprocess multimodal clinical research data from different sources to ensure the effectiveness and accuracy of subsequent analysis. This data transformation process is achieved through an upstream data processing pipeline, which uniformly converts data from different sources into a clinical event data structure that includes timestamps, event types, and event content.

[0117] First, the upstream data processing pipeline can be built as a modular software system, consisting of the following components: a data access module, a data cleaning and transformation module, an event model transformation module, and a data storage module. The data access module is responsible for collecting raw data from various data sources. For example, the Electronic Medical Record (EMR) system provides patient medical record information in XML format via the HL7 interface; the Picture Archiving and Communication System (PACS) provides medical image data in DICOM format; and the Laboratory Information System (LIS) provides test result data in CSV format. The data cleaning and transformation module is responsible for cleaning, transforming, and standardizing the incoming raw data. Specifically, for XML format medical record data provided by the EMR system, this module parses the XML document and extracts medical record information such as diagnosis, medication, and examination results. For DICOM format image data provided by the PACS system, this module extracts metadata information such as patient ID, examination time, and image type. For CSV format test result data provided by the LIS system, this module parses the CSV file and extracts test items, test results, and test times. This module also performs data cleaning, such as removing duplicate records, correcting erroneous data, and handling missing values. The event model conversion module is responsible for transforming the cleaned and converted data according to a unified clinical event model, generating a standardized clinical event data structure. A clinical event model is a predefined data structure used to describe clinical events. In this embodiment, a clinical event data structure contains at least the following three fields: timestamp, indicating the time the event occurred; event type, indicating the type of event, such as diagnosis, medication, examination, etc.; and event content, indicating the specific content of the event, such as the name of the diagnosed disease, the name of the medication used, the test results, etc. The event model conversion module maps data to the corresponding fields of the clinical event model based on different data sources and event types. For example, a diagnosis record from an EMR system contains three fields in its XML document: diagnosis time, diagnosis code, and diagnosis name. The event model conversion module maps the diagnosis time to the timestamp field of the clinical event data structure, the diagnosis code to the event type field, and the diagnosis name to the event content field. Similarly, a test result record from a LIS system contains three fields in its CSV file: test time, test item, and test result. The event model conversion module maps test times to timestamp fields in the clinical event data structure, test items to event type fields, and test results to event content fields. The data storage module is responsible for storing the converted clinical event data in the database for subsequent analysis.

[0118] Optionally, the event model conversion module may also include a Natural Language Processing (NLP) submodule. This NLP submodule can be used to extract clinical event information from unstructured text data. For example, if the EMR system provides medical records in free text format, the NLP submodule can utilize natural language processing techniques, such as named entity recognition and relation extraction, to automatically identify clinical events such as diagnoses, medications, and examinations in the medical records and convert these events into a standardized clinical event data structure. Specifically, the NLP submodule can first use named entity recognition technology to identify medical entities in the medical records, such as disease names, drug names, and examination items. Then, the NLP submodule can use relation extraction technology to identify relationships between medical entities, such as diagnosis-disease relationships, medication-drug relationships, and examination-examination item relationships. Finally, the NLP submodule can convert the identified medical entities and relationships into a clinical event data structure.

[0119] Optionally, to improve data processing efficiency, the upstream data processing pipeline can adopt a parallel processing approach. For example, the data access module, data cleaning and transformation module, event model transformation module, and data storage module can be deployed on different servers, and message queue technology can be used to transfer data between different modules. In this way, different modules can process data in parallel, thereby improving the throughput of data processing.

[0120] Optionally, the upstream data processing pipeline can be integrated with knowledge graphs to achieve more intelligent data transformation. For example, knowledge graphs can be used to standardize and disambiguate medical entities, thereby improving data quality. Specifically, when the NLP submodule identifies a medical entity, it can first look up the entity's standard name and code in the knowledge graph. If the entity exists in the knowledge graph, its standard name and code are added to the clinical event data structure. If the entity does not exist in the knowledge graph, the reasoning capabilities of the knowledge graph can be used to infer the entity's type and attributes, and the inference results are added to the clinical event data structure.

[0121] Through the aforementioned upstream data processing pipeline, multimodal clinical research data from various sources are uniformly transformed into a clinical event data structure that includes timestamps, event types, and event content. This unified data structure facilitates subsequent data analysis, enabling easy integration and comparison of data from different sources. Simultaneously, the timestamp field can be used for time-series analysis, the event type field for event-driven analysis, and the event content field for detailed event descriptions and analysis. Therefore, this embodiment effectively addresses the heterogeneity problem of clinical research data, laying the foundation for subsequent research analysis.

[0122] In some embodiments, the upstream data processing pipeline, in addition to uniformly converting multimodal clinical research data from different sources into a clinical event data structure containing timestamps, event types, and event content, also includes a natural language processing (NLP) model. This NLP model is configured to automatically extract and label standardized medical entity tags from text-based clinical event data. Specifically, this NLP model can employ a pre-trained BERT model, fine-tuned for clinical medical text. For example, the BioBERT model, a variant of BERT pre-trained on biomedical literature, can be used. The fine-tuning process involves training the model to recognize and extract medical entities, such as disease names, drug names, symptoms, signs, and surgical procedures, using a large amount of labeled clinical case text. The labeled data is manually annotated by medical experts and undergoes multiple rounds of proofreading to ensure accuracy.

[0123] The workflow of the natural language processing model is as follows: First, it receives clinical event data containing textual information, such as a patient's medical record text recorded by a doctor: "Patient, male, 68 years old, chief complaint: lower back pain for 3 months, accompanied by radiating pain in the left lower extremity for 1 week. Diagnosis: lumbar disc herniation." Second, the model performs word segmentation, part-of-speech tagging, and named entity recognition on the text. For example, it decomposes the sentence into words such as "patient," ",", "male," ",", "68 years old," ",", "chief complaint," ":", "lower back pain," "3 months," ",", "accompanied by," "left lower extremity," "radiating pain," "1 week," and ".", and identifies "lower back pain" as a symptom, "radiating pain in the left lower extremity" as a sign, and "lumbar disc herniation" as a disease. Then, the model uses a conditional random field model, combined with a medical knowledge graph, to standardize the identified medical entities. For example, it standardizes "lower back pain" to the medical terminology standard code SNOMED CT:2504000, and "lumbar disc herniation" to the ICD-10 code M51.2. Finally, the model embeds standardized medical entities and their corresponding labels into the clinical event data in XML or JSON format. For example, medical record text can be labeled as: `lower back pain for 3 months, accompanied by radiating pain in the left lower extremity for 1 week. Diagnosis: lumbar disc herniation`.

[0124] Optionally, natural language processing models can also employ a combination of rule-based and machine learning-based methods. First, rule-based methods are used for initial entity recognition, followed by machine learning-based methods to refine and improve the results. In other implementations, natural language processing models can also utilize deep learning models, such as recurrent neural networks or Transformer models, for sequence labeling, thereby achieving automatic extraction and labeling of medical entities.

[0125] The natural language processing model described above can transform unstructured clinical text data into structured data, facilitating subsequent data analysis and mining. For example, it can be used to statistically analyze the proportion of patients with a certain disease exhibiting a specific symptom, or to analyze the relationship between the efficacy of a drug and the patient's genotype. This helps researchers gain a deeper understanding of the mechanisms of disease development and provides a more scientific basis for clinical decision-making.

[0126] In some embodiments, to achieve continuous updating and improvement of the medical ontology knowledge graph, the system implements an active ontology evolution mechanism. This mechanism includes the following steps:

[0127] First, the medical entity tags extracted from the upstream data processing pipeline are monitored. As in the aforementioned embodiment, the upstream data processing pipeline is responsible for converting clinical data from various sources into a unified clinical event structure and extracting medical entity tags using a natural language processing model. The monitoring process is implemented through a separate monitoring module that receives the medical entity tag data stream from the upstream data processing pipeline in real time.

[0128] Secondly, high-frequency medical entity tags that cannot be mapped in the existing medical ontology knowledge graph are selected as candidate entities. The monitoring module performs a matching query on each received medical entity tag against the existing medical ontology knowledge graph. The matching query is achieved by calculating the similarity between the tag text and the names of existing entities in the knowledge graph. For example, edit distance, Jaccard coefficient, or a deep learning-based text similarity model can be used to calculate the similarity score. If the similarity score between a medical entity tag and any existing entity in the knowledge graph is lower than a preset threshold, it is considered that the tag cannot find a corresponding entity in the knowledge graph. To avoid introducing low-frequency noise data into the candidate entity set, the monitoring module counts the occurrence frequency of each unmapped medical entity tag. Only when the occurrence frequency of a medical entity tag is higher than a preset frequency threshold is it identified as a candidate entity. For example, the frequency threshold can be set to occur more than 10 times in the past week. The monitoring module stores the identified candidate entities and their related information, such as occurrence frequency and context, in a candidate entity database.

[0129] Then, the system receives user confirmation of candidate entities through a human-computer collaborative verification interface. The system provides a web-based human-computer collaborative verification interface to display candidate entities and their related information. This interface periodically pushes candidate entity verification tasks to users with medical knowledge, such as clinicians or knowledge administrators. Users can view the candidate entity's name, frequency of occurrence, context, and related clinical event records on the interface. Users can determine whether the candidate entity is a valid new medical concept based on their medical knowledge and experience, and choose to "confirm" or "reject". If the user chooses "confirm", they can further specify the candidate entity's category and attributes in the knowledge graph. For example, a user can confirm that "novel fusion device type A" is an "endophyte-fusion device" and specify its manufacturer, model, and other attributes. The human-computer collaborative verification interface records the user's actions and stores the verification results in a verification result database.

[0130] Finally, the medical ontology knowledge graph is automatically updated based on the confirmation operation. The system periodically checks the verification result database to find candidate entities that have been confirmed by the user. For each confirmed candidate entity, the system automatically performs a knowledge graph update operation. The update operation includes creating a new entity node in the knowledge graph and setting the type and attribute values ​​of the node according to the category and attributes specified by the user. At the same time, the system also establishes relationship connections between this new entity and other related entities. For example, if the user confirms that "Novel Fusion Device Type A" is an "Implant-Fusion Device" and specifies that it is related to "Lumbar Fusion Surgery", the system will create a "Novel Fusion Device Type A" entity node in the knowledge graph, set its type to "Implant-Fusion Device", and establish an "Applied To" relationship between this node and the "Lumbar Fusion Surgery" entity node. The updated knowledge graph can be used by subsequent semantic entity grounding modules, thereby improving the accuracy and intelligence of data analysis.

[0131] Optionally, the system can use machine learning models, such as a Transformer-based named entity recognition model, to automatically identify the categories and attributes of candidate entities, thereby reducing the workload of manual verification. This model can be trained using a large amount of medical text data to learn the naming patterns and contextual relationships of medical entities. When identifying candidate entities, the model can predict their most likely categories and attributes and display them as recommended options on the human-machine collaborative verification interface for user reference and confirmation.

[0132] Through the above steps, the system can continuously monitor and update the medical ontology knowledge graph, promptly identify and integrate new medical concepts, thereby improving the accuracy and intelligence of data analysis and better meeting the needs of clinical research. This embodiment achieves the dynamic evolution of the medical ontology knowledge graph through monitoring, screening, verification, and updating, solving the problem of knowledge graph lag caused by the continuous updating of medical knowledge. This enables the system to better understand and process new medical data and information, improving the efficiency and accuracy of research data processing.

[0133] In some embodiments, before extracting and processing multimodal clinical research data based on directed acyclic graph analysis flow, the system first authorizes and verifies the data processing request and its accompanying research purpose identifier through an access control module, according to a preset authorization policy. To achieve this process, the access control module employs the following technical means:

[0134] First, the preset authorization policy is stored in the authorization policy database. This database uses a relational database, such as PostgreSQL, and its table structure includes fields such as user ID, research purpose identifier, accessible data types, and access permissions. The user ID identifies the request initiator, the research purpose identifier corresponds to a research project that has passed ethical review, the accessible data types limit the range of data the user can access, and the access permissions define the type of operation the user can perform on the data, such as read or write. A specific example is a researcher with user ID U001 who participated in a project with research purpose identifier P001. This project authorizes them to access "image data" and "laboratory test data," and they have read permissions.

[0135] Secondly, when the data processing and report generation module prepares to extract and process multimodal clinical research data based on the directed acyclic graph analysis flow, it sends a data access request to the access control module. This request includes the current user's user ID, the accompanying research purpose identifier, and a list of data types requested for access. For example, a researcher with user ID U001 initiates a project with research purpose identifier P001 and requests access to data types including "imaging data," "laboratory test data," and "genomic data."

[0136] Next, upon receiving a data access request, the access control module will verify it according to a preset authorization policy. The verification process includes the following steps:

[0137] 1. Check if the user ID in the request exists in the authorization policy database. If it does not exist, reject the access request and return an error message.

[0138] 2. Check if the research purpose identifier in the request exists in the authorization policy database and corresponds to the user ID in the request. If it does not exist or does not correspond, reject the access request and return an error message.

[0139] 3. Check if each data type in the requested data type list is within the data type range allowed by the user ID and research purpose identifier in the authorization policy database. If any data type is not allowed, the access request is rejected and an error message is returned. If all the above verification steps pass, the access control module authorizes the data access request, allowing the data processing and report generation module to continue performing subsequent data extraction and processing operations. If any verification step fails, the access control module rejects the data access request, returns the corresponding error message, and prevents subsequent data operations.

[0140] Optionally, the access control module can adopt the attribute-based access control (ABAC) model instead of the traditional role-based access control (RBAC) model. The ABAC model allows for more flexible definition of authorization policies; for example, it can dynamically determine whether to authorize access requests based on attributes such as the user's department, the user's title, and the sensitivity of the data.

[0141] Through the above methods, the access control module ensures that only authorized users can access data consistent with their research objectives, thereby guaranteeing data security, complying with research ethics, and preventing data misuse or leakage. This solution internalizes research ethics requirements as part of the system's technical architecture, providing a reliable technical foundation for conducting large-scale, multi-center collaborative research.

[0142] In some embodiments, data processing requests, research purpose identifiers, and summary information of research data analysis reports are recorded in an audit log based on distributed ledger technology. Specifically, a data processing request can be a structured data object containing user authentication information, a request timestamp, the requested data range, and the type of operation. For example, when a user initiates a data query request, the system generates a JSON object containing the username of the requester, the specific time of the request, the database table name to be queried, the field names to be queried, and the query conditions. The research purpose identifier is a unique string associated with a research project approved by an ethics committee, used to identify the legitimate purpose of data access. This identifier is automatically generated by the system when a research project is initiated and is bound to project details such as project name, principal investigator, and research scope. The summary information of the research data analysis report refers to metadata extracted from the final generated research report, such as the report generation time, the dataset version used, the name of the analysis method, and the p-value of the statistical significance index in the final results. This summary information helps subsequent auditors quickly understand the overall picture of data processing and analysis. To implement audit logs based on distributed ledger technology, the system can employ mature blockchain platforms such as Hyperledger Fabric or Ethereum. In Hyperledger Fabric, each data access event is encapsulated as a transaction, containing a data processing request, a research objective identifier, and a summary of the research data analysis report. These transactions are submitted to a sorting service within the blockchain network for sorting, and then verified and endorsed by each peer node. Once a transaction is verified, it is packaged into a block and added to the end of the blockchain. Due to the immutability of the blockchain, all data access and processing records can be securely stored and traced.

[0143] Optionally, audit logs can be implemented using either a private blockchain or a consortium blockchain to better meet the data privacy and security requirements of medical institutions. A private blockchain, controlled by a single institution, allows for customized consensus algorithms and access control policies, enabling stricter control over data access. A consortium blockchain, maintained jointly by multiple institutions, enables data access auditing and oversight in multi-center collaborative research. When generating audit log entries, the system can use hash algorithms to encrypt data processing requests and summary information in research data analysis reports to protect user privacy. For example, user identity information can be one-way hashed, storing only the hash value to avoid directly exposing sensitive user information.

[0144] The audit log is stored on the blockchain, leveraging the immutability of blockchain technology to ensure the integrity and reliability of audit information. Auditors can trace every data access and processing operation by querying transaction records on the blockchain, thereby ensuring the compliant use of data and providing credible evidence for potential ethical issues.

[0145] like Figure 2 As shown, this embodiment of the invention provides a multimodal clinical research data processing system based on neural-symbolic hybrid reasoning. This system aims to achieve automated and intelligent processing and analysis of multimodal clinical research data. The system includes:

[0146] The instruction receiving module M100 is configured to receive natural language analysis instructions for multimodal clinical research data input from user terminals. This instruction receiving module M100 can be a web user interface, for example, built using the popular front-end framework React and deployed on a standard HTTP server. Users can access this interface through a browser and enter natural language instructions in a text box. The content of the text box is sent to the system's backend server via the HTTPS protocol. Optionally, the instruction receiving module M100 can also be a mobile application (App) running on iOS or Android operating systems, communicating with the backend server through a mobile network API interface. Alternatively, the instruction receiving module M100 can be a speech recognition interface, where users input instructions via voice, the speech recognition engine converts the speech into text, and then the text is sent to the backend server. For example, the user inputs "Analyze the blood glucose changes in diabetic patients receiving drug A and drug B treatment."

[0147] The neural symbolic hybrid reasoning module M200, connected to the instruction receiving module M100, is configured to first parse the natural language analysis instruction into a structured formal problem description data object through the language understanding unit, and then select one or more data analysis algorithms from a pre-set algorithm library based on a preset expert rule base and the formal problem description data object through the symbolic reasoning unit. The language understanding unit can be based on a pre-trained large language model (LLM), for example, using HuggingFace's Transformers library to load a BERT model fine-tuned on medical literature. This model receives the natural language instruction text and outputs a structured data object in JSON format, containing the analysis intent and parameter entities. For example, for the instruction "Analyze the changes in blood glucose levels in diabetic patients treated with drug A and drug B", the language understanding unit might output the following JSON: {"analysisIntent": "group_comparison", "parameters": {"disease":"diabetes", "treatment": ["drug A", "drug B"], "metric": "blood glucose"}}. The symbolic reasoning unit can be a rule-based decision engine, for example, using the Drools rule engine or directly implemented using if-else statements written in Python. The expert rule base is a collection of rules constructed by medical statisticians, defining the algorithm to be selected under different analytical intentions and data types. For example, a rule might be: if the analysisIntent is "group_comparison" and the metric is "continuous_variable", then choose the t-test. The algorithm library contains implementations of various data analysis algorithms, such as t-tests, ANOVA, and linear regression, which can be implemented using Python's SciPy library or statistical functions in R. Optionally, the language understanding unit can also use other natural language processing techniques, such as rule-based parsers or statistical parsers. The symbolic reasoning unit can also use fuzzy logic or probabilistic graphical models to handle uncertainty.

[0148] The Analysis Flow Generation Module M300, connected to the Neural Symbolic Hybrid Reasoning Module M200, is configured to generate a Directed Acyclic Graph (DAG) analysis flow based on one or more selected data analysis algorithms and formalized problem description data objects. This DAG defines the data processing steps and the dependencies between them. The Analysis Flow Generation Module M300 can be a Python program that uses the NetworkX library to create and manipulate DAGs. Each node in the DAG represents a data processing step, such as data filtering, data transformation, statistical computation, or visualization. Edges in the DAG represent data dependencies; for example, the output of one node serves as the input of another. For instance, for an analysis task involving a t-test, the Analysis Flow Generation Module M300 might generate a DAG containing the following nodes: 1. Screening diabetic patients; 2. Grouping: Drug A group and Drug B group; 3. Calculating the mean blood glucose level for each group; 4. Performing the t-test; 5. Generating a report. Optionally, the Analysis Flow Generation Module M300 can also use a graphical interface, allowing users to manually edit and adjust the DAG.

[0149] The data processing and report generation module M400, connected to the analysis flow generation module M300, is configured to extract and process multimodal clinical research data stored in a database, containing at least two types of technically meaningful data, based on a directed acyclic graph (DAG) analysis flow, and generate a research data analysis report including quantitative analysis indicators. This module first parses the DAG and executes the data processing steps of each node according to the order defined by the DAG. The data processing steps include extracting data from the database, cleaning, transforming, and calculating the data, and passing the results to the next node. The database can be a relational database, such as MySQL or PostgreSQL, or a NoSQL database, such as MongoDB or Cassandra. Multimodal clinical research data can include electronic medical record data, imaging data, genomic data, physiological signal data, etc. Data processing can be implemented using Python's Pandas library or R's dplyr library. Report generation can use Python's ReportLab library or R's knitr library to generate reports containing quantitative analysis indicators, such as t-test p-values ​​and confidence intervals. The report can be presented in PDF, HTML, or Word format. For example, the module executes a Directed Acyclic Graph (DAG), first extracting data on diabetic patients from the database, then dividing the patients into drug A and drug B groups, calculating the mean blood glucose level for each group, performing a t-test, and generating a report containing p-values. Optionally, the M400 data processing and report generation module can also use distributed computing frameworks, such as Spark or Hadoop, to handle large-scale datasets.

[0150] Through the collaborative work of its various modules, this system enables automated and intelligent processing and analysis of multimodal clinical research data. Researchers only need to input natural language commands, and the system can automatically complete steps such as data extraction, data processing, algorithm selection, and report generation, significantly improving research efficiency.

[0151] In some embodiments, the system further includes a semantic entity grounding module, configured to map the parameter entities contained in the formal problem description data object to specific data fields or data codes in the database before the symbolic reasoning unit of the neural symbolic hybrid reasoning module M200 selects a data analysis algorithm. The function of this semantic entity grounding module is to convert parameter entities parsed from natural language analysis instructions, such as disease names, drug names, and rating scale names, into specific data fields or data codes that can be identified and retrieved in the database. This establishes a bridge from user intent to data reality, ensuring that subsequent data analysis can accurately find the required data in the database. One implementation is that the semantic entity grounding module includes a deterministic mapping submodule and a probabilistic disambiguation submodule. The deterministic mapping submodule mainly handles parameter entities that can be directly mapped through a pre-built medical ontology knowledge graph. The probabilistic disambiguation submodule is used to handle cases where there are multiple mapping results or where direct mapping is not possible.

[0152] First, the deterministic mapping submodule receives parameter entities from the formal problem description data object. For example, suppose the received parameter entity is "lumbar spinal stenosis". The deterministic mapping submodule calls the built-in medical ontology knowledge graph for querying. This medical ontology knowledge graph can be a graph database built on an existing medical knowledge base, such as a network of medical concepts and relationships represented using the Resource Description Framework (RDF), or a medical terminology table built on a relational database. The knowledge graph stores medical terms and their corresponding standard codes, such as ICD-10 codes, MeSH vocabularies, etc. After querying "lumbar spinal stenosis" in the knowledge graph, suppose the knowledge graph returns its corresponding ICD-10 code M48.0. At this point, the deterministic mapping submodule has completed the mapping from the parameter entity "lumbar spinal stenosis" to the database code M48.0. The database query module can then use this code to retrieve relevant patient data in the database. For parameter entities that cannot be directly mapped, or parameter entities with multiple possible mapping results, the probabilistic disambiguation submodule is activated. For example, a user's input might contain the parameter entity "Dr. Li's patient," while there might be multiple "Dr. Li"s in the system. In this case, the probabilistic disambiguation submodule needs to determine which "Dr. Li" the user is referring to based on contextual information. The probabilistic disambiguation submodule obtains the current user's contextual information, such as the user's role, department, and historical operation records. Suppose the current user is a neurosurgeon and has recently been frequently viewing data related to "Dr. Li Ming." Based on this information, the probabilistic disambiguation submodule can calculate the probability that "Dr. Li Ming" is the "Dr. Li" the user is referring to. If this probability exceeds a set threshold, such as 0.9, then it is determined that "Dr. Li" here refers to "Dr. Li Ming," and its employee ID 1024 is used as the mapping result. In other implementations, the probabilistic disambiguation submodule can also utilize natural language processing techniques, such as named entity recognition and relation extraction, to extract more contextual information from the user's instructions, such as Dr. Li's title and areas of expertise, thereby performing disambiguation more accurately. Furthermore, external knowledge bases, such as a hospital's doctor information database, can be combined to assist the disambiguation process.

[0153] Optionally, the semantic entity grounding module can also provide a user interface that allows users to manually select the mapping target for the parameter entity. For example, when multiple possible mapping results exist, the system can display these results on the interface for the user to select. This approach can improve the accuracy of the mapping, but it also requires additional user intervention. After completing the mapping of the parameter entities, the semantic entity grounding module returns the mapping results to the neural symbolic hybrid inference module M200. The neural symbolic hybrid inference module M200 can then use these mapping results to select a suitable data analysis algorithm and generate a directed acyclic graph analysis flow.

[0154] Through this semantic entity grounding module, the system can accurately convert user-input natural language commands into query conditions that the database can understand, thereby ensuring that subsequent data analysis is based on correct data. This greatly improves the efficiency and accuracy of data analysis and lowers the professional knowledge threshold required for users to use the system.

[0155] In some embodiments, for scenarios where the database is a hybrid time-series-graph database, the data processing and report generation module M400 includes a query federated optimizer. The function of the query federated optimizer is to decompose the received data extraction request into sub-requests targeting different types of databases in the hybrid time-series-graph database, and to merge the execution results of the sub-requests.

[0156] One implementation involves the query federated optimizer first parsing the received data extraction request, identifying the data types involved and the relationships between the data. For example, if the data extraction request is "find all patients diagnosed with lumbar spinal stenosis and obtain their VAS scores within three months post-surgery," the query federated optimizer would analyze that the diagnostic information for "lumbar spinal stenosis" is stored in a graph database, while the time-series data for "VAS scores" is stored in a time-series database. Subsequently, the query federated optimizer would decompose the data extraction request into two sub-requests: a graph traversal sub-request targeting the graph database to find all patient IDs diagnosed with lumbar spinal stenosis; and a time-series scan sub-request targeting the time-series database to obtain the VAS score data for these patient IDs within three months post-surgery. The graph traversal sub-request can be implemented using the query language of the graph database. For example, if the graph database is Neo4j, the following query can be written using the Cypher query language: `MATCH (p:Patient)-[:HAS_DIAGNOSIS]->(d:Diagnosis {name: 'Lumbar Spinal Stenosis'}) RETURN p.patient_id`. This query searches the graph database for all patient nodes related to the diagnosis of "lumbar spinal stenosis" and returns the patient IDs of these nodes. Time-series scan sub-requests can be implemented using the query language of time-series databases. For example, if the time-series database is InfluxDB, the following query can be written using the Flux query language: `from(bucket: "clinical_data") |> range(start: treatment_start, stop: treatment_start + 3mo) |> filter(fn: (r) => r.patient_id IN ["patient_id_1", "patient_id_2", ...]) |> filter(fn: (r) => r._measurement == "vas_scores") |> filter(fn: (r) => r._field == "value")`. This query retrieves the VAS scores for a specific list of patient IDs within three months starting from the treatment start time `treatment_start` in the time-series database. `treatment_start` can be obtained from the graph database query results or determined through other methods. The query federated optimizer will send these two sub-requests to their respective databases for execution. After receiving the execution results of the two sub-requests, the query federated optimizer will merge the results.For example, a list of patient IDs obtained from a graph database can be associated with VAS score data obtained from a time-series database to generate a dataset containing patient IDs, timestamps, and VAS score values. Optionally, the query federated optimizer can further optimize sub-requests to improve query efficiency. For example, it can select the optimal query path based on the indexing of the graph database and the time-series database; it can also merge multiple time-series scan sub-requests into one, reducing the number of interactions with the time-series database. Furthermore, the query federated optimizer can provide a unified data access interface, shielding upper-layer applications from the differences in the underlying databases. In this way, upper-layer applications only need to call the query federated optimizer's interface to obtain the required data, without needing to know which database the data is stored in.

[0157] Through the above implementation, the query federated optimizer can effectively decompose data extraction requests into sub-requests targeting the time-series-graph hybrid database and merge the execution results of the sub-requests, thereby achieving rapid and efficient extraction of multimodal clinical research data. This method fully leverages the respective advantages of time-series and graph databases, improving the efficiency and flexibility of data querying. Simultaneously, the unified data access interface simplifies the development of upper-layer applications and reduces system complexity.

[0158] In some embodiments, the system further includes an access control module configured to authorize and verify data processing requests and their accompanying research purpose identifiers according to a preset authorization policy before the data processing and report generation module M400 extracts and processes multimodal clinical research data.

[0159] The access control module can be deployed on a standalone server or integrated into an existing server as part of the data service layer. Its function is to verify user access permissions, prevent unauthorized data access and processing, and ensure data security and compliance with research ethics. First, preset authorization policies are stored in a database. This database can be a relational database such as PostgreSQL or a NoSQL database such as MongoDB. The specific form of the authorization policy can be an Access Control List (ACL) or a Role-Based Access Control (RBAC) model, and can be extended to support Attribute-Based Access Control (ABAC). In this embodiment, an extended RBAC model is used, where roles are associated not only with users but also with research projects, and the granularity of permissions can be refined to the data field level. Data processing requests are initiated by the user through the user interface. For example, a doctor might want to query the changes in VAS scores of patients with lumbar spinal stenosis after minimally invasive decompression surgery. When submitting a request, the user needs to submit a research purpose identifier, such as "LSS-MIS-VAS-2024". This research purpose identifier is a unique identifier assigned after approval by the ethics committee during the research project initiation process and is recorded in the system's metadata management module. When the data processing and report generation module M400 receives a data extraction request, it first forwards the request to the access control module. The access control module first verifies the user's identity, which can be done through various authentication methods such as username / password, multi-factor authentication, or single sign-on. Assuming the user verifies with a username and password, the system obtains the user's role information, such as "neurosurgeon." Next, the access control module queries the authorization policy database to determine whether the user role and research project identifier have the necessary permissions to access the relevant data. The query conditions include the user role, research project identifier, and the data type requested. The authorization policy database may contain the following entries:

[0160] User Role Research project identifier Data types Access permissions ... ... ... ... Neurosurgeon LSS-MIS-VAS-2024 Patient basic information Read Neurosurgeon LSS-MIS-VAS-2024 VAS score Read Data Administrator LSS-MIS-VAS-2024 Surgical Record Reading and writing ... ... ... ...

[0161] Based on these entries, the access control module checks whether the user has permission to access "Patient Basic Information" and "VAS Score". If a user attempts to access "Surgical Records", but the authorization policy indicates that the user role does not have permission to access this data type, the access control module will reject the request and return an error message to the data processing and report generation module M400.

[0162] Optionally, the access control module can also support fine-grained access control. For example, for VAS score data, users can be restricted to accessing score data within a specific time frame, or only to score data for a specific patient group. This fine-grained access control can be achieved by adding additional conditions to the authorization policy database.

[0163] In other implementations, the access control module can also be integrated with the data anonymization module. Before a user accesses sensitive data, the access control module can call the data anonymization module to process the data, for example, replacing patient names with anonymous IDs or obfuscating patient ages. The specific implementation of authorization verification by the access control module can be as follows: A separate microservice can be developed to provide access control functionality via a RESTful API. Before performing data extraction, the data processing and report generation module M400 calls this API for authorization verification. The API request parameters include user identity information, research purpose identifier, and the data type requested. The API return result indicates whether authorization was successful. If authorization fails, the data processing and report generation module M400 will stop performing the data extraction operation and return an error message to the user. Furthermore, the access control module should also have a certain degree of fault tolerance. For example, when the authorization policy database fails, the access control module can adopt a "fail-safe" strategy, i.e., rejecting all access requests by default to ensure data security. Based on the above description, those skilled in the art can understand the specific implementation of the access control module, including its interaction with the data processing and report generation module M400, and how it performs authorization verification of data processing requests according to preset authorization policies.

[0164] Access control modules effectively protect the security of multimodal clinical research data, preventing unauthorized access and leakage, and ensuring ethical compliance in the research process. Furthermore, research purpose identifiers link data access behavior to specific research projects, enabling full traceability of data usage.

[0165] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A multimodal clinical research data processing method based on neural symbolic hybrid reasoning, characterized in that, This includes the following steps performed by a computer: Receive natural language analysis commands from user terminals for multimodal clinical research data; Using a neural symbolic hybrid reasoning engine, the natural language analysis instructions are first parsed into structured formal problem description data objects by the language understanding unit. Then, the symbolic reasoning unit selects one or more data analysis algorithms from a preset algorithm library based on a preset expert rule base and the formal problem description data objects. Based on one or more selected data analysis algorithms and the formal problem description data object, a directed acyclic graph analysis flow is generated that defines the data processing steps and the dependencies between the data processing steps. Based on the directed acyclic graph analysis flow, multimodal clinical research data containing at least two types of technically meaningful data stored in a database are extracted and processed, and a research data analysis report containing quantitative analysis indicators is generated.

2. The method according to claim 1, characterized in that, The formal problem description data object contains at least one analysis intent and at least one parameter entity.

3. The method according to claim 2, characterized in that, Before selecting a data analysis algorithm using the symbolic reasoning unit, the method further includes: Using the semantic entity grounding module, the parameter entities in the formal problem description data object are mapped to specific data fields or data codes in the database.

4. The method according to claim 3, characterized in that, The semantic entity grounding module queries and maps the parameter entities using a pre-built medical ontology knowledge graph.

5. The method according to claim 4, characterized in that, Also includes: During the mapping process, when the parameter entity has multiple mapping results or no mapping result, the mapping target of the parameter entity is determined by combining the user's historical operation data or the context information of the current scientific research project.

6. The method according to claim 1, characterized in that, The database is a time-series graph hybrid database; the steps for extracting the multimodal clinical research data based on the directed acyclic graph analysis flow specifically include: By querying the federated optimizer, the data extraction request is decomposed into a graph traversal sub-request for the graph database portion of the time-series graph hybrid database and a time-series scan sub-request for the time-series database portion, and the data returned after executing the sub-requests is merged.

7. The method according to claim 1, characterized in that, Before receiving the natural language analysis instruction, the method also includes: Through upstream data processing pipelines, the multimodal clinical research data from different sources are uniformly converted into a clinical event data structure that includes timestamps, event types, and event content.

8. The method according to claim 7, characterized in that, The upstream data processing pipeline also includes: Using a natural language processing model, standardized medical entity labels are automatically extracted and labeled from the text-based clinical event data.

9. The method according to claim 8, characterized in that, The method further includes: Monitoring the upstream data processing pipeline to extract high-frequency medical entity tags that cannot be mapped in the existing medical ontology knowledge graph as candidate entities; and The system receives user confirmation of the candidate entity through a human-computer collaborative verification interface and automatically updates the medical ontology knowledge graph based on the confirmation.

10. A multimodal clinical research data processing system based on neural symbolic hybrid reasoning, characterized in that, include: The instruction receiving module is configured to receive natural language analysis instructions for multimodal clinical research data input from the user terminal. The neural symbolic hybrid reasoning module, connected to the instruction receiving module, is configured to first parse the natural language analysis instruction into a structured formal problem description data object through the language understanding unit, and then select one or more data analysis algorithms from a preset algorithm library through the symbolic reasoning unit based on the preset expert rule base and the formal problem description data object. The analysis flow generation module, connected to the neural symbolic hybrid reasoning module, is configured to generate a directed acyclic graph analysis flow that defines the data processing steps and the dependencies between the data processing steps, based on one or more selected data analysis algorithms and the formal problem description data object. The data processing and report generation module is connected to the analysis flow generation module and is configured to extract and process multimodal clinical research data stored in the database, which contains at least two types of data with technical significance, based on the directed acyclic graph analysis flow, and generate a research data analysis report containing quantitative analysis indicators.

Images