A method and device for constructing a multimodal and multi-perspective knowledge graph of depression

Abstract

The present invention discloses a method and device for constructing a multimodal and multi-perspective knowledge graph of depression, which relates to the field of knowledge graphs. The method includes collecting multimodal and multi-perspective data related to adolescent depression, and performing feature representation on the collected data after preprocessing; characterizing the multimodal and multi-perspective data, and constructing a multi-layer knowledge organization basic framework for adolescent depression, wherein the knowledge organization basic framework includes an attribute layer, a concept layer, an event layer, and a narrative layer; according to the multi-layer knowledge organization basic framework for adolescent depression, constructing multimodal diagnosis and treatment attribute and concept layer knowledge, and representing and constructing diagnosis and treatment event layer knowledge based on a matter-of-fact graph; according to the multi-layer knowledge organization basic framework for adolescent depression, constructing and integrating narrative layer multi-perspective knowledge based on knowledge tracing. The present application can make the constructed knowledge graph more in line with clinical practice.

Description

Technical Field

[0001] The present application relates to the field of knowledge graphs, and specifically to a method and device for constructing a multimodal and multi-perspective knowledge graph of depression. Background Art

[0002] Adolescent depression has become a major health and social problem. However, the current psychiatric practitioners are still "scarce". The great challenge facing the mental health care system is how to meet the early and accurate diagnosis and treatment needs of the majority of adolescent depression patients with scarce medical human resources. Artificial intelligence (AI) technology represented by knowledge graphs is rapidly developing into a new generation of information technology and service formats for discovering new knowledge, creating new value and improving new capabilities; this technology has the potential for continuous excavation and in-depth utilization in the field of diagnosis and treatment of adolescent depression. Especially in adolescent depression, a complex mental illness with unknown etiology and efficacy outcome mechanism, its occurrence and development may involve multi-level and multi-modal knowledge, mainly including behavioral phenotype, brain network and epigenetic aspects. Behavioral phenotype: that is, the external symptoms of the disease, such as low voice, depressed facial expression, low mood, loss of interest, self-harm and even suicidal behavior, insomnia, loss of appetite, etc. Brain network: The disease involves neurotransmitter imbalance, abnormal activity in brain regions, and disordered brain structure and functional network connections, which causes abnormal behavioral phenotypes such as emotions, behaviors, learning and memory in patients. Epigenetics: refers to the fact that the DNA (Deoxyribo Nucleic Acid) sequence remains unchanged, but gene modifications (such as common DNA methylation modifications) change, which in turn causes changes in brain networks and behavioral phenotypes.

[0003] Based on AI technology, we can deeply explore the multi-level and multi-modal associations between the behavioral phenotypes, brain networks and epigenetics of adolescent depression, which is expected to clarify the pathogenesis and efficacy mechanisms, thereby assisting in the precise diagnosis and treatment of the disease. It should be noted that AI-assisted diagnosis technology for adolescent depression needs to originate from and return to the clinic, pay close attention to and focus on clinical pain points, so as to more accurately assist in diagnosis and treatment. At present, there are two major pain points in the clinical diagnosis and treatment of adolescent depression: (1) Insufficient mining of behavior-brain-epigenetic multimodal relationships leads to unclear diagnosis and treatment mechanisms: The diagnosis and efficacy prediction mechanisms of adolescent depression are still unclear, mainly due to insufficient mining of behavioral phenotype-brain network-epigenetic multimodal associations. The "behavioral phenotype-brain network-epigenetic" relationship of adolescent depression is complex and interrelated. However, most current AI studies only use a single modality to predict patient diagnosis and treatment, which leads to relatively low prediction accuracy. (2) Incomplete acquisition of multi-perspective disease information leads to inaccurate diagnosis and treatment: Adolescents belong to a special group, and their social activities are mainly in school and family settings. The diagnostic accuracy of this disease is highly dependent on the "multi-perspective" disease information (mainly natural language data related to the disease) provided by doctors, as well as patients, family members, and school authorities. However, the incomplete acquisition of "multi-perspective" disease information in the current clinical diagnosis and treatment of adolescent depression is one of the main clinical pain points that affect the accurate diagnosis and treatment of this disease. The information that doctors obtain from patients is relatively single-perspective, and the patient's performance that can be observed from the perspective of parents and school authorities (teachers, classmates, etc.) cannot be fully known by doctors. This is an important factor that leads to misdiagnosis, missed diagnosis, and inaccurate prediction of efficacy outcomes.

[0004] The intelligent assisted diagnosis and treatment methods for adolescent depression are still in the exploratory and initial stages. Currently, most intelligent treatment methods for adolescent depression use a single modality or a purely data-driven approach, lacking the assistance of multi-perspective and multi-modal knowledge, making it difficult to provide explainable and evidence-based decision-making results and basis, and lacking the ability to integrate and reference unreliable and inaccurate information.

[0005] Therefore, how to construct a knowledge graph that integrates the multi-modalities of "behavior-brain-genetics" and combines the multi-perspectives of "doctor-patient-family-school" has become a problem that needs to be solved urgently. Summary of the invention

[0006] The present application provides a method and device for constructing a multimodal and multi-perspective knowledge graph for depression, which can make the constructed knowledge graph more in line with clinical practice.

[0007] In a first aspect, an embodiment of the present application provides a method for constructing a multimodal and multi-perspective knowledge graph for depression, and the method for constructing a multimodal and multi-perspective knowledge graph for depression includes:

[0008] Collect multimodal and multi-perspective data related to adolescent depression, and perform feature representation after preprocessing the collected data;

[0009] Characterize multimodal and multi-perspective data and construct a multi-layer knowledge organization framework for adolescent depression, where the knowledge organization framework includes attribute layer, concept layer, event layer and narrative layer;

[0010] According to the basic framework of multi-layer knowledge organization of adolescent depression, multimodal diagnosis and treatment attribute and concept layer knowledge is constructed, and the diagnosis and treatment event layer knowledge is represented and constructed based on the event-based graph.

[0011] According to the basic framework of multi-layer knowledge organization of adolescent depression, narrative layer multi-perspective knowledge is constructed and integrated based on knowledge tracing.

[0012] In combination with the first aspect, in one implementation,

[0013] The multimodal and multi-perspective data include behavioral phenotype data, brain network data, and epigenetic data;

[0014] The phenotypic data includes multi-perspective symptom corpus data, demographic and clinical symptom data, and voice and facial expression data.

[0015] In combination with the first aspect, in one implementation, characterizing the multimodal multi-view data specifically includes:

[0016] A pre-trained model is used to encode the features of multimodal and multi-view data, and the correlation between behavioral phenotype data and brain network data, brain network data and epigenetic data, and behavioral phenotype data and epigenetic data is established based on similar features;

[0017] Different groups of similar patients are established based on multi-level user portrait technology, and multi-dimensional vector fusion is performed on the group basis to map different groups into the same semantic vector space.

[0018] In combination with the first aspect, in one implementation,

[0019] The attribute layer is used to describe the disease based on data attributes;

[0020] The concept layer is used to model the medical concept relationship of depression;

[0021] The event layer is used to describe adolescent depression diagnosis and treatment events and patients' daily life events based on the event graph, and supports the expression of event association relationships;

[0022] The narrative layer is used to perform knowledge representation modeling on the characteristics of patient diagnosis and treatment information.

[0023] In combination with the first aspect, in one embodiment, the multi-layer knowledge organization basic framework of adolescent depression is used to construct multimodal diagnosis and treatment attributes and concept layer knowledge, and to represent and construct diagnosis and treatment event layer knowledge based on the event map. Among them, the construction of multimodal diagnosis and treatment attributes and concept layer knowledge specifically includes:

[0024] For unstructured clinical guidelines and medical literature related to the diagnosis and treatment of adolescent depression, paragraphs are divided according to the standard format of medical documents;

[0025] Text classification of clinical guideline paragraphs based on medical topic categories based on language representation models and long short-term memory networks;

[0026] In view of the different references of clinical guidelines and medical literature entities in multiple documents, a knowledge enhancement method is used to extract chapter-level fine-grained knowledge from clinical guidelines and medical literature.

[0027] Based on the label classification method, for the extracted results, the probability of the relationship type r between any entity pair is calculated through a bilinear function to obtain the evaluation result;

[0028] Among them, the knowledge enhancement method includes knowledge injection and knowledge extraction. The knowledge injection is to model the semi-structured document knowledge in the text and find the corresponding matching knowledge graph knowledge. The knowledge extraction is to use the matched knowledge graph knowledge as background knowledge to perform fine-grained extraction of medical entities and relationships in the text.

[0029] In combination with the first aspect, in one embodiment, the multi-layer knowledge organization basic framework of adolescent depression is used to construct multimodal diagnosis and treatment attribute and concept layer knowledge, and to represent and construct diagnosis and treatment event layer knowledge based on the event graph. Among them, the representation and construction of diagnosis and treatment event layer knowledge based on the event graph specifically includes:

[0030] Model the knowledge ontology of diagnosis and treatment events from different sources and stages;

[0031] Based on R2RML, structured diagnosis and treatment event extraction is performed;

[0032] Perform unstructured disease description event recognition and relationship extraction based on background knowledge attention.

[0033] In combination with the first aspect, in one implementation,

[0034] The diagnosis and treatment event graph knowledge ontology is used to represent the event type, attribute and association according to the event graph and the event association mechanism in complex event processing;

[0035] The diagnosis and treatment event graph knowledge ontology is divided into behavior representation events and diagnosis and treatment process events in terms of event types.

[0036] In combination with the first aspect, in one implementation, the construction and integration of narrative layer multi-perspective knowledge based on knowledge tracing based on the basic framework of multi-layer knowledge organization of adolescent depression specifically includes:

[0037] Model the narrative layer based on the representation method of knowledge tracing, and realize the narrative layer knowledge representation of adolescent depression diagnosis and treatment;

[0038] Use evidence theory to construct and reason knowledge, and realize narrative knowledge construction of disease description from multiple perspectives;

[0039] Aiming at multimodal and heterogeneous clinical complex causal relationships, a knowledge reasoning method based on DS theory is used to identify and construct relationships.

[0040] In combination with the first aspect, in one implementation, the knowledge reasoning method based on DS theory performs relationship identification and construction, specifically including:

[0041] A multimodal pre-trained model is used to encode the features of multimodal and multi-view data, so that the multi-source heterogeneous features can be mapped into the same high-dimensional vector space;

[0042] A transformer encoder is constructed based on the feature vector to capture the mapping relationship between the feature vector and the narrative causal relationship;

[0043] Take the feature vector as input and update the transformer encoder parameters by stochastic gradient descent;

[0044] A multimodal pre-training model is used to take the narrative encoding obtained in the process of constructing narrative knowledge of multi-perspective disease description as a feature vector;

[0045] Converting the support confidence belief distribution of each narrative causal relationship into a causal relationship adjacency matrix, wherein the values ​​of the adjacency matrix represent the causal relationship between the two narratives;

[0046] Based on the feature vector and adjacency matrix of the narrative, GNN is used to learn the transformation between narratives in the clinical decision-making process, and the narrative relationship is updated to obtain a new relationship matrix.

[0047] In a second aspect, an embodiment of the present application discloses a device for constructing a multimodal and multi-perspective knowledge graph for depression, wherein the device for constructing a multimodal and multi-perspective knowledge graph for depression comprises:

[0048] The acquisition module is used to collect multi-modal and multi-view data related to adolescent depression, and perform feature representation after pre-processing the collected data;

[0049] A construction module is used to characterize multimodal and multi-perspective data and construct a multi-layer knowledge organization framework for adolescent depression, wherein the knowledge organization framework includes an attribute layer, a concept layer, an event layer, and a narrative layer;

[0050] Create a module for constructing multimodal diagnosis and treatment attribute and concept-level knowledge based on the basic framework of multi-layer knowledge organization of adolescent depression, and for representing and constructing diagnosis and treatment event-level knowledge based on the event-diagnosis graph;

[0051] The execution module is used to construct and integrate narrative-level multi-perspective knowledge based on knowledge tracing according to the basic framework of multi-layer knowledge organization of adolescent depression.

[0052] The beneficial effects brought by the technical solution provided in the embodiments of the present application include:

[0053] Based on the knowledge and experience of front-line clinical psychiatric staff, and targeting the clinical pain points of adolescent depression, on the basis of the construction of the "doctor-patient-family-school" four-party multi-perspective natural corpus and the "behavioral phenotype-brain network-epigenetic" multi-modal multi-perspective dataset, we ensure that the associations between the modes of multimodal data are more accurate and rich, and provide a higher-quality dataset that is more in line with clinical practice for the construction of knowledge graphs. This application can effectively solve the problems of absence and blanks in the construction of knowledge graphs, and organizes knowledge according to the "behavior-brain-epigenetic" theory, and proposes a four-layer knowledge representation mechanism of "attribute-concept-event-narrative", so that the constructed knowledge graph is more in line with clinical practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flowchart of the method for constructing a multimodal and multi-perspective knowledge graph for depression in this application;

[0055] Figure 2 It is a structured diagnosis and treatment event knowledge mapping diagram based on ontology alignment;

[0056] Figure 3 This is the flow chart of the KB2LSTM model;

[0057] Figure 4 This is the BiLSTM-KB model structure diagram;

[0058] Figure 5 A flowchart for denoising and mining causal relationships at the narrative level;

[0059] Figure 6 Schematic diagram of the functional modules of the device for constructing a multimodal and multi-perspective knowledge graph for depression in this application. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0062] On the first aspect, the embodiments of the present application provide a method for constructing a multimodal and multi-perspective knowledge graph for depression. In order to address the problems of insufficient existing multimodal relationship mining, incomplete multi-perspective disease information, and the inability of knowledge organization methods to meet the current clinical precision diagnosis and treatment needs, a "behavioral phenotype-brain network-epigenetic" knowledge organization system is established from multiple perspectives and combined with multimodal data to improve knowledge representation capabilities, thereby realizing the construction of a multi-perspective and multi-modal diagnosis and treatment knowledge graph for adolescent depression.

[0063] In one embodiment, referring to Figure 1 , Figure 1 This is a flowchart of the method for constructing a multi-modal and multi-perspective knowledge graph for depression in this application. Figure 1 As shown in Figure 1, the method for constructing a multimodal and multi-perspective knowledge graph for depression includes:

[0064] S1: Collect multimodal and multi-view data related to adolescent depression, and perform feature representation after preprocessing the collected data;

[0065] S2: Characterize multimodal and multi-perspective data and construct a multi-layer knowledge organization framework for adolescent depression, where the knowledge organization framework includes attribute layer, concept layer, event layer and narrative layer;

[0066] S3: Based on the basic framework of multi-layer knowledge organization of adolescent depression, construct multimodal diagnosis and treatment attribute and concept layer knowledge, and represent and construct diagnosis and treatment event layer knowledge based on the event-based graph;

[0067] S4: Based on the basic framework of multi-layer knowledge organization of adolescent depression, construct and integrate narrative-level multi-perspective knowledge based on knowledge tracing.

[0068] Furthermore, the multimodal multi-view data includes behavioral phenotype data, brain network data and epigenetic data. That is, in step S1, it involves the collection, preprocessing and feature representation of behavioral phenotype data, the preprocessing and feature representation of brain network data, and the preprocessing and feature representation of epigenetic data.

[0069] For the collection, preprocessing and feature representation of behavioral phenotypic data, specifically, the behavioral phenotypic data of adolescent depression patients and healthy controls are collected, including the "doctor-patient-family-school" multi-perspective symptom corpus, clinical information (demographic information and clinical symptoms), voice and facial expression data, and feature representation is performed. Specifically, the phenotypic data includes multi-perspective symptom corpus data, demographic and clinical symptom data, voice and facial expression data.

[0070] For the collection and feature representation of multi-perspective symptom corpus data, audio collection equipment is used to collect multiple rounds of conversation voices during the patient and / or family member's medical treatment process, and the family member and school leader's feedback on the condition is collected in the form of telephone recordings to obtain a "doctor-patient-family member-school" multi-perspective symptom text description (for example, the doctor's side - low mood, loss of interest, lack of motivation, poor appetite; the patient's side - depressed and unhappy; the family member's side - no appetite, poor sleep, early awakening, unwilling to go to school; the school side - no communication with classmates, no talking, and lack of concentration in class). The voices of patients, family members, and school (teachers, classmates) are converted into textual knowledge, and the trust value of these multi-perspective knowledge is scored based on the doctor's experience.

[0071] For the collection and characteristic representation of demographic and clinical symptom data, demographic data such as age, gender, years of education, place of residence, and family economic status were collected; HAMD-24 (Hamilton Depression Rating Scale 24 items), Hamilton Anxiety Rating Scale (HAMA), Brief Psychiatric Rating Scale (BPRS) and other peer-rated scales were used to assess the severity of depression, anxiety, and psychotic symptoms in patients; PHQ-9, GAD-7, Pittsburgh Sleep, SHARPS and other self-rated scales were used to assess the severity of clinical symptoms such as depression, anxiety, insomnia, and anhedonia. Each item in the above scales can correspond to a quantitative score, and the scale score is used as a clinical behavioral characteristic.

[0072] For the collection and feature representation of speech and facial expression data, during the doctor's interview with the subject, an audio and video collection device was used to collect the subject's speech and facial expression data, and the subject's facial expression and speech features were quantified through OpenFace and OpenSmile. Among them, facial action units were used as the feature representation of facial expressions, and 12-dimensional Mel-spectrogram cepstral coefficients, root mean square energy, phonation probability, zero crossing rate, and fundamental frequency were used as the feature representation of speech.

[0073] For the preprocessing and feature representation of brain network data, we preprocessed the gray matter (T1weighted imaging, T1W) and white matter (Diffusion Tensor Imaging, DTI) multimodal brain imaging data of patients and controls collected by a 3.0T magnetic resonance imaging system, and constructed a morphological similarity network (Morphological Similarity Network, MSN). The specific process is as follows:

[0074] Based on the collected multimodal brain imaging data, Freesurfer software was used under Linux system to preprocess the T1W structural imaging data of the subjects. According to the brain atlas provided by Freesurfer, the human brain was divided into 68 cortical areas including temporal lobe, parietal lobe, occipital lobe, and prefrontal lobe. The cortical thickness (Thickness), volume (Volume), and mean curve (Mean) corresponding to the 68 cortical areas were obtained. Six T1W imaging indicators, including curve, Gaussian curve, curvature, and folding coefficient, were used to calculate the DTI data of the subjects. Matlab platform and FSL were used to process the DTI data of the subjects, and two DTI imaging indicators, including the anisotropy fraction (FA) and mean diffusivity (MD) corresponding to the fiber connections of the standard template brain area, which were the same as those in the Freesurfer preprocessing mentioned above, were extracted. The above eight structural imaging indicators (six T1W + two DTI indicators) were used to construct the MSN brain network. Pearson correlation analysis was performed on the morphological feature vectors between each paired cortical area to generate a 68×68 MSN matrix for each subject. Regional MSN was measured by using the sum of the weighted correlation coefficients of the pairwise associations between the above eight imaging features in a given brain area and all other areas. The regional MSNs of all healthy volunteers were averaged to evaluate the spatial pattern. The above steps can produce an MSN value for each of the 68 brain areas, and the MSN values ​​of these 68 brain areas are used as the characteristics of the subject's brain network.

[0075] For the preprocessing and feature representation of epigenetic data, the Lumi and BMIQ R packages were used to standardize the raw epigenetic (DNA methylation) data, and linear regression was used to calculate the average methylation level β value of all CpG sites in each gene to represent the DNA methylation characteristics of the gene.

[0076] Furthermore, in one embodiment, characterizing the multimodal multi-view data specifically includes:

[0077] S201: Use a pre-trained model to encode the features of multimodal and multi-view data, and establish the correlation between behavioral phenotype data and brain network data, brain network data and epigenetic data, and behavioral phenotype data and epigenetic data based on similar features;

[0078] S202: Different similar patient groups are established based on multi-level user portrait technology, multi-dimensional vector fusion is performed on the group basis, and different groups are mapped into the same semantic vector space.

[0079] Specifically, based on multimodal and multi-perspective data collection, preprocessing and feature representation, the pre-trained model BioGPT in the biomedical field is used to encode the features extracted from the above multimodal and multi-perspective data, and the "Patient ID" between the "behavioral phenotype-brain network", "brain network-epigenetic" and "behavioral phenotype-epigenetic" data is used as a connection to model the correlation between them; in addition to individual data connections, multi-level user portrait technology is used to establish different similar patient groups, and then multi-dimensional vector fusion is attempted in groups, and then they are mapped to the same semantic vector space and fused through co-attention (common attention mechanism) to learn finer-grained feature representations between different types and modal data, so that it can make full use of the multi-level and multi-dimensional information of the data, and evaluate the difference in the effects of portraits of different user groups.

[0080] Furthermore, in one embodiment, the attribute layer is used to describe the disease based on data attributes; the concept layer is used to model the medical concept relationship of depression; the event layer is used to describe the diagnosis and treatment events of adolescent depression and the daily life events of patients based on the event graph, and supports the representation of event association relationships; the narrative layer is used to model the knowledge representation of the characteristics of patient diagnosis and treatment information.

[0081] That is, on the basis of multimodal and multi-perspective data collection, preprocessing and feature representation, a four-layer knowledge organization framework is designed, which is divided into attribute layer, concept layer, event layer and narrative layer, so as to organize adolescent depression medical knowledge more effectively. The attribute layer mainly uses data attributes to describe disease-related entities; the concept layer models the common medical concept relationship of depression; the event layer draws on the event map to describe adolescent depression diagnosis and treatment events and patients' daily life events, and supports the representation of event association relationships; the narrative layer is a new level of medical diagnosis and treatment knowledge description, which can model knowledge representation for the traceability, subjectivity, uncertainty, causal relationship and other characteristics of patient diagnosis and treatment information to establish more effective association relationships. The narrative layer is a new level of medical diagnosis and treatment knowledge description. The goal of introducing the narrative layer is to organize various medical knowledge in a goal-driven way, and to model knowledge representation for the traceability, subjectivity, uncertainty, causal relationship and other characteristics of adolescent depression patient diagnosis and treatment information to establish more effective association relationships and provide more comprehensive support capabilities for downstream tasks.

[0082] For medical entities and their relationships, see Table 1 below.

[0083] Table 1 Medical entities and their relationships

[0084]

[0085] Furthermore, in one embodiment, based on the basic framework of multi-layer knowledge organization of adolescent depression, multi-modal diagnosis and treatment attributes and concept layer knowledge are constructed, as well as the diagnosis and treatment event layer knowledge representation and construction based on the event graph. Among them, the construction of multi-modal diagnosis and treatment attributes and concept layer knowledge specifically includes:

[0086] S301: For unstructured clinical guidelines and medical literature related to the diagnosis and treatment of adolescent depression, divide the paragraphs according to the standard format of medical documents;

[0087] S302: Classify the clinical guideline paragraphs based on the medical topic categories based on the language representation model and the long short-term memory network;

[0088] S303: Based on the different references of clinical guidelines and medical literature entities in multiple documents, a knowledge enhancement method is used to extract chapter-level fine-grained knowledge from clinical guidelines and medical literature;

[0089] S304: Based on the label classification method, for the extracted results, the probability that the relationship type between any entity pair is r is calculated by a bilinear function to obtain an evaluation result;

[0090] Among them, the knowledge enhancement method includes knowledge injection and knowledge extraction. The knowledge injection is to model the semi-structured document knowledge in the text and find the corresponding matching knowledge graph knowledge. The knowledge extraction is to use the matched knowledge graph knowledge as background knowledge to perform fine-grained extraction of medical entities and relationships in the text.

[0091] Specifically, for the modeling of multimodal diagnosis and treatment attributes and conceptual layer knowledge, it is expanded on the basis of clinical medical knowledge ontology (HealthOnto) and fine-grained evidence-based medical literature ontology (MDO) to meet the needs of disease ontology modeling. In the knowledge construction stage, facing the complex, multi-source and heterogeneous "behavioral phenotype-brain network-epigenetic" multimodal clinical and literature data, how to unify the feature representation and knowledge extraction is an important part of the knowledge organization of depression diagnosis and treatment. Based on the preprocessing and feature extraction of the "behavioral phenotype-brain network-epigenetic" multimodal data, this application performs chapter-level extraction based on knowledge enhancement for the unstructured knowledge in the literature related to the diagnosis and treatment of adolescent depression.

[0092] For existing unstructured clinical guidelines and medical literature, paragraphs are split according to the standard format of medical documents, and text classification of guideline paragraphs is performed based on the BERT+BILSTM model according to medical subject categories, providing good data support for the subsequent extraction of chapter-level knowledge.

[0093] On the basis of the above-mentioned semi-structured data, in view of the complexity of clinical guidelines and medical literature and the different references of entities in multiple documents, a knowledge enhancement-based method is used to extract fine-grained guideline knowledge at the chapter level. This method is mainly divided into two parts, knowledge injection and knowledge extraction. Among them, knowledge injection mainly models the semi-structured document knowledge in the text and finds the corresponding matching knowledge graph knowledge; knowledge extraction uses the matched knowledge graph knowledge as background knowledge to assist the model in fine-grained extraction of medical entities and relationships in the text. Finally, in the output evaluation stage, the extraction results are regarded as a multi-label classification problem, and the probability of the relationship type r between any entity pair is calculated through a bilinear function, and the accuracy of the final result is evaluated.

[0094] Furthermore, in one embodiment, based on the basic framework of multi-layer knowledge organization of adolescent depression, multimodal diagnosis and treatment attribute and concept layer knowledge are constructed, as well as the representation and construction of diagnosis and treatment event layer knowledge based on the event graph. Among them, the representation and construction of diagnosis and treatment event layer knowledge based on the event graph specifically includes:

[0095] S311: Model the diagnosis and treatment causal map knowledge ontology for depression diagnosis and treatment events from different sources and stages; the diagnosis and treatment causal map knowledge ontology is used to represent the event types, attributes and associations based on the causal map and the event association mechanism in complex event processing; the diagnosis and treatment causal map knowledge ontology is divided into behavioral representation events and diagnosis and treatment process events based on event types.

[0096] Specifically, the ontology model of adolescent depression diagnosis and treatment events draws on the event-related mechanism in the event-related graph and complex event processing to represent the knowledge of event types, attributes and associations. In terms of event types, they are mainly divided into two categories: behavioral representation events and diagnosis and treatment process events. According to the scope of clinical data and diagnosis and treatment literature, diagnosis and treatment process events are further divided into subcategories such as examination, diagnosis, and intervention. On this basis, event types can be further subdivided into subtypes; event attributes mainly include temporal and spatial attributes, associated persons, triggering conditions, and results; event associations are captured through event patterns. Event patterns represent the order or logical relationship of a series of events, including four basic associations: sequence, repetition, conjunction, and disjunction. Through event patterns, a series of fine-grained events can be combined into a coarse-grained complex event, thereby realizing the inclusion and combination relationship of events.

[0097] S312: Extract structured diagnosis and treatment events based on R2RML (a language used to represent custom mappings from relational databases to RDF data sets);

[0098] Specifically, for structured medical information, the organization of its events can be achieved through the knowledge mapping method, that is, by writing R2RML rules, mapping the tables in the relational database to the entities and relationships in the graph. For large databases, manually creating mapping rules will consume a lot of human resources. Therefore, this application adopts an OBDA-based approach to semi-automatically create mapping rules from RDB to RDF. Figure 2 The ontology matching and R2RML rule generation process based on OBDA is demonstrated. In this process, the input is the existing relational database table and the target ontology, and the output is the R2RML rule. The RDB table will first be generated and optimized by the OBDA algorithm, and then the matching relationship between the intermediate ontology generated by OBDA and the target ontology will be calculated using the algorithm in the ontology alignment framework AML, and the corresponding rules of the fully matched concepts and relations will be used to generate R2RML rules.

[0099] S313: Perform unstructured disease description event recognition and relationship extraction based on background knowledge attention.

[0100] Specifically, two models are mainly used to extract text event knowledge in the process of adolescent depression diagnosis and treatment: one is the KB2LSTM model for extracting diagnosis and treatment events, and the other is the BiLSTM-KB model for extracting the relationship between diagnosis and treatment events. Among them, the KB2LSTM model integrates background knowledge and attention mechanism. The above KB2LSTM model process is as follows Figure 3 As shown in the figure: First, the corpus is segmented and features are extracted. The extracted features are mapped to a vector table and concatenated with the word vector as the input of the model; then the neural network is used to learn deep context features, and the attention mechanism is used to calculate the importance of context information and related background knowledge; finally, the Softmax function predicts the entity category based on the output feature vector. In addition, the BiLSTM-KB model uses relationship discovery words and background knowledge. The specific structure is shown in Figure 4 ,The network structure mainly includes six layers of input layer, bidirectional LSTM layer, attention layer, softmax layer, knowledge layer and output layer.

[0101] At the narrative level, the diagnosis and treatment of adolescent depression requires the integration of a large amount of information from uncertain, subjective and multi-perspective sources. Therefore, a new set of knowledge representation methods, construction and reasoning methods are needed to organize relevant knowledge. This application first uses a representation method based on knowledge tracing to model the narrative layer ontology, and then uses evidence theory to construct and reason knowledge for multi-perspective subjective and objective information; for multi-modal and heterogeneous clinical complex causal relationships, a knowledge reasoning method based on DS theory is used to identify and construct relationships.

[0102] Therefore, in one embodiment, according to the basic framework of multi-layer knowledge organization of adolescent depression, the construction and integration of narrative layer multi-perspective knowledge based on knowledge tracing is carried out, which specifically includes:

[0103] S401: Model the narrative layer based on the representation method of knowledge tracing to realize the narrative layer knowledge representation of adolescent depression diagnosis and treatment;

[0104] In the early version of the RDF standard, triples about triples were allowed to be represented by reification. In the updated W3C standard, the triple reification syntax and semantics based on RDF-star were proposed. This application adopts this method to describe the source, reliability, weight and other knowledge metadata attributes of the event layer, concept layer and attribute layer knowledge, and realizes uncertainty knowledge reasoning based on these attributes.

[0105] The concept of narrative layer mainly includes narrative perspective (E n ), narrative attributes (P n ) and narrative relations (R n ) three categories, the narrative layer knowledge can be represented as a triple set Ψn , each narrative layer triplet t n ∈Ψ n =(e n |T,p n |r n ,e n |T), where e n ∈E n , p n ∈P n , r n ∈R n , and T is the individual or collection of triples in other layers. The narrative perspective is an entity type that can describe the source of knowledge (patient, family, school, medical examination, etc.); narrative attributes are entity relationships or data attributes that can describe the uncertainty metadata of knowledge (weight, reliability, evidence-based level, etc.); narrative relationships are mainly causal or supporting relationships, which are used to describe the logical association of knowledge based on perspectives and metadata, mainly including logical relationships such as cause / parallel / complementary.

[0106] S402: Use evidence theory to construct and reason knowledge, and realize multi-perspective disease description narrative knowledge construction;

[0107] In the field of uncertainty reasoning, evidence theory, as a generalization of the Bayesian model, has been shown to effectively integrate unreliable and differentially weighted evidence information. In the clinical description of adolescent depression, there are often descriptions of the patient's condition from multiple perspectives. These descriptions have different sources, importance, and credibility. It is necessary to integrate descriptions from multiple perspectives to obtain consistent and credible conclusions.

[0108] This application uses evidence theory to perform a combined reliability calculation based on weight and reliability learning for the text information in different disease descriptions. The specific approach is to first collect disease description texts from multiple perspectives (including the patient himself, family, school, and doctor), and manually mark the complementary / conflicting / parallel relationships between the disease descriptions; in particular, for conflicting sentence fragments, mark their true and false relationships (a description is true compared to another description), and construct a narrative sample training space for multi-perspective disease texts. In this space, the credibility of the provider of the sentence and the importance of the morphemes in the sentence will affect the classification of the relationship; then, the evidence reliability distribution is modeled. This application uses morphemes with reliability and weight as evidence, then all possible narrative relationship spaces (domain of discernment, Θ) may be supported by evidence from different morphemes, and the reliability distribution of each morpheme evidence is shown in the following formula:

[0109]

[0110] Among them, θ∈Θ is a narrative perspective relationship hypothesis, p θ，i Yesi For the belief of hypothesis θ, on this basis, we can recursively calculate two or more independent belief and orthogonal joint belief containing evidence of different reliability and weight according to evidence theory, and then build the target loss function based on the above calculation method to train the evidence belief of each morpheme in each perspective. Finally, we calculate the narrative relationship type containing the highest belief evidence for the new input perspective information, and make classification predictions to obtain a consistent description of the condition from multiple perspectives.

[0111] S403: For multi-modal and heterogeneous clinical complex causal relationships, a knowledge reasoning method based on DS theory is used to identify and construct relationships. Since the multi-perspective disease description narrative relationship constructed based on text may be subjective, this application uses the multi-modal data features of "behavioral phenotype-brain network-epigenetic" to enhance the accuracy of existing narrative causal relationships.

[0112] This application mainly adopts a two-stage automated construction method. First, the causal relationship encoder is trained from multimodal data, and then the causal relationship matrix is ​​updated according to the narrative changes of the diagnosis and treatment process to complete the noise reduction and mining of existing causal relationships.

[0113] Specifically, the knowledge reasoning method based on DS theory is used to identify and construct relationships, including:

[0114] S4031: Use a multimodal pre-trained model (such as Med-PaLM M, etc.) to encode the multimodal and multi-view data so that the multi-source heterogeneous features are mapped to the same high-dimensional vector space;

[0115] S4032: Construct a transformer encoder based on the feature vector to capture the mapping relationship between the feature vector and the narrative causal relationship, ensuring the accuracy of causal relationship denoising and mining;

[0116] S4033: Take the feature vector as input and update the transformer encoder parameters by stochastic gradient descent, as follows:

[0117]

[0118] Where φ represents the encoder parameter, f i Represents the feature vector of the i-th multimodal data, y i Indicates narrative relationships;

[0119] S4034: Use a multimodal pre-training model to use the narrative encoding obtained in the process of constructing multi-perspective disease description narrative knowledge as a feature vector;

[0120] S4035: converting the support confidence belief distribution of each narrative causal relationship into a causal relationship adjacency matrix, wherein the value of the adjacency matrix represents the causal relationship between the two narratives;

[0121] S4036: Based on the feature vector and adjacency matrix of the narrative, GNN is used to learn the transformation between narratives in the clinical decision-making process, and the narrative relationship is updated to obtain a new relationship matrix.

[0122] That is, the narrative-based feature vector sequence F = {f 0 , f 1 , ..., f n} and the adjacency matrix S, and use GNN (graph neural network) to learn the transformation between narratives in the clinical decision-making process and update the narrative relationship to obtain a new relationship matrix As shown below:

[0123] S aug =GNN(f t , f t+1 , S)

[0124] After the above training is completed, the model is used to generate a new relationship matrix, which reduces the noise relationship and retains the potential associations mined by the model (such as Figure 5 , Flowchart of denoising and mining of narrative-level causal relationships).

[0125] In a second aspect, an embodiment of the present application also provides a device for constructing a multimodal and multi-perspective knowledge graph for depression.

[0126] In one embodiment, referring to Figure 6 , Figure 6 This is a schematic diagram of the functional modules of the device for constructing a multi-modal and multi-perspective knowledge graph for depression in this application. Figure 6 As shown, the multimodal and multi-perspective knowledge graph construction device for depression includes: an acquisition module, a construction module, a creation module, and an execution module.

[0127] The acquisition module is used to collect multimodal and multi-perspective data related to adolescent depression, and perform feature representation after preprocessing the collected data; the construction module is used to characterize the multimodal and multi-perspective data, and construct a basic framework for multi-layer knowledge organization of adolescent depression, where the basic knowledge organization framework includes attribute layer, concept layer, event layer and narrative layer; the creation module is used to construct multimodal diagnosis and treatment attribute and concept layer knowledge based on the basic framework of multi-layer knowledge organization of adolescent depression, as well as to represent and construct diagnosis and treatment event layer knowledge based on the causal map; the execution module is used to construct and integrate narrative layer multi-perspective knowledge based on knowledge tracing based on the basic framework of multi-layer knowledge organization of adolescent depression.

[0128] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0129] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0130] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0131] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0133] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for constructing a multimodal and multi-perspective knowledge graph for depression, characterized in that: The method for constructing a multimodal and multi-perspective knowledge graph of depression includes: Collect multimodal and multi-perspective data related to adolescent depression, and perform feature representation after preprocessing the collected data; Characterize multimodal and multi-perspective data and construct a multi-layer knowledge organization framework for adolescent depression, where the knowledge organization framework includes attribute layer, concept layer, event layer and narrative layer; According to the basic framework of multi-layer knowledge organization of adolescent depression, multimodal diagnosis and treatment attribute and concept layer knowledge is constructed, and the diagnosis and treatment event layer knowledge is represented and constructed based on the event-based graph. According to the basic framework of multi-layer knowledge organization of adolescent depression, the narrative layer multi-perspective knowledge is constructed and integrated based on knowledge tracing; Among them, the construction and integration of narrative layer multi-perspective knowledge based on knowledge tracing is carried out according to the basic framework of multi-layer knowledge organization of adolescent depression, which specifically includes: Model the narrative layer based on the representation method of knowledge tracing, and realize the narrative layer knowledge representation of adolescent depression diagnosis and treatment; Use evidence theory to construct and reason knowledge, and realize narrative knowledge construction of disease description from multiple perspectives; Aiming at multi-modal and heterogeneous clinical complex causal relationships, the knowledge reasoning method based on DS theory is used to identify and construct relationships; The knowledge reasoning method based on DS theory is used to identify and construct relationships, specifically including: A multimodal pre-trained model is used to encode the features of multimodal and multi-view data, so that the multi-source heterogeneous features can be mapped into the same high-dimensional vector space; A transformer encoder is constructed based on the feature vector to capture the mapping relationship between the feature vector and the narrative causal relationship; Take the feature vector as input and update the transformer encoder parameters by stochastic gradient descent; A multimodal pre-training model is used to take the narrative encoding obtained in the process of constructing narrative knowledge of multi-perspective disease description as a feature vector; Converting the support confidence belief distribution of each narrative causal relationship into a causal relationship adjacency matrix, wherein the values ​​of the adjacency matrix represent the causal relationship between the two narratives; Based on the feature vector and adjacency matrix of the narrative, GNN is used to learn the transformation between narratives in the clinical decision-making process, and the narrative relationship is updated to obtain a new relationship matrix.

2. A method for constructing a multimodal and multi-perspective knowledge graph for depression as claimed in claim 1, characterized in that: The multimodal and multi-perspective data include behavioral phenotype data, brain network data, and epigenetic data; The phenotypic data includes multi-perspective symptom corpus data, demographic and clinical symptom data, and voice and facial expression data.

3. A method for constructing a multimodal and multi-perspective knowledge graph for depression as claimed in claim 2, characterized in that: The characterizing of the multi-modal and multi-view data specifically includes: A pre-trained model is used to encode the features of multimodal and multi-view data, and the correlation between behavioral phenotype data and brain network data, brain network data and epigenetic data, and behavioral phenotype data and epigenetic data is established based on similar features; Different groups of similar patients are established based on multi-level user portrait technology, and multi-dimensional vector fusion is performed on the group basis to map different groups into the same semantic vector space.

4. A method for constructing a multimodal and multi-perspective knowledge graph for depression as claimed in claim 2, characterized in that: The attribute layer is used to describe the disease based on data attributes; The concept layer is used to model the medical concept relationship of depression; The event layer is used to describe the diagnosis and treatment events of adolescent depression and the daily life events of patients based on the event graph, and supports the expression of event association relationships; The narrative layer is used to perform knowledge representation modeling on the characteristics of patient diagnosis and treatment information.

5. A method for constructing a multimodal and multi-perspective knowledge graph for depression as claimed in claim 4, characterized in that: According to the basic framework of multi-layer knowledge organization of adolescent depression, the multi-modal diagnosis and treatment attributes and concept layer knowledge are constructed, and the diagnosis and treatment event layer knowledge is represented and constructed based on the event map. Among them, the construction of multi-modal diagnosis and treatment attributes and concept layer knowledge specifically includes: For unstructured clinical guidelines and medical literature related to the diagnosis and treatment of adolescent depression, paragraphs are divided according to the standard format of medical documents; Text classification of clinical guideline paragraphs based on medical topic categories based on language representation models and long short-term memory networks; In view of the different references of clinical guidelines and medical literature entities in multiple documents, a knowledge enhancement method is used to extract chapter-level fine-grained knowledge from clinical guidelines and medical literature. Based on the label classification method, for the extracted results, the probability of the relationship type r between any entity pair is calculated through a bilinear function to obtain the evaluation result; Among them, the knowledge enhancement method includes knowledge injection and knowledge extraction. The knowledge injection is to model the semi-structured document knowledge in the text and find the corresponding matching knowledge graph knowledge. The knowledge extraction is to use the matched knowledge graph knowledge as background knowledge to perform fine-grained extraction of medical entities and relationships in the text.

6. A method for constructing a multimodal and multi-perspective knowledge graph for depression as claimed in claim 4, characterized in that: According to the basic framework of multi-layer knowledge organization of adolescent depression, multimodal diagnosis and treatment attribute and concept layer knowledge are constructed, and the diagnosis and treatment event layer knowledge based on the event map is represented and constructed, wherein the representation and construction of the diagnosis and treatment event layer knowledge based on the event map specifically includes: Model the knowledge ontology of diagnosis and treatment events from different sources and stages; Based on R2RML, structured diagnosis and treatment event extraction is performed; Perform unstructured disease description event recognition and relationship extraction based on background knowledge attention.

7. A method for constructing a multimodal and multi-perspective knowledge graph for depression as claimed in claim 6, characterized in that: The diagnosis and treatment event graph knowledge ontology is used to represent the event type, attribute and association according to the event graph and the event association mechanism in complex event processing; The diagnosis and treatment event graph knowledge ontology is divided into behavior representation events and diagnosis and treatment process events in terms of event types.

8. A multi-modal and multi-perspective knowledge graph construction device for depression, characterized in that: The multimodal and multi-perspective knowledge graph construction device for depression includes: The acquisition module is used to collect multi-modal and multi-view data related to adolescent depression, and perform feature representation after pre-processing the collected data; A construction module is used to characterize multimodal and multi-perspective data and construct a multi-layer knowledge organization framework for adolescent depression, wherein the knowledge organization framework includes an attribute layer, a concept layer, an event layer, and a narrative layer; Create a module for constructing multimodal diagnosis and treatment attribute and concept-level knowledge based on the basic framework of multi-layer knowledge organization of adolescent depression, and for representing and constructing diagnosis and treatment event-level knowledge based on the event-diagnosis graph; An execution module is used to construct and integrate narrative-level multi-perspective knowledge based on knowledge tracing according to the basic framework of multi-layer knowledge organization of adolescent depression; Among them, the construction and integration of narrative layer multi-perspective knowledge based on knowledge tracing is carried out according to the basic framework of multi-layer knowledge organization of adolescent depression, which specifically includes: Model the narrative layer based on the representation method of knowledge tracing, and realize the narrative layer knowledge representation of adolescent depression diagnosis and treatment; Use evidence theory to construct and reason knowledge, and realize narrative knowledge construction of disease description from multiple perspectives; Aiming at multi-modal and heterogeneous clinical complex causal relationships, the knowledge reasoning method based on DS theory is used to identify and construct relationships; The knowledge reasoning method based on DS theory is used to identify and construct relationships, specifically including: A multimodal pre-trained model is used to encode the features of multimodal and multi-view data, so that the multi-source heterogeneous features can be mapped into the same high-dimensional vector space; A transformer encoder is constructed based on the feature vector to capture the mapping relationship between the feature vector and the narrative causal relationship; Take the feature vector as input and update the transformer encoder parameters by stochastic gradient descent; A multimodal pre-training model is used to take the narrative encoding obtained in the process of constructing narrative knowledge of multi-perspective disease description as a feature vector; Converting the support confidence belief distribution of each narrative causal relationship into a causal relationship adjacency matrix, wherein the values ​​of the adjacency matrix represent the causal relationship between the two narratives; Based on the feature vector and adjacency matrix of the narrative, GNN is used to learn the transformation between narratives in the clinical decision-making process, and the narrative relationship is updated to obtain a new relationship matrix.

Images