Construction method of medical auxiliary diagnosis system model based on multi-modal breast cancer knowledge graph

Abstract

The invention discloses a construction method of a medical auxiliary diagnosis system model based on a multi-modal breast cancer knowledge graph. The method comprises three parts of constructing a multi-modal breast cancer mapping knowledge domain, constructing a breast cancer mapping knowledge domain enhanced diagnosis system module and constructing a medical auxiliary diagnosis system model: 1) constructing the multi-modal breast cancer mapping knowledge domain, including data acquisition and processing, ontology modeling, entity and relation extraction model construction and image data combination; 2) constructing a breast cancer knowledge graph enhanced diagnosis system module, including knowledge query and knowledge screening and processing, and 3) constructing a medical auxiliary diagnosis system model, including data acquisition and processing, text data feature extraction, image data feature extraction, joint representation of learning texts and images, and constructing a breast cancer diagnosis module and a training model.

Description

Technical Field

[0001] The present invention belongs to the field of knowledge graph application, and specifically relates to a method for constructing a medical auxiliary diagnosis system model based on a multimodal breast cancer knowledge graph. Background Art

[0002] The medical auxiliary diagnosis system based on the multimodal breast cancer knowledge graph plays a key role in computer-assisted breast cancer diagnosis and prognosis. The General Office of the State Council clearly pointed out in the "Opinions on Promoting the Development of "Internet + Medical Health" that all sectors of society should use information technology to solve the "difficulties" and "blocking points" in the process of people seeking medical treatment. In this context, computer-assisted medical decision-making has become a key area for the combination of artificial intelligence technology and medical care. It is a breakthrough point for information technology to empower the entire process of disease diagnosis and treatment. It plays a positive role in improving medical quality and efficiency, optimizing the allocation of medical resources between regions, and improving people's experience of seeking medical treatment.

[0003] According to the World Health Organization's World Cancer Report, breast cancer is the most common cancer with the highest incidence and mortality among women worldwide. Early screening can effectively reduce the mortality rate of breast cancer. The current auxiliary diagnosis system is a knowledge base system driven by electronic medical record ratings, which can only meet the basic needs of diagnosis and treatment decisions. However, various specialties such as oncology and cardiovascular medicine often face more complex medical records and require more differentiated specialist knowledge systems. In addition, the current auxiliary diagnosis system relies more on the data of interviews and examination / test results, but image data provides additional information for diagnosis and subsequent treatment options, especially for breast cancer, which is often diagnosed using images. There is currently a lack of auxiliary diagnosis systems that can integrate clinical data and image data. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a medical auxiliary diagnosis system for breast cancer, which can accurately and effectively provide reliable diagnosis results of patients diagnosed with benign breast tumors such as adenosis, fibroma, ductal carcinoma, lobular carcinoma, or breast cancer based on the multimodal information formed by combining the patient's electronic medical records and various examinations and test results including imaging examinations, histopathological examinations, etc. In order to further improve the accuracy and effectiveness of the system, a multimodal breast cancer knowledge graph is used to enhance the system in combination with the multimodal knowledge related to breast cancer in the graph.

[0005] The present invention discloses a method for constructing a medical auxiliary diagnosis system based on a multimodal breast cancer knowledge graph, the method comprising the following steps: 1) Construct a multimodal breast cancer knowledge graph. Collect a large amount of breast cancer-related information from a large number of high-quality medical data from different sources as data, model the ontology in the knowledge graph, use the entity and relationship joint learning method based on deep learning to extract entities and relationships from the data, and use the corresponding typical image data such as X-ray images, histopathology images, etc. as associated entities for breast cancer disease entities to generate a multimodal, large-scale, and highly professional breast cancer knowledge graph; 2) Construct a breast cancer knowledge graph-enhanced diagnosis system module. For each input of the diagnosis system, select the entities involved, query the corresponding triples in the knowledge graph, and use a rule-based method to filter out the entities with greater weight for diagnosing breast cancer. These entities are divided into text data and image data. These data are processed to obtain joint features as additional features for the diagnosis system input; 3) Construct a medical auxiliary diagnosis system model. The de-identified electronic medical records of patients diagnosed with benign breast tumors and malignant breast cancer and the multimodal data obtained from various related examination results are used as model training data. The multimodal data in the sample is processed to obtain joint features. Combined with the joint features of the corresponding knowledge obtained from the knowledge graph, the features are mapped to the potential label space. The model parameters are gradient updated according to the obtained label representation and the actual label. By minimizing the loss function, the distance between the predicted label and the actual label is continuously shortened.

[0006] Furthermore, the step of constructing a multimodal breast cancer knowledge graph specifically includes: 1) Data collection and processing: Collect breast cancer-related data from a large number of high-quality medical data from different sources and standardize the data for subsequent knowledge extraction; 2) Ontology modeling. Ontology is the main data model of the knowledge graph, which specifies the entity types and relationship types involved in the knowledge graph; 3) Build entity and relationship extraction models. Select entity and relationship extraction models based on deep learning and joint learning of entities and relationships, and train the models with labeled data sets; 4) Combined with image data. For breast cancer disease entities, we collected various types of breast cancer typical X-ray images, histopathological images and other related image data, and used them as associated entities connected to the corresponding breast cancer disease entities using a deep learning-based method.

[0007] Furthermore, the step of constructing a breast cancer knowledge graph enhanced diagnosis system module specifically includes: 1) Knowledge query. The input of the diagnosis system is mainly the patient's medical record information. The entities involved are selected and the corresponding triples are queried in the knowledge graph; 2) Knowledge screening and processing: A rule-based approach is used to screen out entities in triples that have a greater weight for diagnosing breast cancer. The set of these entities belongs to multimodal data, which is divided into text data and image data. These data are processed to obtain joint features as additional input to the diagnostic system.

[0008] Furthermore, the step of constructing the medical auxiliary diagnosis system model specifically includes: 1) Data collection and processing. Collect de-identified electronic medical records of patients diagnosed with benign breast tumors and malignant breast cancer and related multimodal data from various examination results, perform standardized processing, and divide them into text data and image data; 2) Extract text data features. Use deep learning-based methods to extract corresponding features from different text data; 3) Extract image data features. Use deep learning-based methods to extract corresponding features from different image data; 4) Learn the joint representation of text and images. Use a deep learning-based method to use text data representation and image data representation to learn the joint representation of text and images, and align text and image data into a unified semantic space; 5) Construct a breast cancer diagnosis module. Select a suitable deep neural network as the diagnosis model, combine the joint representation of text and image with the corresponding joint representation of knowledge as the input of the model, and obtain the predicted label representation; 6) Training the model. The model parameters are gradient updated based on the obtained label representation and the actual label, and the distance between the predicted label and the actual label is continuously shortened by minimizing the loss function.

[0009] Compared with the prior art, the present invention has the following advantages: 1) This invention is aimed at the field of breast cancer specialty, has a breast cancer knowledge system, and can specifically handle the complex medical records of breast cancer; 2) The present invention is an auxiliary diagnosis system of multimodal data fusion, which not only utilizes the data of medical interview and examination / test results, but also makes full use of image data that can provide additional information for diagnosis, and combines the two to obtain a more reliable diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 The figure is a flowchart of a medical auxiliary diagnosis system based on a multimodal breast cancer knowledge graph implemented according to the present invention.

[0011] Figure 2 Flowchart for constructing a multimodal breast cancer knowledge graph.

[0012] Figure 3Flowchart for constructing the breast cancer knowledge graph enhanced diagnosis system module.

[0013] Figure 4 Flowchart for constructing a medical auxiliary diagnosis system model. DETAILED DESCRIPTION

[0014] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with embodiments and drawings. Figure 1 A flowchart of a method for constructing a medical auxiliary diagnosis system model based on a multimodal breast cancer knowledge graph is shown.

[0015] like Figure 1 As shown, the method comprises the following steps: Construct a multimodal breast cancer knowledge graph: 1) Collect a large amount of high-quality breast cancer-related data and standardize it; 2) Define the ontology structure of the breast cancer knowledge graph; 3) Use a deep learning-based entity and relationship extraction model and train the model with annotated data sets; 4) Add typical X-ray images, histopathological images and other image data corresponding to various types of breast cancer diseases to the knowledge graph.

[0016] The specific steps include: 1. Constructing a breast cancer knowledge graph: 1) Data collection and processing. Collect breast cancer-related information from a large number of high-quality medical data from different sources. These data can be divided into three types: structured data, semi-structured data, and unstructured data. Standardize these data for subsequent knowledge extraction; 2) Construct an ontology model. Define entities and relationship types for the breast cancer knowledge graph. The ontology model refers to professional clinical terminology and is modified and expanded based on doctors’ suggestions to improve the applicability of the ontology model; 3) Construct entity and relationship extraction models. The joint learning model of entities and relationships based on deep learning is not prone to error propagation problems, and considering the correlation between named entity recognition and relationship extraction, the present invention selects entity and relationship extraction models based on joint learning of entities and relationships based on deep learning, and trains the models with labeled data.

[0017] 2. Breast cancer knowledge graph combined with image data: 1) Collect image data. For breast cancer disease entities in the knowledge graph, collect relevant image data such as typical X-ray images, histopathological images, etc. of various breast cancers; 2) Combined with image data. The important basis for breast cancer diagnosis is related image data including X-ray images, histopathological images, etc. Using deep learning methods, the collected image data is used as an associated entity connected with the corresponding breast cancer disease entity.

[0018] Construct a breast cancer knowledge graph enhanced diagnosis system module: 1) Query the knowledge graph for the triples corresponding to the entities involved in the diagnosis system input; 2) Filter out entities with greater weights for diagnosing breast cancer from the triples, and process them to obtain the corresponding joint features.

[0019] The specific steps include: 1) Knowledge query. For the input of the diagnosis system, mainly the patient's medical record information, select the entities involved and query the corresponding triples in the knowledge graph; 2) Knowledge screening and processing. Different entities in the triples obtained through the query have different effects on the diagnosis of breast cancer. A rule-based method is used to screen out the predefined entities in the triples that have a greater weight on the diagnosis of breast cancer. The collection of these entities constitutes multimodal data, which is divided into text data and image data. These data are processed consistently with the multimodal input data of the diagnostic system to obtain the corresponding representation as an additional input to the diagnostic system.

[0020] Construct a medical auxiliary diagnosis system model: 1) Collect training data, mainly including medical records and related examination results of patients with benign breast tumors and malignant breast cancer; 2) Use deep learning-based methods to extract corresponding features from different text data; 3) Use deep learning-based methods to extract corresponding features from different image data; 4) Learn the joint representation of text and image; 5) Select a suitable deep learning network as the diagnosis model; 6) Train the model to perform gradient updates on the model parameters.

[0021] The specific steps include: 1) Data collection and processing. Collect de-identified electronic medical records of patients diagnosed with benign breast tumors and malignant breast cancer and related multimodal data from various examination results, perform standardized processing, and divide them into text data and image data; 2) Extract features of text data. For each sample, various text data, such as the patient's basic personal information, chief complaint, medical history, etc., use deep learning methods to extract corresponding features from different text data. The present invention proposes to use the BERT model framework to first process various text data into corresponding sentence sets, and then use each sentence set as the input of the BERT model, and take the output of the last hidden layer of the corresponding [CLS] as the overall representation of each sentence. Finally, since some sentence sets may have multiple sentences, namely [CLS] representations, these representations of the sentence sets with multiple [CLS] representations are used as the input of the BERT model again, and the input of the last hidden layer of the corresponding [CLS] is taken as the overall representation of the sentence set. In this way, the representations corresponding to various text data can be obtained; 3) Extract image data features. Each sample corresponds to various image data, such as X-ray images of patients, histopathological images, etc., and the deep learning method is used to extract corresponding features from different image data. The present invention proposes to use the CNN model framework, take different image data as input, and obtain the representation corresponding to different image data through convolution, pooling, and fully connected layers; 4) Learn the joint representation of text and image. What the model needs to learn is the joint representation of multimodal data. The deep learning method is used to learn the joint representation of text and image using text data representation and image data representation, and align the text and image data into a unified semantic space. The present invention refers to the ViLBERT model framework and proposes an Encoder model framework using two Transformers. Different text data representations and image data representations corresponding to a sample are used as the inputs of the two Encoders. When each layer of the Encoder where the text data is located performs self-attention, it uses its own input as Q, and the input of the layer corresponding to the Encoder where the image data is located is used as K and V. Similarly, when each layer of the Encoder where the image data is located performs self-attention, it uses its own input as Q, and the input of the layer corresponding to the Encoder where the text data is located is used as K and V. That is, each time the two Encoders perform self-attention, they are actually performing cross-attention. Finally, the [CLS] representation of the last layer of the two Encoders is taken out, and they are concatenated and sent to a fully connected layer to obtain the final output as the joint representation of the text and image; 5) Construct a breast cancer diagnosis module. Select a suitable deep neural network as the diagnosis model. The present invention proposes to concatenate the joint representation of text and image with the corresponding knowledge representation as the input of the diagnosis model to obtain the representation of the predicted label; 6) Train the model. Perform gradient updates on the model parameters based on the obtained label representation and the actual label, continuously narrowing the distance between the predicted label and the actual label.

Claims

1. A method for constructing a medical auxiliary diagnosis system model based on a multimodal breast cancer knowledge graph. It is characterized in that The method comprises: (1) Construct a multimodal breast cancer knowledge graph; (2) Construct a breast cancer knowledge graph-enhanced diagnosis system module; (3) Construct a medical auxiliary diagnosis system model.

2. According to claim 1, a method for constructing a medical auxiliary diagnosis system model based on a multimodal breast cancer knowledge graph, It is characterized in that The construction of a multimodal breast cancer knowledge graph specifically includes: 1) Data collection and processing; 2) Model the ontology of the knowledge graph, including entity types and relationship types; 3) Build entity and relationship extraction models. Select entity and relationship extraction models based on deep learning, and train the models with labeled data; 4) Combined with image data. For breast cancer disease entities, we collected various types of breast cancer typical X-ray images, histopathological images and other related image data, and used them as associated entities connected to the corresponding breast cancer disease entities using a deep learning-based method.

3. According to claim 1, a method for constructing a medical auxiliary diagnosis system model based on a multimodal breast cancer knowledge graph, It is characterized in that The construction of the breast cancer knowledge graph enhanced diagnosis system module specifically includes: 1) Knowledge query. Select the entities involved in the input of the diagnostic system and query the corresponding knowledge in the knowledge graph; 2) Knowledge screening and processing: A rule-based approach is used to screen out entities in the triples that have a greater impact on the diagnosis of breast cancer. These data are processed to obtain the corresponding joint features as additional features for the diagnostic system input.

4. According to claim 1, a method for constructing a medical auxiliary diagnosis system model based on a multimodal breast cancer knowledge graph, It is characterized in that The construction of the medical auxiliary diagnosis system model specifically includes: 1) Data collection and processing. Collect multimodal data from medical records of patients with benign breast tumors and malignant breast cancer and related examination results; 2) Extract text data features. Use deep learning-based methods to extract corresponding features from different text data; 3) Extract image data features. Use deep learning-based methods to extract corresponding features from different image data; 4) Learn the joint representation of text and images. Use a deep learning-based method to use text data representation and image data representation to learn the joint representation of text and images, and align text and image data into the same semantic space; 5) Construct a breast cancer diagnosis module. Select a suitable deep neural network as the diagnosis model, and combine the joint representation of text and image with the corresponding joint representation of knowledge as the input of the model; 6) Training the model: Based on the predicted label representation and the true label obtained from the breast cancer diagnosis model, the model parameters are gradient updated.

Images