PubMed-based medical text mining method, device and equipment

Abstract

The application discloses a PubMed-based medical text mining method, device and equipment, retrieves relevant literatures of different stages of a target disease from a PubMed database, stores the literatures into a preset database, and obtains a literature set of different stages. A text mining model takes a clinical physical examination index set as an index set, performs retrieval in the literature set, and determines a target disease correlation index set of each stage according to the frequency of each index appearing in the literature set of different stages. The indexes in the correlation index set of each stage are sorted by improving a TF-IDF method, and a target correlation index set of each stage of the target disease is obtained. The application performs retrieval based on the PubMed database, obtains relatively comprehensive relevant literatures of different stages of the disease, reduces the limitation of the data source, and makes the data more comprehensive. The indexes in the target correlation index set have higher relevance to the target disease, and medical staff can refer to the target correlation index set for diagnosis.

Description

Technical Field

[0001] This invention relates to the field of computer data mining, and more specifically, to a method, apparatus, and device for medical text mining based on PubMed. Background Technology

[0002] In medical research, studying the correlation between diseases and detection indicators is crucial for disease prevention, diagnosis, and treatment. While there is extensive research on the relationship between diseases and genes, research on the relationship between detection indicators and diseases is relatively limited.

[0003] In recent years, data mining methods, including graph algorithms, machine learning, and text mining algorithms, have been widely applied in the field of correlation research, such as studying the relationship between diseases and genes, diseases and drugs, and the relationship between diseases themselves. Based on graph theory, graph models and various algorithms have been constructed, such as neighborhood association analysis, shortest path algorithms, and walking models. However, graph-based algorithms have not performed well in studying the correlation between diseases and detection indicators. Machine learning algorithms have high requirements for data format and quality, but for disease data, there are no good methods for formatting and quantifying it. Therefore, the effectiveness of general machine learning applications in deciphering the correlation between diseases and detection indicators may be limited by the availability of source data. Text mining algorithms have also been widely used in studying various biological problems, such as functional genomics, biological pathway interactions, protein expression, pharmacogenomic associations, comparative toxicogenomics, neuropsychiatric diseases, and large-scale bioinformatics analysis.

[0004] DISEASES predicts disease-gene associations using a co-occurrence method; MimMiner uses TF-IDF to study the relationship matrix between OMIM text and diseases; CATAPULT predicts disease-related genes using graphical models and semi-supervised machine learning algorithms. LGscore studies disease-gene correlations using the Google search engine. However, these algorithms primarily analyze disease-related genes through text mining. Besides the relationships between gene disease examination indicators, extracting and processing key information such as diseases and medical indicators from medical literature is crucial. Data extraction and analysis of cases in medical literature involves the identification and summarization of clinical symptoms and medical indicators. Searching based on disease names and corresponding medical indicators yields relevant literature, which is then fully analyzed to determine if a specific sentence contains the disease name and corresponding medical indicator and value. However, this method is limited to cases where the relationship between the disease and indicator is known, and requires the disease name and medical indicator to be in the same sentence, resulting in significant limitations and low accuracy. Summary of the Invention

[0005] In view of this, the present invention provides a medical text mining method, apparatus and equipment based on PubMed, which is used to solve the problems of existing medical indicator text mining methods having large limitations and low accuracy of text mining algorithms.

[0006] To achieve the above objectives, the following solution is proposed:

[0007] A PubMed-based medical text mining method includes:

[0008] Relevant literature on different stages of the target disease is retrieved from the PubMed database and stored in a pre-defined database to obtain literature sets for different stages, including four stages of risk assessment, diagnosis, treatment and prognosis for the target disease.

[0009] Obtain a set of clinical physical examination indicators;

[0010] The text mining model uses the set of clinical physical examination indicators as an index set to search the literature set, and determines the set of associated indicators for each stage of the target disease based on the frequency of each indicator in the literature set at different stages.

[0011] By improving the TF-IDF method, the indicators in the correlation indicator set of each stage are sorted to obtain the target correlation indicator set of each stage of the target disease, which can be used as a reference for medical staff when diagnosing the target disease.

[0012] Preferably, the process of retrieving relevant literature on different stages of the target disease from the PubMed database and storing the literature in a preset database includes:

[0013] Enter each stage of the target disease into PubMed for retrieval to obtain a list of PubMed IDs containing relevant literature;

[0014] Download the list of IDs and use the EFETCH tool to download the corresponding XML document for each ID based on the ID list;

[0015] Store documents in XML format into a pre-defined database.

[0016] Preferably, before storing the XML format documents into the preset database, the method further includes:

[0017] The relevant literature should be annotated, and the information to be annotated includes the title, author, abstract, and keywords.

[0018] Preferably, the process of ranking the correlation indicators in the correlation indicator set of each stage by improving the TF-IDF method includes:

[0019] The TF-IDF value of each indicator is calculated using the following formula:

[0020] p ml (t|M d )=tf(t ,d) / dl d ,

[0021]

[0022]

[0023] If tf (t,d) >0,

[0024]

[0025] Where, p ml (t|M d R is the maximum likelihood estimate of index t in literature d. t,d Let p(Q|M) be the risk function, Q be the set of clinical physical examination indicators, and p(Q|M) be the risk function. d ) represents the query probability generated based on a set of clinical physical examination indicators, i.e., the TF-IDF value, p(t|M d ) represents the probability of a set of clinical physical examination indicators appearing in the literature, 1.0-p(t|M d ) represents the probability that a set of clinical physical examination indicators does not appear in the literature, ∏ t∈Q p ml (t,d) represents the ranking of document d, tf (t,d) It is the original probability of index t appearing in literature d, dl d It is the total number of indicators in literature d, df t For the document frequency of index t, This refers to the average frequency of t in the literature collection;

[0026] The correlation indicators in the correlation indicator sets of different stages are sorted according to their TF-IDF values.

[0027] A PubMed-based medical text mining device includes:

[0028] The literature acquisition unit retrieves relevant literature on different stages of the target disease from the PubMed database and stores the literature in a preset database to obtain literature sets for different stages, including four stages of risk assessment, diagnosis, treatment and prognosis for the target disease.

[0029] The index retrieval unit retrieves the clinical physical examination indicator set;

[0030] The indicator retrieval unit uses a text mining model to use the clinical physical examination indicator set as an index set to search the literature set, and determines the associated indicator set of the target disease at each stage based on the frequency of each indicator in the clinical physical examination indicator set appearing in the literature set at different stages.

[0031] The indicator confirmation unit sorts the indicators in the relevant indicator set for each stage by improving the TF-IDF method, and obtains the target relevant indicator set for each stage of the target disease, which can be used as a reference for medical staff when diagnosing the target disease.

[0032] A PubMed-based medical text mining device includes: a memory and a processor;

[0033] The memory is used to store programs;

[0034] The processor is used to execute the program to implement the various steps of the PubMed-based medical text mining method described above.

[0035] As can be seen from the above technical solution, the medical text mining method based on PubMed provided by this invention retrieves relevant literature on different stages of a target disease from the PubMed database and stores the relevant literature in the database, obtaining literature sets for different stages. The text mining model uses a set of clinical physical examination indicators as an index set, searches within the literature sets, and determines the set of associated indicators for each stage of the target disease based on the frequency of occurrence of each indicator in the clinical physical examination indicator set in the literature sets of different stages. By improving the TF-IDF method, the associated indicators in the associated indicator sets of each stage are sorted to obtain the target associated indicator set for each stage of the target disease. This invention uses the PubMed retrieval system to obtain a relatively comprehensive set of relevant literature on different stages of the disease, reducing the limitations of data sources and making the data more comprehensive.

[0036] This invention utilizes a text mining model that searches relevant literature based on a set of clinical examination indicators to identify the associated indicator sets for each stage of a target disease. The text mining model considers the correlation between the clinical examination indicator set and the disease. This invention retrieves relevant clinical examination indicators from literature related to the target disease, reducing search limitations and resulting in more comprehensive search results.

[0037] This invention improves the TF-IDF method to further verify the association between the correlation indicators in each stage of the correlation indicator set and the disease, thereby enhancing the accuracy of the correlation indicators in the target correlation indicator set. The indicators in the target correlation indicator set show a high correlation with the target disease, and medical personnel can refer to this set for diagnosis. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 This embodiment provides a flowchart of a medical text mining method based on PubMed.

[0040] Figure 2 This embodiment provides a flowchart for document retrieval;

[0041] Figures 3a-3d This embodiment provides a radar diagram illustrating the correlation indicators between different stages of pancreatic cancer.

[0042] Figure 4 This is a schematic diagram showing the distribution of TF-IDF values ​​for different associated indicators of pancreatic cancer provided in this embodiment;

[0043] Figure 5 This is a schematic diagram showing the distribution of TF-IDF values ​​for another pancreatic cancer-related indicator provided in this embodiment;

[0044] Figure 6 This embodiment provides a schematic diagram of a medical text mining device based on PubMed.

[0045] Figure 7 This embodiment provides a hardware structure block diagram of a medical text mining device based on PubMed. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] First, combined Figure 1 This embodiment provides a PubMed-based medical text mining method, such as... Figure 1 As shown, the method includes:

[0048] Step S01: Retrieve relevant literature on different stages of the target disease from the PubMed database and store the literature in a preset database.

[0049] Specifically, PubMed is a biomedical information retrieval system, primarily based on the MEDLINE database, and also includes over a dozen other databases such as ADIS, CANCER, and OLDMEDLINE. Therefore, the PubMed database has a wide coverage and relatively complete literature. This invention utilizes the PubMed database to retrieve relevant literature on the target disease at different stages, including risk assessment, diagnosis, treatment, and prognosis, and stores the relevant literature in a pre-defined database. Each document has a corresponding PubMed-ID, title, author, abstract, keywords, and MeSH terminology. First, each stage of the target disease is entered into PubMed for retrieval, resulting in an ID list containing the corresponding PubMed-IDs for each stage. Then, the EFETCH tool is used to download the XML format documents corresponding to each ID based on the ID list, obtaining the XML format documents for each stage. The EFETCH tool is a tool for batch data downloading; it retrieves the full-text information based on the document ID and downloads the full-text information of the document in XML format. Then, each document is stored in a pre-defined database, which can be a local PostgreSQL database or a database built using other database software. The database can be updated annually with PubMed updates.

[0050] In addition, in order to better explore the correlation between target diseases and clinical examination indicators, the documents in XML format need to be annotated before storing them in the preset database. The information to be annotated includes title, author, abstract, keywords, etc.

[0051] Based on this, the process of storing XML format documents into the preset database is as follows: store the annotated XML format documents into the preset database.

[0052] Taking pancreatic cancer as an example, this section introduces the literature retrieval and storage processes, such as... Figure 2 As shown: The stages of pancreatic cancer risk assessment, diagnosis, treatment, and prognosis are input into PubMed separately. A search is performed in the PubMed database to obtain a list of PubMed IDs containing relevant literature for each stage. The literature in XML format is downloaded using the EFETCH tool and stored in a local PostgreSQL database. To reduce the amount of data processing, literature within a specific timeframe can be searched and downloaded.

[0053] Step S02: Obtain the set of clinical physical examination indicators.

[0054] Specifically, to better identify potential indicators related to a disease and a target disease, a set of clinical examination indicators can be obtained. This set of clinical examination indicators can be downloaded from the Mayo Clinic's official website: http: / / www.mayomedicallaboratories.com.

[0055] In step S03, the text mining model uses the clinical physical examination indicator set as an index set to search the literature set, and determines the associated indicator set of the target disease at each stage based on the frequency of each indicator in the clinical physical examination indicator set appearing in the literature set at different stages.

[0056] Specifically, the literature sets and clinical examination indicator sets for each stage are input into a pre-trained text mining model. The text mining model then mines the associated indicator sets for each stage of the target disease. Using the indicators in the clinical examination indicator set as indexes, the text mining model searches the titles, abstracts, keywords, and MeSH terms of the documents in the literature set. Based on the frequency of each indicator in the clinical examination indicator set appearing in the literature sets at different stages, the model determines the associated indicator sets for each stage of the target disease.

[0057] Taking pancreatic cancer as an example, this paper introduces the working process of a text mining model. The text mining model searches for each clinical examination indicator in the clinical examination indicator set within the literature sets of each stage, obtaining the frequency of occurrence of each clinical examination indicator in each literature set. Then, based on the frequency of occurrence, it obtains the set of associated indicators for each stage of pancreatic cancer. Finally, the clinical examination indicators in the associated indicator set are represented in radar format, as shown in the image below. Figures 3a-3d As shown, Figure 3a A set of related indicators for the pancreatic cancer risk assessment phase; Figure 3b A set of association indicators related to the diagnostic stage of pancreatic cancer; Figure 3c A set of correlation indicators related to the treatment stages of pancreatic cancer; Figure 3d This is a set of association indicators related to the prognostic stages of pancreatic cancer. Among them, [the following is a list of indicators]. Figure 3a It can be seen that there are 16 association indicators related to pancreatic cancer risk, among which somatostatin and hemoglobin appeared with frequencies of 59 and 57 respectively, showing a strong correlation with pancreatic cancer risk; Figure 3b It can be seen that there are 35 association markers related to the diagnosis of pancreatic cancer. Among them, in addition to somatostatin and hemoglobin, secretin and interleukin-2 also appear frequently, at 85 and 45 times respectively; Figure 3c It was found that a set of 30 correlation markers were associated with pancreatic cancer treatment, and neurotensin, leptin, cholecystokinin (CCK), adiponectin, and amyloid-β protein showed high correlations in pancreatic cancer treatment. Figure 3dIt can be seen that there are 23 correlation indicators related to the prognosis of pancreatic cancer. Although the retinol-binding protein, bilirubin (serum), gastrin (serum), and albumin ratio appear only twice, these clinical examination indicators still have a certain correlation with the prognosis of pancreatic cancer.

[0058] Step S04: Sort the indicators in the correlation indicator set of each stage by improving the TF-IDF method.

[0059] Specifically, TF-IDF theory involves finding and ranking keywords with key information from a corpus. TF-IDF is used to assess the importance of a word to a document. Here, TF stands for Term Frequency, and IDF stands for Document Frequency. Term frequency is the number of times a given word appears in a document, and document frequency can be obtained by dividing the total number of documents by the number of documents containing the word, and then taking the logarithm of the quotient. The importance of a keyword is directly proportional to the number of times it appears in a document, but inversely proportional to its frequency in the corpus. Therefore, the higher the TF-IDF value of a keyword, the stronger its relevance to the document it appears in, meaning the higher its influence on that document. Based on the TF-IDF values, a target association indicator set is determined, which medical professionals can use to diagnose target diseases.

[0060] This invention ranks the importance of detection indicators related to the target disease using TF-IDF, as follows:

[0061] Based on the traditional TF-IDF calculation method: we can first assume D to be a collection of documents, w to be a keyword, and d to be a specific document, where d∈D, and obtain the following formula (1):

[0062]

[0063] Among them, f w,d The value of f represents the number of times keyword w appears in document d, where the keyword is a clinical physical examination indicator; |D| represents the size of the document set, then f w,D |d| represents the number of times keyword w appears in the entire document set D, and |d| represents the total number of words in document d.

[0064] Index set q consists of a set of keywords w i Composition, where the index set is the set of related indicators, w i,d Calculate according to the above formulas respectively, and finally add the calculation results together, as shown in formula (2), where d∈D.

[0065]

[0066] This invention improves upon the traditional TF-IDF calculation method: the calculation method for the maximum likelihood estimation of the distribution of index t in document d is shown in formula (3):

[0067] p ml (t|M d )=tf (t,d) / dl d (3)

[0068] Where, p ml (t|M d ) represents the maximum likelihood estimate of the correlation index t in literature d, tf (t,d) It is the raw probability of the correlation index t appearing in literature d, dl d This represents the total number of indicators in document d. Assuming that the indicators are independent of each other, the ranking formula for the document is ∏. t∈Q p ml (t,d).

[0069] To ensure accurate estimation even with large datasets, the estimation formula is shown in formula (4):

[0070]

[0071] Among them, df t Let t be the document frequency of the indicator t.

[0072] To improve the robustness and accuracy of the algorithm, the following formula for the risk function of the geometric distribution is derived:

[0073]

[0074] in, This refers to the average frequency of t in the literature collection. As can be seen from the formula, as tf... (t,d) The further the value deviates from the normalized mean, the greater the risk estimated by the average probability. Therefore, the risk function will be applied as a mixture parameter to p(Q|M). d ) calculation,

[0075] Assume the following formula (6)

[0076]

[0077] Then we can obtain the following formula (7).

[0078]

[0079] Wherein, p(Q|M d The expression ) represents the query probability generated based on a specific set of clinical physical examination indicators Q, i.e., the TF-IDF value. The first term of Q, p(t|M... dThe second term, 1.0 - p(t|M), represents the probability of a set of clinical physical examination indicators appearing in the literature. d The ) represents the probability that the clinical physical examination indicator set Q does not appear in the literature. The clinical physical examination indicators at each stage are sorted based on TF-IDF values ​​to obtain the target-related indicator set for each stage of the target disease. For example, based on the improved TF-IDF method, the TF-IDF of the clinical physical examination indicator set related to pancreatic cancer risk assessment, diagnosis, treatment, and prognosis is calculated, yielding... Figure 4 The results are shown. Figure 4 As shown, based on the TF-IDF values ​​of various clinical examination indicators, it can be concluded that the clinical examination indicators related to pancreatic cancer are concentrated. The TF-IDF value of insulin-like growth factor 2 (IGF-2) is 9.8632, and the TF-IDF value of prothrombin fragment 1+2 is 7.5618, indicating that these two clinical examination indicators are key correlation indicators in pancreatic cancer.

[0080] This embodiment provides a PubMed-based medical text mining method that retrieves relevant literature on different stages of a target disease from the PubMed database and stores the relevant literature in the database, resulting in literature sets for different stages. The text mining model uses a set of clinical physical examination indicators as an index set to search within the literature sets, and determines the set of associated indicators for each stage of the target disease based on the frequency of occurrence of each indicator in the clinical physical examination indicator set across literature sets at different stages. An improved TF-IDF method is used to sort the associated indicators in each stage's associated indicator set, obtaining the target associated indicator set for each stage of the target disease. This embodiment uses the PubMed retrieval system to obtain a relatively comprehensive collection of relevant literature on different stages of the disease, reducing the limitations of data sources and making the data more comprehensive.

[0081] In this embodiment, the text mining model searches the literature set based on a set of clinical examination indicators to determine the associated indicator sets for each stage of the target disease. The text mining model considers the relationship between clinical examination indicators and the disease.

Claims

1. A medical text mining method based on PubMed, characterized in that, include: Relevant literature on different stages of the target disease is retrieved from the PubMed database and stored in a pre-defined database to obtain literature sets for different stages, including four stages of risk assessment, diagnosis, treatment and prognosis for the target disease. Obtain a set of clinical physical examination indicators; The text mining model uses the set of clinical physical examination indicators as an index set to search the literature set, and determines the set of associated indicators for each stage of the target disease based on the frequency of each indicator in the literature set at different stages. By improving the TF-IDF method, the indicators in the correlation indicator set of each stage are sorted to obtain the target correlation indicator set of each stage of the target disease, which can be used as a reference for medical staff when diagnosing the target disease.

2. The medical text mining method based on PubMed according to claim 1, characterized in that, The process of retrieving relevant literature on different stages of a target disease from the PubMed database and storing the literature in a predefined database includes: Enter each stage of the target disease into PubMed for retrieval to obtain a list of PubMed IDs containing relevant literature; Download the list of IDs and use the EFETCH tool to download the corresponding XML document for each ID based on the ID list; Store documents in XML format into a pre-defined database.

3. The medical text mining method based on PubMed according to claim 2, characterized in that, Before storing XML-formatted documents into the preset database, the following steps are also included: The literature should be annotated, and the information to be annotated includes the title, author, abstract, and keywords.

4. The medical text mining method based on PubMed according to claim 1, characterized in that, The process of ranking the correlation indicators in each stage's correlation indicator set by improving the TF-IDF method includes: The TF-IDF value of each indicator is calculated using the following formula: p ml (t|M d )=tf (t,d) / dl d , Where, p ml (t|M d R is the maximum likelihood estimate of index t in literature d. t,d Let p(Q|M) be the risk function, Q be the set of clinical physical examination indicators, and p(Q|M) be the risk function. d ) represents the query probability generated based on a set of clinical physical examination indicators, i.e., the TF-IDF value, p(t|M d ) represents the probability of a set of clinical physical examination indicators appearing in the literature, 1.0-p(t|M d ) represents the probability that a set of clinical physical examination indicators does not appear in the literature, ∏ t∈Q p ml (t,d) represents the ranking of document d, tf (t,d) It is the original probability of index t appearing in literature d, dl d It is the total number of indicators in literature d, df t For the document frequency of index t, This refers to the average frequency of t in the literature collection; The correlation indicators in the correlation indicator sets of different stages are sorted according to their TF-IDF values.

5. A medical text mining device based on PubMed, characterized in that, include: The literature acquisition unit retrieves relevant literature on different stages of the target disease from the PubMed database and stores the literature in a preset database to obtain literature sets for different stages, including four stages of risk assessment, diagnosis, treatment and prognosis for the target disease. The index retrieval unit retrieves the clinical physical examination indicator set; The indicator retrieval unit uses a text mining model to use the clinical physical examination indicator set as an index set to search the literature set, and determines the associated indicator set of the target disease at each stage based on the frequency of each indicator in the clinical physical examination indicator set appearing in the literature set at different stages. The indicator confirmation unit sorts the indicators in the relevant indicator set for each stage by improving the TF-IDF method, and obtains the target relevant indicator set for each stage of the target disease, which can be used as a reference for medical staff when diagnosing the target disease.

6. A medical text mining device based on PubMed, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the various steps of the PubMed-based medical text mining method as described in any one of claims 1-4.

Images