Method for mining literature source toxicity effect test information

Abstract

The invention relates to the technical field of information extraction, and particularly provides a method for mining literature source toxicity effect test information, which comprises the following steps: acquiring a plurality of related literatures from a literature library according to related nouns of a target toxicity effect; preprocessing the plurality of related literatures to obtain a plurality of target toxicity effect corpora; adding a preset information mining rule into the large language model; inputting the multiple target toxic effect corpora into a large language model, so that the large language model performs information mining on the multiple target toxic effect corpora based on an information mining rule; a mining result which aims at the first corpus set and comprises a plurality of preset mining items is output through a large language model, and the preset mining items comprise a test compound, an experimental object type, exposure duration, an effect and a result; and screening the mining results according to a preset threshold value set for the exposure duration to obtain a target result.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and information extraction technology, and in particular to a method for mining toxic effect test information from literature sources. Background Art

[0002] Information on the toxic effects of compounds plays an important role in environmental science, toxicology, pharmacy and other fields, and is helpful for risk assessment, mechanism explanation, activity prediction, etc. At present, the publicly available databases on the toxic effects of compounds are very limited, mainly including institutions or projects that implement in vitro high-throughput screening tests (such as Tox21, Broad Institute, and Scripps Research Institute Molecular Screening Center) and manually organized literature source information (such as ChEMBL). Among them, the latter is limited in the effects and compounds covered due to the high manpower and time costs.

[0003] Therefore, there is a need for a feasible method to automatically extract literature information using text mining tools to provide more comprehensive information. Summary of the invention

[0004] In view of this, the present invention provides a method for mining toxic effect test information from literature sources, comprising:

[0005] Obtain multiple relevant documents from the literature database based on the relevant terms of the target toxic effect;

[0006] Preprocessing multiple relevant documents to obtain multiple target toxicity effect corpora, wherein the multiple target toxicity effect corpora include a first corpus set;

[0007] Add the preset information mining rules to the large language model;

[0008] Inputting multiple target toxic effect corpora into the large language model so that the large language model can perform information mining on the multiple target toxic effect corpora respectively based on the information mining rules;

[0009] Outputting mining results including a plurality of preset mining items for the first corpus through the large language model, wherein the plurality of preset mining items include a test compound, an experimental subject type, an exposure duration, an effect, and a result;

[0010] According to the preset threshold set for the exposure duration, the mining results are filtered to obtain the target results.

[0011] According to an embodiment of the present invention, information mining rules are set through prompt word engineering, wherein the information mining rules include role prompt rules, input text rules, task description rules, detailed rules and error-prone prompt rules.

[0012] According to an embodiment of the present invention, setting information mining rules through prompt word engineering includes:

[0013] Set role prompt rules based on the area of ​​target toxic effects;

[0014] Setting input text rules according to the text format corresponding to the target toxic effect corpus;

[0015] Setting task description rules according to multiple preset mining items and preset target output formats;

[0016] Set detailed rules for each preset mining item;

[0017] Set error-prone prompt rules based on the error records in the historical mining records generated by the large language model.

[0018] According to an embodiment of the present invention, the plurality of preset mining items further include cell or model animal information, exposure method, evidence type, exposure concentration, and detection tissue or organ.

[0019] According to an embodiment of the present invention, multiple relevant documents are preprocessed to obtain multiple target toxic effect corpora including:

[0020] Data cleaning was performed on multiple relevant documents to remove table contents, spaces, headers, footers, marginal notes and page numbers, and multiple cleaned documents were obtained;

[0021] Perform data normalization on multiple cleaned documents to retain relevant sections of title, abstract, method and results, and obtain multiple normalized documents;

[0022] According to the preset compound names, effect types, experimental subject types and exposure conditions, the relevant chapters including methods and results in multiple regular documents are further screened and retained to obtain method result chapters to improve the density of target toxic effect information;

[0023] The relevant chapters of titles and abstracts in multiple organized documents, as well as the methods and results chapters, were spliced ​​according to the original order of multiple related documents to obtain multiple target toxic effect corpora.

[0024] According to an embodiment of the present invention, the mining results are screened according to a preset threshold set for the exposure duration, and the target results obtained include:

[0025] Based on a preset expression format, the numerical values ​​of the test compounds in the mining results are normalized to obtain reference data, wherein the preset expression format includes an InChiKey format or a SMILES format;

[0026] According to a preset threshold, the reference data is classified to obtain a plurality of data subsets representing various degrees of contamination;

[0027] Obtain target results based on reference data and multiple data subsets.

[0028] According to an embodiment of the present invention, prompt information for the second corpus is outputted through a large language model, and the prompt information is used to prompt that no mining results containing multiple preset mining items are mined for the target toxic effect corpus in the second corpus.

[0029] According to an embodiment of the present invention, n relevant documents from a plurality of relevant documents are randomly selected to form a first evaluation set;

[0030] Determine the real mining results of n relevant documents, wherein the real mining results include having real mining results and having no real mining results;

[0031] The precision, recall and F1 value of the large language model algorithm are determined based on the real mining results of n relevant documents and the mining results of the large language model. The precision, recall and F1 value are used to characterize the algorithm performance of the large language model.

[0032] According to an embodiment of the present invention, randomly selecting n relevant documents from a plurality of relevant documents to form a first evaluation set includes:

[0033] According to formula (1), the minimum number n of multiple relevant documents in the first evaluation set is determined, where n is a positive integer;

[0034] (1)

[0035] Wherein, in formula (1), 1-α is the preset target confidence level based on the expected large language model algorithm performance, Z is the Z score corresponding to the preset target confidence level, E is the preset target error range, and σ is the estimated value of the population standard deviation determined based on the size of the first evaluation set in the historical data.

[0036] According to an embodiment of the present invention, the minimum number m of the second evaluation set is determined according to formula (2);

[0037] (2);

[0038] Wherein, in formula (2), π is the estimated value of the overall proportion determined according to the size of the second evaluation set in the historical data, 1-α is the target confidence level preset according to the expected large language model algorithm performance, Z is the Z score corresponding to the preset target confidence level, and E is the preset target error range;

[0039] The second evaluation set is formed based on the minimum number m and the relevant documents with no actual mining results;

[0040] Determine the number of relevant documents in the second evaluation set that belong to the relevant documents in the second corpus;

[0041] The false positive rate of information mining using the large language model is obtained based on the minimum number m belonging to the second corpus and the number of prompt information returned, wherein the false positive rate is used to characterize the probability that the real mining result is no result but has a mining result.

[0042] According to an embodiment of the present invention, by collecting and preprocessing relevant literature on target toxic effects, a target toxicity corpus is obtained to improve information density, so that information mining can be performed through a large language model and mining rules set for the target toxic effects. By using a large language model to automatically process a large amount of literature, the model can identify complex patterns and relationships, significantly reducing the time and labor intensity of manual reading and data extraction. At the same time, the mining rules can be adjusted as needed, and can be flexibly adapted to different research needs or concerns. It can also ensure that the same mining standards and rules are applied in different documents, thereby improving the consistency and comparability of the mining results. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0044] Figure 1 A flowchart of a method for mining toxic effect test information from literature sources according to an embodiment of the present invention is schematically shown;

[0045] Figure 2 A schematic diagram of a flow chart for obtaining target toxic effect corpus according to an embodiment of the present invention is shown;

[0046] Figure 3 A schematic diagram of a flow chart for obtaining a target result according to an embodiment of the present invention is shown;

[0047] Figure 4 The structure block diagram of the device for mining toxic effect test information from literature sources according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0048] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0049] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0050] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0051] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0052] In the embodiments of the present invention, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage and other aspects of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.

[0053] In the process of implementing the present invention, it was found that traditional text mining uses rule-based methods and supervised learning methods, both of which require manual coding of complex rules or manual annotation of training corpora, and both perform poorly in the task of obtaining relationships between entities. In the attempts to obtain chemical-biological-pharmaceutical entity relationships using traditional text mining methods, most are limited to binary or ternary relationships, and the precision and recall rates obtained are low, rarely reaching 80%. The recently emerging large language model can directly extract semantic information, which has revolutionized text mining technology and can be used for multivariate relationship extraction with high precision and recall.

[0054] The challenges in mining information on the toxic effects of compounds through large language models mainly include how to obtain plain text files of relevant literature, how to shorten the text length while retaining the target information, how to design prompt word engineering to guide the response of large language models, and how to evaluate workflow performance.

[0055] To achieve the above-mentioned purpose, an embodiment of the present invention provides a method for mining toxic effect test information from literature sources, the flow chart of which is as follows: Figure 1As shown, including operations S101 to S106:

[0056] Operation S101: Acquire multiple relevant documents from a literature library according to relevant terms of the target toxic effect.

[0057] According to an embodiment of the present invention, the target toxic effect is a toxic effect set according to the purpose of the current literature mining, for example, it can be an oxidative stress toxic effect, a neurotoxic effect, a hepatotoxic effect, etc. The relevant terms of the target toxic effect include terms related to the target toxic effect, for example, the research field may be limited to environmental science, chemistry and pharmacy, toxicology; related experimental terms, biomarkers, pathological processes, known toxic pathways, etc. At the same time, the terms related to the experimental subjects can be used to limit the research topic. Use the method of excluding related subject words to exclude population studies, nanomaterials and microplastics related research, and research on specific gene knockout of experimental subjects.

[0058] For example, if the target toxic effect is hepatotoxic effect, related terms include hepatocellular damage, fatty degeneration, fibrosis or cirrhosis, as well as related specific biomarkers, pathological processes and toxic pathways.

[0059] According to an embodiment of the present invention, the document library may include multiple document libraries, not limited to the type and region of the document library, and the acquired documents may also include multiple forms. The method for acquiring relevant documents is generally to construct a document search formula and then download and obtain them.

[0060] For example, if the target toxic effect is oxidative stress toxicity, literature related to three oxidative stress markers (reactive oxygen species, malondialdehyde, and 8-hydroxydeoxyguanosine) is obtained, and cells, rats, mice, and zebrafish are selected as experimental subjects.

[0061] The search query can be constructed as follows:

[0062] WC = (“Environmental Sciences” OR “Chemistry, Medicinal” OR “Toxicology”) AND PY=2000-2023 AND (TS =((“oxidative stress” OR “oxidativedamage” OR “DNA damage” OR “lipid peroxidation”) AND (“ROS” OR “reactiveoxygen species” OR “8-OHdG” OR “8-hydroxy-2'-deoxyguanosine” OR “MDA” AND “serum” OR “blood” OR "plasma")) NOTTS = ("nano*" OR "microplastic$") NOTTS = ("knock$out")).

[0063] Operation S102: preprocessing a plurality of relevant documents to obtain a plurality of target toxic effect corpora, wherein the plurality of target toxic effect corpora include a first corpus set.

[0064] According to an embodiment of the present invention, the target toxicity corpus is text information obtained after preprocessing of relevant documents. Preprocessing may include extracting text content in relevant documents, removing parts that have no practical effect, and retaining content related to the target toxic effect. The first corpus set is a part of the target toxic effect corpus.

[0065] Operation S103: adding the preset information mining rules to the large language model.

[0066] According to an embodiment of the present invention, a large language model is an artificial intelligence model with deep learning capabilities, which can understand and generate natural language text. This model is usually trained with a large amount of text data to understand and generate language in a similar way to humans. Information mining rules are predefined guidelines or standards for large language models, which are used to guide the mining process of large language models when processing specific tasks. Information mining rules may include the type of information to be found, the depth of mining, the relevance threshold, etc.

[0067] Operation S104: inputting the plurality of target toxic effect corpora into the large language model so that the large language model performs information mining on the plurality of target toxic effect corpora respectively based on information mining rules.

[0068] According to an embodiment of the present invention, the target toxic effect corpus is input into a large language model so that the model can mine multiple target toxic effect corpora one by one according to information mining rules, wherein there should be a mined result for each target toxic effect corpus.

[0069] Operation S105: outputting mining results including a plurality of preset mining items for the first corpus through the large language model, wherein the plurality of preset mining items include a test compound, an experimental subject type, an exposure duration, an effect, and a result.

[0070] According to an embodiment of the present invention, a plurality of preset mining items are preset mining items. The first corpus is a part of a plurality of target toxicity corpora, which can be understood as each target toxicity corpus in the first corpus has a preset mining item. In other words, only target toxicity effect corpora that can mine the preset mining item can be used as part of the first corpus. The mining results include mining results corresponding to each target toxicity effect corpus, which can be understood as each target toxicity effect corpus has a mining result, and the mining results including multiple preset mining items for the first corpus are a collection.

[0071] The preset mining items include multiple preset items, which can be set according to actual needs. They can be understood as items that must have values ​​in the mining process. The test compound refers to the chemical substance whose toxic effects are tested in the experiment; the experimental subject type refers to the type of organism used for the test, such as cells, mice, etc.; the exposure duration refers to the length of time the experimental subject is exposed to the test compound; the effect refers to the biological effect caused by the test compound, such as oxidative stress, neurotoxicity, etc.; the result refers to the final conclusion of the experiment or the observed toxic effect.

[0072] Operation S106: Filter the mining results according to a preset threshold set for the exposure duration to obtain a target result.

[0073] According to an embodiment of the present invention, the target result refers to the classification that meets the preset threshold after screening. For example, after classification according to the preset threshold, the mining results are divided into two types, and the target result will include the number of relevant documents of the first type or the target toxicity corpus, and the number of relevant documents of the second type or the target toxicity corpus. Further, analysis and subsequent research can be performed based on the target result.

[0074] According to an embodiment of the present invention, by collecting and preprocessing relevant literature on target toxic effects, a target toxicity corpus is obtained to improve information density, so that information mining can be performed through a large language model and mining rules set for the target toxic effects. By using a large language model to automatically process a large amount of literature, the model can identify complex patterns and relationships, significantly reducing the time and labor intensity of manual reading and data extraction. At the same time, the mining rules can be adjusted as needed, and can be flexibly adapted to different research needs or concerns. It can also ensure that the same mining standards and rules are applied in different documents, thereby improving the consistency and comparability of the mining results.

[0075] According to an embodiment of the present invention, information mining rules are set through prompt word engineering, and the information mining rules include role prompt rules, input text rules, task description rules, detailed rules and error-prone prompt rules.

[0076] According to an embodiment of the present invention, prompt engineering is a method for optimizing model input, which guides the large language model to produce the desired output by designing effective prompts. The information mining rules of the large language model can be set through prompt engineering, which can include the setting of multiple rules, including role prompt rules, input text rules, task description rules, detailed rules and error-prone prompt rules.

[0077] According to an embodiment of the present invention, setting information mining rules through prompt word engineering includes:

[0078] Set role prompt rules based on the area of ​​target toxic effects;

[0079] Setting input text rules according to the text format corresponding to the target toxic effect corpus;

[0080] Setting task description rules according to multiple preset mining items and preset target output formats;

[0081] Set detailed rules for each preset mining item;

[0082] Set error-prone prompt rules based on the error records in the historical mining records generated by the large language model.

[0083] Role-hinting rules specify the role that the model should play when performing a task, such as an environmental scientist or a medical researcher, to adapt to the context and terminology of a specific domain.

[0084] For example, the role prompting rules are set for roles that are experts in the fields of environmental science, toxicology, and pharmacology, especially in the area of ​​target toxic effects.

[0085] The input text rules define the format and structure of the input text processed by the large language model to ensure the consistency and readability of the text. The target toxic effect corpus obtained after preprocessing has a format corresponding to the input text format in the input text rules.

[0086] For example, encapsulate input text using symbols that do not usually appear in literature, such as triple quotes, triple curly braces, etc.

[0087] Task description rules refer to the clear description of the tasks to be completed, including the mining objectives and expected results. The expected results can specify the output format so that the model can understand the context and purpose of the task.

[0088] Detailed rules refer to specific requirements for every detail of the task, including how to process data, what information points to extract, etc. For example, each element should be described in detail in combination with the target toxic effect and the experimental subject, including but not limited to the following rules:

[0089] (1) When the test compound is an agonist type, the extraction results are either increased or have no significant change.

[0090] (2) When the test compound is an antagonist type, the extraction results are decreased or have no significant change.

[0091] (3) The test compound should be a single substance, excluding compound categories (such as polycyclic aromatic hydrocarbons, metals), mixtures (such as organic extracts, water extracts, secondary organic aerosols, combinations of two or more compounds), environmental media (such as atmospheric fine particles, indoor dust), solvents, microplastics, and nanomaterials. If both the full name and abbreviation are available, the full name will be returned first.

[0092] (4) The types of experimental subjects are limited to the specified animal names, excluding plants and microorganisms. For example, the specified experimental subject types include cell lines, rodents (mice, rats, guinea pigs, hamsters), rabbits, monkeys, chickens, dogs, pigs, sheep, fish (zebrafish, medaka, rainbow trout, goldfish, tilapia, etc.), frogs (African clawed frogs, North American green frogs, etc.), oysters, mussels, fleas (large fleas, long-limbed fleas, sword fleas, etc.), fruit flies, nematodes, and non-target experimental subjects are excluded, such as plants (Arabidopsis, rice, wheat, green algae, etc.) and microorganisms (yeast, Escherichia coli, Salmonella).

[0093] (5) Set the output format of the information corresponding to the experimental subject type. For example, for rats and mice, the output should be (strain, age, sex, weight).

[0094] (6) Set the reference value of the exposure method corresponding to the experimental object type. For example, for cells, there is no need to extract the exposure method; for rats and mice, common exposure methods include diet, tube feeding, subcutaneous injection, intravenous injection, etc.

[0095] (7) Set the effect to the full name or abbreviation of the target toxic effect; exclude other effects.

[0096] (8) Set the output type of the results to include increase, decrease or no significant change; only include the results directly generated by the experimental test, exclude the results generated by the relevant analysis, and exclude the results of the cited literature. Only return the confirmed results.

[0097] (9) Output values ​​for setting the evidence type include specific dose or dose-dependent; where the results are obtained by comparing a specific dose with a control group or by observing a dose-effect relationship, respectively.

[0098] (10) Setting the exposure dose includes reference values ​​for each experimental subject type; for example, for cells and zebrafish, common units are nM (or nmol / L, nmol L -1 ), µM (or µmol / L, µmol L -1 ), mM (or mmol / L, mmolL -1 ), ng / L (or ng L -1 ), µg / L (or µg L -1 ), mg / L (or mg L -1 ), ng / mL (or ng mL -1 ), µg / mL (or µg mL -1 ), mg / mL (or mg mL -1 ); for rats and mice, common units are mg per kg body weight perday (or mg / kg bw / day, mg / kg bw / d, mg / kg / day, mg / kg / d, mg / kg, mg kg-1 day-1, mg kg-1d-1, mg kg-1).

[0099] (11) Set the reference value of the exposure time unit; for example, common units include h (hours), d (days), months, hpf, and dpf.

[0100] (12) Set the test tissues or organs including reference values ​​corresponding to each experimental object type. For example, it is not necessary to extract this information for cells; for rats and mice, common test tissues / organs include heart, liver, lung, kidney, brain, ovary, testis, etc.

[0101] Error-prone warning rules point out errors or common problems that may be encountered during information mining, reminding the model to pay attention to and avoid these errors. They include but are not limited to the following rules:

[0102] (1) Return the results in a specified output format without any further interpretation or analysis;

[0103] (2) The test compound was a single compound, excluding combinations of two or more compounds;

[0104] (3) When the full name of the test compound is available, the full name of the test compound should be extracted instead of the abbreviation;

[0105] (4) All elements should correspond to the test compound, especially the exposure dose, and only output the target effect while excluding other effects;

[0106] (5) For exposure dose, exposure time, and test tissue / organ, only output the values ​​directly related to the effect and result;

[0107] (6) If the target toxic effect corpus does not contain the required information, return no result and do not fabricate any information.

[0108] According to an embodiment of the present invention, by designing large language model mining rules through prompt word engineering, the large language model can perform information mining tasks more effectively. Through clear prompt word guidance, the model is more likely to return accurate results, improve accuracy and help eliminate ambiguity in natural language. The settings in the prompt word engineering can be adjusted according to different task requirements to improve the adaptability of the model.

[0109] According to an embodiment of the present invention, the plurality of preset mining items further include cell or model animal information, exposure method, evidence type, exposure concentration, and detection tissue or organ.

[0110] Cell or model animal information refers to the specific information of the cell type used in the toxic effect test or the model animal used as the experimental subject. Model animals include organisms used in scientific research, such as nematodes, fruit flies, zebrafish, mice, etc., because their genetic, physiological and developmental processes are similar to humans to some extent, they are often used as models of human diseases and biological research. Exposure method refers to the way in which the test compound contacts the experimental subject (such as cells or model animals), which may include oral, injection, inhalation, etc. Different exposure methods may affect the observation of toxic effects. Evidence type may refer to different sources and forms of evidence supporting the observation of toxic effects. Exposure concentration refers to the concentration of the test compound to which the experimental subject is exposed. The test tissue or organ is the specific part of the organism being evaluated for toxic effects, which may be damaged or dysfunctional due to exposure to the test compound.

[0111] In some specific embodiments of the present invention, the output format is also included in the task description rule, including setting the format of the output mining results, such as outputting through a table, and also including a description of the output value, such as setting the output indicating the test compound, experimental subject type, cell or model animal information, exposure method, effect, result, and evidence type to have a unique value, and setting the exposure concentration, exposure time, and test tissue / organ to have at least one value. In other words, if multiple exposure concentrations, exposure times, and test tissues / organs produce the same test results, they need to be merged into the same record.

[0112] For example, the preset mining items are set as compounds, effects, experimental subjects and conditions, and test results. The output format is specified as [test compound; experimental subject type; (cell or model animal information); exposure method; effect; result; evidence type; (exposure concentration 1, exposure concentration 2, exposure concentration 3...); (exposure time 1, exposure time 2, exposure time 3...); (test tissue / organ 1, test tissue / organ 2, test tissue / organ 3...)].

[0113] The output results are shown in Table 1 below:

[0114] Table 1

[0115] Source article DOI Compound Experimental subjects Experimental subject information Exposure effect result Type of evidence Exposure dose Exposure time Detection tissue / organ 10.1002 / xxxx [cyclophosphamide mice (Swissalbino,adult,female,-) intraperitoneal injection ROS increase specificdoses (100 mg / kg) (72 hr) (ovary)]

[0116] According to an embodiment of the present invention, the obtained relevant literature is preprocessed to obtain the target toxic effect corpus process as follows: Figure 2 As shown, the following operations S201 to S204 are included:

[0117] Operation S201: performing data cleaning on a plurality of related documents to remove table contents, spaces, headers, footers, marginal notes and page numbers, and obtaining a plurality of cleaned documents.

[0118] Operation S202: performing data normalization on the plurality of cleaned documents to retain relevant sections of the title, abstract, method and result, and obtaining a plurality of normalized documents.

[0119] Operation S203: further screening and retaining relevant chapters including methods and results in multiple organized documents according to preset compound names, effect types, experimental subject types and exposure conditions to obtain method result chapters to increase the density of target toxic effect information.

[0120] Operation S204: splicing the relevant sections of the titles and abstracts in the multiple organized documents, as well as the method and result sections, according to the original order of the relevant documents, to obtain multiple target toxic effect corpora.

[0121] According to an embodiment of the present invention, before step S201, the method further includes converting the formats of the downloaded multiple relevant documents into a plain text format in batches through software for subsequent information processing.

[0122] According to an embodiment of the present invention, in step S201, the relevant documents are cleaned to improve the quality of the document data in preparation for subsequent information mining. For example, by deleting paragraphs with less than or equal to 10 words, the charts in the document are deleted to remove the content of the charts, because these usually do not contain text information; delete unnecessary spaces to standardize the text format; delete headers and footers, which usually contain publishing information rather than research content; delete marginal notes, which may be comments or editorial annotations and are irrelevant to the main research content; delete page numbers to facilitate digital processing.

[0123] According to an embodiment of the present invention, in operation S202, the title is identified and retained by matching preset keywords; the background and references are deleted, and whether to retain is determined paragraph by paragraph based on whether the preset keywords appear and their positions, so as to retain only the title, abstract, methods and results chapters.

[0124] According to an embodiment of the present invention, in operation S203, paragraph selection is performed on the methods and results chapters to further improve information density, for example, whether to retain a specific paragraph is determined by whether and where four types of predefined named entities representing compounds, effects, experimental subjects, and exposure conditions appear.

[0125] According to an embodiment of the present invention, in operation S204, after multiple target toxic effect corpora are concatenated, the target toxic effect corpora are encapsulated using an input format set in an input text rule, such as triple quotation marks and triple curly braces.

[0126] According to an embodiment of the present invention, through the preprocessing process, automated data processing saves researchers time in manually screening and organizing literature; accurate screening through preset conditions reduces human errors and deviations; at the same time, the information most relevant to the research objectives is screened out, which improves the relevance and practicality of the information, and the standardized processing process ensures consistency between different documents, which is convenient for comparison and analysis, and the regularized and cleaned literature data is more suitable for text mining, machine learning and data analysis.

[0127] According to an embodiment of the present invention, the mining results are screened according to a preset threshold set for the exposure duration to obtain the target result process as follows: Figure 3 As shown, the following operations S301 to S303 are included:

[0128] Operation S301: normalizing the values ​​of the test compounds in the mining results based on a preset expression format to obtain reference data, wherein the preset expression format includes an InChiKey format or a SMILES format;

[0129] Operation S302: Classifying the reference data according to a preset threshold value to obtain a plurality of data subsets representing various degrees of contamination;

[0130] Operation S303: Obtain a target result according to the reference data and the multiple data subsets.

[0131] According to an embodiment of the present invention, in operation S301, the preset expression format is an expression method for uniformly expressing the test compound format. Since the compound may be expressed in different ways in the target toxicity corpus, such as structure, structural formula, molecular formula, etc., it is necessary to perform normalization. The reference data is a combination of the numerical values ​​of the test compound after normalization, and the reference data corresponds to the mining result, and the difference is only that the expression method of the test compound is different.

[0132] InChIKey is a compound identifier, a hash string generated by InChI (International Compound Identifier) ​​to ensure accurate and unique identification of compounds. SMILES is a specification for representing molecular structures using ASCII strings, describing a three-dimensional chemical structure through a string of characters. The InChiKey and SMILES corresponding to the test compound name can be determined by searching the relevant query website.

[0133] In some specific embodiments of the present invention, the mining results for which no structural information can be obtained are completely deleted during the normalization process.

[0134] According to an embodiment of the present invention, in operation S302, the reference data includes exposure durations that are consistent with the mining results, and the exposure durations are classified according to preset thresholds to obtain multiple data subsets representing different degrees of contamination, for example, four data subsets representing different degrees of contamination, namely, acute, subacute, subchronic and chronic.

[0135] According to an embodiment of the present invention, in operation S303, after the reference data is divided into different data subsets, the number and names of the outputted multiple data subsets may be set as target results.

[0136] For example, the target result is: a total of 12,988 records have corresponding structural information, covering 2,558 compounds. The exposure duration is classified into 5,292 acute records, 2,872 subacute records, 1,340 subchronic records, and 144 chronic records.

[0137] According to an embodiment of the present invention, by using a normalized expression format such as InChIKey or SMILES format, a unified and standardized representation of compounds is ensured, which facilitates comparison and identification; through normalization and classification, the effects of compounds under specific exposure durations can be retrieved and analyzed more quickly, thereby improving research efficiency, and normalized data subsets facilitate comparative analysis under different compounds, different studies or different experimental conditions.

[0138] According to an embodiment of the present invention, the method for mining toxic effect test information from literature sources also includes: outputting prompt information for the second corpus through a large language model, the prompt information being used to prompt: no mining results containing multiple preset mining items have been mined for the target toxic effect corpus in the second corpus.

[0139] According to an embodiment of the present invention, in the process of mining the target toxic effect corpus with a large language model, if the preset mining items: test compound, experimental subject type, exposure duration, effect and result are not mined, it will be considered that there is no result, and a prompt message will be returned. The prompt message can be prompted in the form of text prompts such as "no result" and "mining failure". The second corpus includes the target toxic effect corpus that outputs the prompt message, which can be understood as the target toxic effect corpus without mining results will be used as the second corpus. Further, in other error cases, such as failure to mine, garbled target toxic effect corpus, etc., prompt information is also returned. It can be understood that multiple copies of target toxic effect corpora are classified into the first corpus when mining with preset mining items, and all other cases are classified into the second corpus, and the multi-point target toxic effect corpus is equal to the sum of the first corpus and the second corpus.

[0140] According to an embodiment of the present invention, by returning a prompt message to clearly inform the user that no results containing preset mining items are found in the target toxic effect corpus, the user does not need to manually check a large amount of data to confirm whether there is missing information, thereby improving the transparency of the process.

[0141] According to an embodiment of the present invention, a method for evaluating a large language model algorithm is also provided, comprising:

[0142] Randomly select n relevant documents from multiple relevant documents to form the first evaluation set;

[0143] Determine the real mining results of n relevant documents, wherein the real mining results include having real mining results and having no real mining results;

[0144] The precision, recall and F1 value of the large language model algorithm are determined based on the real mining results of n relevant documents and the mining results of the large language model. The precision, recall and F1 value are used to characterize the algorithm performance of the large language model.

[0145] According to an embodiment of the present invention, the real mining results can be determined by manual mining. Precision P refers to the proportion of correct results to all returned results; recall R refers to the proportion of returned correct results to all correct results; F1 value is an indicator of comprehensive precision and recall, and the larger its value, the better the algorithm performance. P, R, and F1 values ​​are calculated by calculating the number of TP (true positive), FP (false positive), TN (true negative), and FN (false negative) based on the output mining results of the algorithm and the real mining results.

[0146] The values ​​of P, R, and F1 can be calculated using the following equations (3), (4), and (5).

[0147] (3);

[0148] (4);

[0149] (5);

[0150] Among them, in formula (3), formula (4) and formula (5), TP is the number of documents whose real mining results are consistent with the mining results of the large language model, FP is the number of documents whose real mining results are no mining results and the large language model returns mining results, and FN is the number of documents whose real mining results have results and the large language model returns no mining results, that is, prompt information.

[0151] According to an embodiment of the present invention, the minimum size of the first evaluation set can be determined by formula (1), and the minimum number n of multiple relevant documents in the first evaluation set is determined according to formula (1), where n is a positive integer;

[0152] (1)

[0153] Wherein, in formula (1), 1-α is the preset target confidence level based on the expected large language model algorithm performance, Z is the Z score corresponding to the preset target confidence level, E is the preset target error range, and σ is the estimated value of the population standard deviation determined based on the size of the first evaluation set in the historical data.

[0154] For example, use 30 documents as the initial evaluation set, run the algorithm and calculate the average of precision, recall and F1 value. Based on the results of these 30 documents, calculate the standard deviation (σ) of precision, recall and F1 value. The standard deviation of F1 value can be used as the estimated value of σ, or σ can be the estimated value of the overall standard deviation. Z corresponds to the selected confidence level (such as the Z value at 95% confidence level is about 1.96), and E is the acceptable error range (such as 0.05). Round the calculation result up to the nearest integer to obtain the final minimum evaluation set size.

[0155] In some specific embodiments of the present invention, the actual results of the target toxicity corpus can also be determined first, that is, by manually preprocessing the relevant literature, and then comparing the actual preprocessing results with the preprocessing results processed by the model, so as to calculate the P, R, and F1 values ​​of the target toxic effect corpus obtained by preprocessing the relevant literature with the large language model, and evaluate the preprocessing performance of the large language model.

[0156] Furthermore, when the P value and the R value are set to be above 80%, the large language model algorithm is considered to be reliable.

[0157] For example, the estimated value of the overall standard deviation of the first evaluation set of randomly selected 30 articles in the historical data is 0.3, the 95% confidence interval and the error range are 0.9, and the evaluation set size n is calculated to be 35 documents. The P, R, and F1 values ​​of the text preprocessing step obtained from the evaluation set are 0.41, 0.91, and 0.57, respectively, and the P, R, and F1 values ​​of information extraction are 0.91, 0.81, and 0.86, respectively.

[0158] According to an embodiment of the present invention, the evaluation of the performance of the large language model algorithm further includes:

[0159] Determine the minimum number m of the second evaluation set according to formula (2);

[0160] (2);

[0161] Wherein, in formula (2), π is the estimated value of the overall proportion determined according to the size of the second evaluation set in the historical data, 1-α is the target confidence level preset according to the expected large language model algorithm performance, Z is the Z score corresponding to the preset target confidence level, and E is the preset target error range;

[0162] The second evaluation set is formed based on the minimum number m and the relevant documents with no actual mining results;

[0163] Determine the number of relevant documents in the second evaluation set that belong to the relevant documents in the second corpus;

[0164] The false positive rate of information mining using the large language model is obtained based on the minimum number m and the number of prompt information returned, wherein the false positive rate is used to characterize the probability that the actual mining result is no result but has a mining result.

[0165] According to an embodiment of the present invention, the actual mining results of all relevant documents in the second evaluation set are no results, and the false positive rate can be determined by determining the number of relevant documents in the second evaluation set that belong to the large language model corpus, i.e., the large language model outputs prompt information. The false positive rate can also be understood as a machine hallucination, that is, when there is no result but the large language model mines the result.

[0166] For example, the estimated value of the correct proportion of 30 randomly selected articles is 0.9, and the 95% confidence interval and error range are 0.9, and the second evaluation set size m is calculated to be 35 articles. The proportion of "no results" that can be correctly returned from the second evaluation set is 97%. It can be understood that overall, the performance of this workflow on this task is reliable.

[0167] As another aspect of the present invention, a device for mining toxic effect test information from literature sources is provided.

[0168] like Figure 4 As shown, the device 400 for mining toxic effect test information from literature sources in this embodiment includes an acquisition module 410 , a preprocessing module 420 , a rule module 430 , an input module 440 , a first output module 450 and a second output module 460 .

[0169] The acquisition module 410 is used to acquire a plurality of relevant documents from the document library according to the relevant terms of the target toxic effect. In one embodiment, the acquisition module 410 can be used to perform the operation S101 described above, which will not be described in detail here.

[0170] The preprocessing module 420 is used to preprocess the multiple relevant documents to obtain multiple target toxic effect corpora. In one embodiment, the preprocessing module 420 can be used to perform the operation S102 described above, which will not be repeated here.

[0171] The rule module 430 is used to add the preset information mining rules to the large language model. In one embodiment, the rule module 430 can be used to perform the operation S103 described above, which will not be described in detail here.

[0172] The input module 440 is used to input multiple target toxic effect corpora into the large language model so that the large language model can perform information mining on the multiple target toxic effect corpora based on the information mining rules. In one embodiment, the input module 440 can be used to perform the operation S104 described above, which will not be repeated here.

[0173] The first output module 450 is used to output the mining results including multiple preset mining items for the first corpus through the large language model, wherein the multiple preset mining items include test compounds, experimental subject types, exposure durations, effects, and results. In one embodiment, the first output module 450 can be used to perform the operation S105 described above, which will not be repeated here.

[0174] The second output module 460 is used to filter the mining results according to a preset threshold set for the exposure duration to obtain a target result. In one embodiment, the second output module 460 can be used to perform the operation S106 described above, which will not be described in detail here.

[0175] According to an embodiment of the present invention, any multiple modules among the acquisition module 410, the preprocessing module 420, the rule module 430, the input module 440, the first output module 450, and the second output module 460 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the acquisition module 410, the preprocessing module 420, the rule module 430, the input module 440, the first output module 450, and the second output module 460 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in a proper combination of any of them. Alternatively, at least one of the acquisition module 410, the preprocessing module 420, the rule module 430, the input module 440, the first output module 450 and the second output module 460 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0176] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.

[0177] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for mining toxic effect test information from literature sources, comprising: Obtain multiple relevant documents from the literature database based on the relevant terms of the target toxic effect; Preprocessing the plurality of relevant documents to obtain a plurality of target toxic effect corpora, wherein the plurality of target toxic effect corpora include a first corpus set; Add the preset information mining rules to the large language model; Inputting the plurality of target toxic effect corpora into a large language model, so that the large language model performs information mining on the plurality of target toxic effect corpora respectively based on the information mining rule; Outputting mining results including a plurality of preset mining items for the first corpus through the large language model, wherein the plurality of preset mining items include a test compound, an experimental subject type, an exposure duration, an effect, and a result; According to a preset threshold set for the exposure duration, the mining results are screened to obtain target results.

2. The method according to claim 1, further comprising: in, The information mining rules are set through the prompt word engineering, wherein the information mining rules include role prompt rules, input text rules, task description rules, detailed rules and error-prone prompt rules.

3. The method according to claim 2, wherein: The information mining rules set by the prompt word engineering include: Setting role prompting rules according to the field of the target toxic effect; Setting input text rules according to a text format corresponding to the target toxic effect corpus; Setting task description rules according to multiple preset mining items and preset target output formats; Set detailed rules for each preset mining item; The error-prone prompting rules are set according to the error records in the historical mining records generated by using the large language model.

4. The method according to claim 1, wherein: The multiple preset mining items also include cell or model animal information, exposure method, evidence type, exposure concentration, and detection tissue or organ.

5. The method according to claim 1, wherein: The preprocessing of the plurality of relevant documents to obtain a plurality of target toxic effect corpora comprises: Performing data cleaning on the multiple relevant documents to remove table contents, spaces, headers, footers, marginal notes and page numbers, thereby obtaining multiple cleaned documents; Performing data normalization on the multiple cleaned documents to retain relevant sections of the title, abstract, method and result to obtain multiple normalized documents; According to the preset compound names, effect types, experimental subject types and exposure conditions, the relevant chapters including methods and results in the multiple organized documents are further screened and retained to obtain method result chapters to improve the density of target toxic effect information; The relevant sections of the titles and abstracts in the multiple organized documents, as well as the method results section, are spliced ​​according to the original order of the multiple related documents to obtain multiple target toxic effect corpora.

6. The method according to claim 1, wherein: The mining results are screened according to the preset threshold set for the exposure duration to obtain the target results including: Based on a preset expression format, normalizing the values ​​of the test compounds in the mining results to obtain reference data, wherein the preset expression format includes an InChiKey format or a SMILES format; Classifying the reference data according to the preset threshold to obtain a plurality of data subsets representing various degrees of contamination; A target result is obtained according to the reference data and the multiple data subsets.

7. The method according to claim 1, further comprising: Prompt information for the second corpus is outputted through the large language model, and the prompt information is used to prompt that no mining results containing the multiple preset mining items are mined out for the target toxic effect corpus in the second corpus.

8. The method according to claim 7, wherein: Also includes: Randomly selecting n relevant documents from the plurality of relevant documents to form a first evaluation set; Determine the real mining results of the n related documents, wherein the real mining results include having real mining results and having no real mining results; The precision, recall and F1 value of the large language model algorithm are determined according to the real mining results of the n relevant documents and the mining results of the large language model, wherein the precision, recall and F1 value are used to characterize the algorithm performance of the large language model.

9. The method according to claim 8, wherein: The randomly selecting n relevant documents from the plurality of relevant documents to form a first evaluation set comprises: According to formula (1), the minimum number n of multiple relevant documents in the first evaluation set is determined, where n is a positive integer; (1) Wherein, in formula (1), 1-α is the target confidence level preset according to the expected large language model algorithm performance, Z is the Z score corresponding to the preset target confidence level, E is the preset target error range, and σ is the estimated value of the population standard deviation determined according to the size of the first evaluation set in the historical data.

10. The method according to claim 8, wherein: Also includes: Determine the minimum number m of the second evaluation set according to formula (2); (2)； Wherein, in formula (2), π is the estimated value of the overall proportion determined according to the size of the second evaluation set in the historical data, 1-α is the target confidence level preset according to the expected large language model algorithm performance, Z is the Z score corresponding to the preset target confidence level, and E is the preset target error range; The second evaluation set is formed according to the minimum number m and the relevant documents whose actual mining results are no results; Determine the number of relevant documents in the second evaluation set that belong to the second corpus; The false positive rate of information mining using the large language model is obtained according to the minimum number m and the number of prompt information returned, wherein the false positive rate is used to characterize the probability that the real mining result is no result but has a mining result.

Images