Corrosion inhibitor field literature text mining and data cleaning method based on large language model

Abstract

The invention provides a corrosion inhibitor field literature text mining and data cleaning method based on a large language model, and belongs to the field of material corrosion inhibitor and data mining. The method comprises the steps that firstly, a table of an original text is extracted, the table and the text are preprocessed through an embedding model, and respective vector embedding is obtained; meanwhile, manually setting two sets of corresponding system cue words, and inputting the system cue words into the large language model for vector embedding; performing similarity retrieval with vector embedding of a corresponding table / text, and reserving table / text fragments higher than a threshold value as uncleaned corrosion inhibitor data; and extracting the corresponding json data to the excel table. Creating batch tasks in batches, traversing original contents of tables / texts in all the tasks, setting the batch tasks of the tables / texts of the same original text as the same request-id, and storing the batch tasks as the same file name, so as to align the tables and the texts; and finally, respectively cleaning. According to the method, the data integrity and accuracy are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of material corrosion inhibitors and data mining, and relates to a method for mining literature texts in the field of corrosion inhibitors based on a large language model (LLM) and data cleaning. BACKGROUND

[0002] The screening and design of corrosion inhibitors are of great significance in metal corrosion protection. However, the current method for obtaining corrosion inhibitor-related data (such as name, concentration, test method, temperature, time, and corrosion inhibition efficiency) is for researchers to manually search through a large number of academic papers, which is time-consuming and labor-intensive.

[0003] Text mining processes a large amount of literature data through computers to screen valuable knowledge content, usually relying on NLP and ML techniques. Text mining can convert unstructured data into structured information, thereby improving the efficiency and accuracy of data analysis, making subsequent data processing more efficient and reliable, and greatly improving research efficiency.

[0004] Table accurate extraction has always been a big problem in the application field of large models. Existing literature mining techniques are mainly based on manual annotation or traditional text information extraction methods, which have the following problems:

[0005] 1) Incomplete table information extraction: Traditional text parsing methods have low recognition rates for table content, especially for tables with cross-row, cross-column merged cells or complex formats, which are prone to data loss or misplacement.

[0006] 2) Difficulty in aligning text and table information: The text description in the literature has a corresponding relationship with the table content, but existing methods lack effective automated alignment strategies, resulting in loss of context information and affecting the integrity and usability of the data.

[0007] 3) Low data cleaning efficiency: The extracted data often has inconsistent formats, mixed units, or spelling errors, and manual cleaning is time-consuming and prone to errors.

[0008] 4) Performance of large language model direct extraction of tables is limited: Due to the limitations of the model in understanding table structure, the direct extraction result accuracy is low, especially in the specific format of the chemical field.

[0009] 5) Large language model extraction efficiency is low, and tables and texts cannot be aligned when parallel extraction:

[0010] For calling large models, parallel processing through API calls will cause sequential disorder, and lack of important markers to align tables and texts, resulting in data errors.

[0011] In the literature of corrosion inhibitors and related scientific research fields, the arrangement of the text and tables has a relatively fixed rule:

[0012] The key experimental parameters and results (such as concentration, temperature, medium, test method, inhibition efficiency, etc.) are recorded in the form of tables; the text is mostly background or summary description, although the information density is low, but it is indispensable for distinguishing the context.

[0013] If the above two types of content are directly combined and input into a large language model without distinction, it is easy to cause the generation of fictional content ("hallucination") and factual deviation due to inconsistent semantic granularity and field alignment confusion.

[0014] Data quality plays a decisive role in the training of machine learning and large language models, and its importance is significantly higher than that of data size. After completing the literature data mining, the data cleaning steps such as term specification, unit unification, abnormal value and missing value correction in the field of corrosion inhibitors are indispensable, among which the standardized processing based on professional terminology system is particularly important.

[0015] Therefore, in the field of corrosion inhibitors and other scientific research, there is an urgent need for a method that can accurately analyze tables and separate tables from text content, in order to efficiently obtain structured information, align tables-text and evidence traceability in large quantities of literature, and complete standardization and quality checking through the cleaning mechanism combined with domain knowledge and rules / models, thereby improving the reliability and efficiency of downstream modeling and analysis. SUMMARY

[0016] In order to solve the problems of low accuracy and recall rate of table extraction and non-uniformity of units or professional term description in the field of corrosion inhibitors, the present application provides a large language model-based corrosion inhibitor field literature text mining and data cleaning method, which significantly improves the completeness and accuracy of the data, and is suitable for high-throughput processing of large quantities of corrosion inhibitor literature.

[0017] The large language model-based corrosion inhibitor field literature text mining and data cleaning method has the following specific steps:

[0018] Step 1: For the original text of the corrosion inhibitor field in XML format, the existing tables are extracted, and the original text is divided into two parts: tables and text;

[0019] The extraction of the table is as follows: first, convert the XML file to HTML format, and then use tools such as BeautifulSoup to extract the table row by row.

[0020] Step 2: Use the Embeddings embedded model to preprocess the table and text respectively, and get their respective vector embeddings input into the large language model;

[0021] For text, preprocessing is performed using the LangChain framework. The preprocessing is as follows:

[0022] First, the long text in the original text is split into text segments chunks by the RecursiveCharacterTextSplitter class in the langchain_text_splitters module.

[0023] Then, the text segments chunks are converted into embedding vectors using the embedding model Embeddings, and the original text is converted into numerical values.

[0024] For tables, the embedding model Embeddings is used to convert the table content into embedding vectors, and the original table content is converted into numerical values.

[0025] Step three, for table and text content, two sets of corresponding system prompt words are manually set and input into the large language model for vector embedding of system prompt words.

[0026] Step four, the vector embedding of the two sets of system prompt words is respectively compared with the vector embedding of the corresponding table / text, and the table / text segments with a similarity higher than the set threshold (or the top k with the highest similarity) are retained as the uncleaned corrosion inhibitor data; otherwise, they are discarded.

[0027] Step five, the uncleaned corrosion inhibitor data is extracted to obtain the corresponding json structure data, which is uniformly extracted into the excel table of the table / text.

[0028] Step six, multi-task parallel processing, batch creation batch task, traversal of all tasks in the original content of several tables / texts, the original content is processed to obtain high similarity segments, output as json data, the same original text table / text batch task is set to the same request-id, saved as the same file name, so as to align the table and text.

[0029] The same original text is preprocessed to obtain embedding vectors input into the large language model, and after internal similarity retrieval, the output is uncleaned table / text, which is extracted to obtain respective json structure and converted into respective excel table; the table and text in the two excel tables are aligned according to the file name and request-id.

[0030] Combined with batch multi-task parallel processing, the efficiency can be doubled.

[0031] Step seven, clean up the aligned table / text, and unify the different expressions of the original data.

[0032] Cleaning is to align the numerical information units and unify the professional terms; for regular data-temperature and time, such as: conversion of Celsius to Kelvin temperature, use the rules combined with python script to assist batch cleaning;

[0033] For the numerical information contained in the sentence information, such as the element composition statement of steel materials, after extracting the content of each element, the api of deepseekR1 large language model is used to clean up the prompt words.

[0034] The number of each cleaning is set according to the complexity of the problem.

[0035] The advantages of the present application are:

[0036] 1) Strong pertinence: by combining LangChain framework, prompt word optimization and RAG retrieval, the recall rate and accuracy of literature information in the field of corrosion inhibitor are significantly improved.

[0037] 2) Strong table analysis ability: effectively solve the problem of complex table analysis and alignment.

[0038] 3) High data quality: semi-automatic cleaning and correction greatly reduce the manual workload and improve the data consistency.

[0039] 4) Good scalability: the method can be applied to data mining and knowledge extraction in other scientific fields.

[0040] 5) High efficiency: the method combines batch task submission to realize multi-task parallel processing, which significantly improves the efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 A flowchart of the present application, a literature text mining and data cleaning method in the field of corrosion inhibitor based on a large language model.

[0042] Figure 2 A data sample extracted by the large language model of the present application.

[0043] Figure 3 The final cleaned data format of the embodiment of the present application.

[0044] Figure 4 The workflow diagram of the large language model extraction module of the present application.

[0045] Figure 5 The excel storage example of the text information of the embodiment of the present application.

[0046] Figure 6For example of table analysis result in the embodiment of the application.

[0047] Figure 7 For example of table information extracted in the embodiment of the application. DETAILED DESCRIPTION

[0048] In order to facilitate those skilled in the art to understand and implement the present application, the present application will be further described in detail below in combination with the drawings and embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor shall belong to the scope of protection of the present application.

[0049] The present application belongs to the field of literature data processing for corrosion inhibitor research, and specifically relates to automatic extraction and alignment of corrosion inhibitor literature text and table information based on large language model (LLM), retrieval-augmented generation (RAG) and prompt-based generation; and a method for cleaning and correcting mined semi-structured data.

[0050] Through LangChain framework, retrieval-augmented generation, prompt engineering and step-by-step processing flow, the literature text and table are separated and structured extracted, and automatic alignment and error correction across sources are realized; at the same time, combined with Python script and large model prompt template, semi-automatic professional data cleaning such as unit standardization, field verification and abnormal value processing is performed, so as to significantly improve the integrity and accuracy of the data, and is suitable for high-throughput processing of large amount of corrosion inhibitor literature.

[0051] The corrosion inhibitor field literature text mining and data cleaning method based on large language model, as shown in Figure 1 The specific steps are as follows:

[0052] Step one, for the original text of the corrosion inhibitor field in XML format, the existing table is extracted, and the original text is divided into table and text two parts;

[0053] LangChain framework is used for data mining to obtain the original text of historical literature in XML format, and the table is extracted as follows: first, the XML file is converted into HTML format, and then tools such as BeautifulSoup are used to extract the table row by row.

[0054] Step two, use Embeddings embedded model to pre-process the table and text respectively to obtain their respective vector embedding input into the large language model;

[0055] For text, preprocessing is performed using the LangChain framework. The preprocessing is as follows:

[0056] First, the long text in the original text is cut into text segments chunks by the RecursiveCharacterTextSplitter class in the langchain_text_splitters module.

[0057] The specific cutting method uses fixed character length slicing, with each 1500 tokens sliced. To avoid semantic loss, the overlapping part of the adjacent two segments is 50 tokens.

[0058] Then, the text segments chunks are converted into embedding vectors using the embedding model Embeddings, and the original text is converted into numerical values.

[0059] Here, the vector embedding space inside the large model is utilized. The closer the semantics of each segment, the greater the cosine similarity of the embedding vectors.

[0060] For tables, the table content is converted into embedding vectors using the embedding model Embeddings, and the original table content is converted into numerical values.

[0061] Step three, for table and text content, two sets of corresponding system prompt words are manually set and input into the large language model to perform vector embedding of the system prompt words.

[0062] By reading a large number of corrosion inhibitor field literature, the main knowledge fragments to be mined are extracted, especially tables. Combined with corrosion inhibitor knowledge, the law of each data occurrence is observed, and the most suitable system prompt word is selected by combining the optimized prompt word designed by the mainstream large language model, thereby constructing a large language model in the field of corrosion inhibitors. The system prompt word is also vector embedded, and the embedding vector of the system prompt word is compared with the vector embedding of each segment after slicing. The high similarity part is mined, and the low similarity part is not mined. Combined with the large language model, the literature content to be mined is input to obtain the uncleaned corrosion inhibitor data. The specific data mining examples can be referred to in the attached Figure 2 .

[0063] Step four, the vector embedding of the two sets of system prompt words is compared with the vector embedding of the corresponding table / text, and the table / text segments with a similarity higher than the set threshold (or the top k with the highest similarity) are retained as uncleaned corrosion inhibitor data; otherwise, they are discarded.

[0064] Step five, extract the data of the uncleaned corrosion inhibitor, get the corresponding json structure data, and uniformly extract into the excel table of the table / text respectively.

[0065] Step six, multi-task parallel processing, batch creation of batch tasks, traversal of all tasks in several tables / texts of the original content, the original content is processed to get high similarity fragments, output as json data of the same original text of the table / text of the batch task, set the same request-id, save the same file name, and align the table and text;

[0066] The same original text is preprocessed to get embedded vector input into a large language model, and after internal similarity retrieval, it outputs the uncleaned table / text, extracts the respective json structure, and converts it into respective excel tables; The table and text in the two excel tables are aligned according to the file name and request-id.

[0067] Combined with batch multi-task parallel processing, the efficiency can be doubled.

[0068] Step seven, clean the aligned table / text, and unify the different expressions of the original data.

[0069] For regular data-temperature and time, such as: Celsius to Kelvin temperature, use the rule to assist batch cleaning with python script;

[0070] For the numerical information contained in the sentence information, such as the element composition statement of the steel material, after extracting the content of each element, the api of the deepseekR1 large language model is used to assist in cleaning the prompt words for cleaning.

[0071] The number of each cleaning is set according to the complexity of the problem.

[0072] The excavated data has many phenomena of misalignment of numerical information units and non-uniformity of professional terms. Through professional knowledge, the prompt words are designed and optimized, and the optimal version prompt words are selected. The large language model api is called to extract in batches, and the numerical information units in the corrosion inhibitor data are aligned and the professional terms are unified. For each extracted feature data to be cleaned, the cleaning method is also different. For features with certain regularity, use python script to clean. For some irregular features such as concentration "100mg·L−1, 1.1 × 10−6 wt.%"; For the sentence information containing numerical information, such as the material composition sentence "0.05% C, 0.02% S, 0.28% Mn, 0.02% P, 0.02% Si, 99.61% Fe", the content of each element needs to be extracted.

[0073] Generally, the deepseekR1 large model is used to assist in cleaning the design prompt words. The cleaning process does not change the original data, but only unifies the different expressions of the original data. Combined with batch cleaning of the large model to improve efficiency and facilitate subsequent machine learning training. The cleaned data example can refer to the attached Figure 3 .

[0074] The whole large language model workflow is attached Figure 4 .

[0075] The present application can be divided into three modules: text information extraction, table information extraction, and table text information alignment and cleaning.

[0076] Text information extraction: this method constructs a large language model-based body text mining process. The process combines the LangChain framework, semantic vector retrieval (Embedding-Based Retrieval) and prompt-based generation (Prompt-Based Generation) three parts, which belongs to the typical retrieval augmented generation (Retrieval-Augmented Generation, RAG) architecture. The literature is first converted and split, then converted into semantic vectors and stored in the database, and finally in the task execution process, the most relevant text blocks are retrieved and combined with task prompts, and the large language model is called to complete structured information extraction.

[0077] The downloaded XML is converted into a simple format such as Markdown, and the literature data mining process uses the LangChain framework. In the text preprocessing part, the RecursiveCharacterTextSplitter class in the langchain_text_splitters module is used to split long text into smaller chunks. Finally, the text chunks are converted into embedding vectors using the embedding model ZhipuAIEmbeddings. By generating embedding vectors, the original text is converted into a numerical representation that the model can directly process.

[0078] Each task design corresponds to a prompt template (Prompt Template) prompt word example see Table 1, used to clearly indicate the LLM to extract the specified information content from the input article segment. The generated long text is passed to the corresponding LLMChain instance to perform specific information extraction tasks. LLMChain combines the pre-defined prompt template (Prompt Template) for knowledge extraction, and combines few shot to ensure that the large model output is in json format.

[0079] Table

[0080]

[0081] The method designs a batch automatic task process for question and answer structured extraction based on the existing large model platform. Each text content is combined with a prompt word and placed in a request, which is structured into a standardized JSON request format and processed uniformly through the Batch API interface provided by the large model website. Each Batch task can upload up to 100 requests, and the average processing time is 30 minutes. Multiple tasks can be processed simultaneously, and text information can be extracted in batches.

[0082] After information extraction and analysis, the execution results of each task are written to the result file (result_jsonl). The extracted examples are shown in Figure 5 , the red circle part is the request-id, and the script is used to extract the json format data uniformly into the excel table. The table text alignment method will use the request-id.

[0083] Table information extraction: used to accurately extract corrosion efficiency data from structured tables in literature. The overall process includes a table parsing stage and a large language model structured extraction stage.

[0084] First, convert the XML file to a simplified HTML format to facilitate row-by-row extraction of the table using tools such as BeautifulSoup. During the actual table parsing process, there are still some structural problems that affect the accuracy of information extraction. Some columns in the table (such as inhibitor names) use the "empty line inheritance" mode for aesthetics, i.e. when using the same inhibitor for multiple consecutive rows, only the first row displays the name and the remaining rows are left blank.

[0085] For specific cases, see Figure 6 .

[0086] Design prompt words, in order to ensure that the data output is in json format, use few shot to ensure that the output of the large model is in the json format we specify.

[0087] Submit batch tasks, download batch task json results, and use scripts to extract json format data uniformly into excel tables. The extracted table information is shown in Figure 7 .

[0088] Table text information alignment and cleaning: use file name alignment method to align table text: when downloading documents, make two copies of each document, one for extracting tables and one for extracting documents, and ensure that the file name of each document does not change, which can also ensure that the request-id does not change, so that the text and table can be aligned.

[0089] For data cleaning, combine mainstream large models such as DeepseekR1, call api and process in parallel in batches, which greatly improves efficiency while ensuring accuracy. Each extracted feature data cleaning, each feature cleaning method is also different, for some regular features, use python script to clean, for some irregular features such as concentration "100 mg·L−1, 1.1 × 10−6 wt.%"; For sentence information containing numerical information, such as material composition sentence "0.05% C, 0.02% S, 0.28% Mn, 0.02% P, 0.02% Si, 99.61% Fe", need to extract the content of each element, etc. Generally, design prompt words are selected using DeepseekR1 large model to assist cleaning.

[0090] Since large models have boundaries, it is not ideal to clean a large amount of data at once, so this method designs a technique to clean individual data at a time, and the specific number of data is also determined according to the complexity of the problem. Using large models and scripts greatly improves data cleaning efficiency, and an average of 3 hours can process 10,000 data, and can be processed in parallel, running multiple tasks at a time. Table 2 is an example of data after concentration cleaning. The first column is the original data, and the second column is the cleaned data. Table 3 is an example of cleaning the composition of the matrix material. The first column is the original statement, and the following columns are the cleaned data.

[0091] Table

[0092]

[0093] Table 3

[0094]

[0095] To verify the effectiveness of the proposed information extraction strategy, this example selects 100 samples from real literature to construct an artificial annotated validation set, and evaluates the extraction accuracy of the RAG extraction path based on the text and the question and answer extraction path based on the table, combined with the glm-4-plus type large model. The evaluation adopts field-level matching as the basic standard, and statistics the precision, recall and F1 value and other indicators.

[0096] (1) Precision: The proportion of correct fields predicted by the model to the total number of its predictions, used to measure the "precision" of the extraction results.

[0097]

[0098] (2) Recall: The proportion of correctly extracted fields successfully predicted by the model to all fields that should be extracted, used to measure the "completeness" of the extraction results.

[0099]

[0100] (3) F1 value: The harmonic mean of precision and recall, which is a comprehensive evaluation of the overall performance of the model.

[0101]

[0102] Among them, TP represents the correctly extracted fields (True Positive), FP represents the incorrectly extracted fields (False Positive), and FN represents the fields that have not been extracted (False Negative).

[0103] To ensure the comprehensiveness of the evaluation, the method performs field-level comparison on each record from the structured output, checking whether the field name, value, and unit are consistent with the manual annotation. For continuous variable fields such as efficiency, temperature, and time, a certain range of value tolerance (e.g., ±0.1% or ±1K) is allowed to be considered as a match. At the same time, for omitted fields in the table (such as the omission of inhibitor names in a certain row), if the model can correctly inherit the previous value, it is also considered as an effective match.

[0104] The evaluation results in Table 4 show that the accuracy, recall rate, and F1 score of the corrosion inhibition efficiency module are the highest, indicating that the model performs best in this module. The accuracy of the corrosion inhibitor name module is relatively low, at 0.8333, but the recall rate is high, at 0.9524, indicating that although there is a certain misjudgment in identifying corrosion inhibitor names, the model can cover most of the true corrosion inhibitor names. This is related to the inconsistent naming methods of corrosion inhibitors in some literature, ambiguous expressions, or omitted fields in the table (such as the omission of repeated corrosion inhibitor names), leading to occasional false positives or false negatives. Overall, the model performs well in text mining tasks, but there is still room for improvement in identifying corrosion inhibitor names.

[0105] Table

[0106]

Claims

1. A large language model-based corrosion inhibitor field literature text mining and data cleaning method, characterized by, The specific steps are as follows: Step one, for the original text of the corrosion inhibitor field in XML format, the existing table is extracted, and the original text is divided into table and text two parts; Step two, use the embedded model Embeddings to preprocess the table and text respectively, and get the respective vector embedding input to the large language model; Step three, for the table and text content, two sets of corresponding system prompt words are manually set and input into the large language model for vector embedding of system prompt words; Step four, the vector embedding of the two sets of system prompt words is respectively searched for similarity with the vector embedding of the corresponding table / text, and the table / text segments with similarity higher than the set threshold are retained as uncleaned corrosion inhibitor data; otherwise, discard; Step five, extract the uncleaned corrosion inhibitor data to get the corresponding json structure data, and uniformly extract them into the excel table of the table / text; Step six, multi-task parallel processing, batch creation batch task, traversal of the original content of several tables / texts in all tasks, and output of the original content after retrieval as json data, the batch task of the table / text of the same original text is set with the same request-id and saved as the same file name, so as to align the table and text; Step seven, clean the aligned table / text, and unify the different expressions of the original data.

2. The method of claim 1, wherein, In step one, the table is extracted as follows: first, convert the XML file to HTML format, and then use the BeautifulSoup tool to extract the table row by row.

3. The method of claim 1, wherein, In step two, for the text, use the LangChain framework for preprocessing, which specifically includes: First, use the RecursiveCharacterTextSplitter class in the langchain_text_splitters module to divide long text in the original text into text segments chunks; Then, use the embedding model Embeddings to convert the text segments chunks into embedding vectors, and convert the original text into numerical values; For the table, use the embedding model Embeddings to convert the table content into embedding vectors, and convert the original table content into numerical values.

4. The method of claim 1, wherein, In step six, the same original text is preprocessed to get embedding vectors input into the large language model, and after internal similarity retrieval with system prompt words, it outputs uncleaned table / text, extracts respective json structure, and converts it into respective excel table; the table and text in the two excel tables are aligned according to the file name and request-id.

5. The method of claim 1, wherein, In step seven, cleaning is to align the numerical information units and unify the professional terms; for regular data-temperature and time, convert Celsius to Kelvin temperature, and use the regularity to assist batch cleaning with python script. For the sentence information containing numerical information, the element composition sentence of steel material, after extracting the content of each element, the api of DeepseekR1 large language model is selected to assist in cleaning the prompt word for cleaning; The number of each cleaning is set according to the complexity of the problem.

Images