An environment chemical pollution key information extraction method and system based on a large language model

Abstract

The application relates to an environment chemical pollution key information extraction method and system based on a large language model, and relates to the technical field of environment pollutant element information extraction. The method collects literature metadata from a data source and constructs a candidate literature set accordingly. A prompt word is designed by combining a specific task with a prompt word engineering, a large language model is called to screen the candidate literature, and the required text set for subsequent information extraction is obtained after preprocessing. A differentiated blocking strategy is designed for text processing, a context window is divided for a support information file, the main text is not divided, and a target text set is obtained by merging. A multi-step information extraction process is constructed for key information extraction, complex tasks are decomposed into subtasks, accurate extraction of different environment pollutant element information is realized, multi-round dialogue verification is introduced to check the extraction result, the large model illusion is effectively slowed down, the reliability of the result is improved, and structured data related to environment chemical pollution can be efficiently extracted from massive literature.

Description

Technical Field

[0001] This application relates to the field of environmental pollutant element information extraction technology, and in particular to a method and system for extracting key information on environmental chemical pollution based on a large language model. Background Technology

[0002] Over the past few decades, the field of environmental science has accumulated a wealth of knowledge, but much of this knowledge exists in the form of unstructured text (such as research papers). The environmental pollutant information contained within these texts (such as pollutant names, sampling times, sampling locations, concentration data, and experimental conditions) is not presented in a structured data format that can be directly calculated and analyzed. However, machine learning-driven environmental modeling research (such as pollutant screening, pollutant concentration prediction, and risk simulation) heavily relies on structured data. Traditional data extraction methods rely on manual preparation or semi-automated processes specific to certain scenarios (such as pattern / rule-based methods and supervised learning methods). Manual preparation is time-consuming and labor-intensive, while semi-automated processes specific to certain scenarios rely on the manual encoding of complex rules and the manual annotation of training corpora. These methods require extensive domain knowledge and complex data processing, making them difficult to adapt to different extraction tasks and thus limiting their versatility and practicality.

[0003] Furthermore, pollution exposure data possesses unique complexity and heterogeneity, primarily manifested in its high dependence on spatiotemporal context and monitoring conditions, as well as the data's inherent dispersion and contextual dependence. For instance, this information is often scattered across different locations in the literature (e.g., introductions, methods, tables, results paragraphs), requiring cross-paragraph and cross-chapter integration to establish complete connections. Its presentation is diverse (e.g., environmental parameters appear both in structured tables and within paragraph text), lacking a unified format standard (e.g., pollutant names may have multiple forms such as generic names, trade names, and abbreviations; concentration information may use different unit systems such as mg / L, μg / m³, and ppm). Clearly, this inherent unstructured characteristic poses a significant challenge to traditional data extraction methods.

[0004] Therefore, how to efficiently extract structured data from massive amounts of literature has become an important problem that urgently needs to be solved in the environmental field. Summary of the Invention

[0005] This application provides a method and system for extracting key information on environmental chemical pollution based on a large language model. By utilizing the powerful workflow of the large language model (LLM), it can automatically mine key information on environmental chemical pollution such as pollutant name, sampling site, sampling time, environmental concentration and analysis method, and realize the simple and efficient extraction of environmental pollutant element information from massive literature.

[0006] Firstly, this application provides a method for extracting key information on environmental chemical pollution based on a large language model, including:

[0007] Based on the keywords corresponding to the target pollutant and the target environmental element, metadata of relevant literature is obtained from the data source, and a candidate literature set corresponding to the metadata is obtained. The target pollutant is a specific pollutant related to environmental chemical pollution.

[0008] Based on the prompt word engineering, using the aforementioned metadata as a benchmark, a large language model is used to perform relevance screening on the candidate document set to obtain the target document set;

[0009] The target document set is transformed and parsed using a parser, and preprocessed using regular expression matching and word segmentation tools to obtain the required text set;

[0010] A differentiated block-based strategy is designed for text processing. The supporting information files in the required text set are segmented using context windows, while the main text is not segmented. The resulting text sets are then merged to obtain the target text set.

[0011] A multi-step information extraction process is constructed, which decomposes the complex task of extracting environmental pollutant element information into sub-tasks, and introduces a multi-round dialogue verification mechanism to verify the intermediate extraction results. Environmental pollutant element information is extracted from the target text set to obtain a structured dataset.

[0012] Secondly, this application provides a system for extracting key information on environmental chemical pollution based on a large language model, including:

[0013] The candidate literature collection module is used to obtain metadata of relevant literature from the data source based on the keywords corresponding to the target pollutant and the target environmental element, and to obtain the candidate literature set corresponding to the metadata. The target pollutant is a specific pollutant related to environmental chemical pollution.

[0014] The target document filtering module is used to perform relevance filtering on the candidate document set based on the prompt word engineering and the metadata, and to obtain the target document set by using a large language model.

[0015] The text preprocessing module is used to convert and parse the target document set using a parser, and to preprocess it using regular expression matching and word segmentation tools to obtain the required text set.

[0016] The differentiated text processing module is used to design differentiated block-based strategies for text processing. It performs context window segmentation on the supporting information files in the required text set, does not segment the main text, and merges them to obtain the target text set.

[0017] The information extraction module is used to construct a multi-step information extraction process, which breaks down the complex task of extracting environmental pollutant element information into sub-tasks and introduces a multi-round dialogue verification mechanism to verify the intermediate extraction results. It extracts environmental pollutant element information from the target text set to obtain a structured dataset.

[0018] In summary, this application's embodiments collect metadata related to environmental chemical pollution from data sources based on keywords, thereby obtaining a candidate literature set. Subsequently, prompting words are designed using prompting word engineering, tailored to the specific task. Based on the designed prompting words and metadata, a large language model is invoked to screen the candidate literature set for relevance, resulting in a target literature set. This target literature set undergoes parsing, transformation, and cleaning preprocessing to obtain the desired text set. Then, a differential segmentation strategy is designed for text processing. For different text types within the desired text set, the main body of the paper retains its complete context without segmentation; text outside the main body (such as appendices and supplementary materials) is segmented using context windows. The designed prompting words are used to invoke the large language model to extract target text containing key information about environmental chemical pollution, forming the target text set. Finally, in the key information extraction stage, a multi-step information extraction process is constructed, breaking down the complex task of extracting environmental pollutant element information into several sub-tasks. The designed prompting words are used to sequentially invoke the large language model to complete each sub-task, extracting structured data from the target text set. For the extraction results output by the model, multi-round dialogue verification is introduced to check the extraction results, effectively mitigating the illusion of a large model and improving the reliability of the results. As can be seen, this application has achieved efficient extraction of structured data related to environmental chemical pollution from massive amounts of literature. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a method for extracting key information on environmental chemical pollution based on a large language model, as provided in this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] To facilitate understanding of the embodiments of this application, further explanations and descriptions will be provided below in conjunction with the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of this application.

[0024] This application provides a method for extracting key information on environmental chemical pollution based on a large language model, which may specifically include the following steps:

[0025] Step 110: Based on the keywords corresponding to the target pollutant and the target environmental element, obtain the metadata of relevant documents from the data source, and obtain the candidate document set corresponding to the metadata.

[0026] The target pollutant is a specific pollutant related to environmental chemical pollution.

[0027] This embodiment can use peer-reviewed literature databases such as Web of Science, Scopus, and Google Scholar as data sources, and retrieve metadata of relevant literature from the data sources using Boolean logic keywords. The keywords can be determined based on target pollutant information and target environmental medium information; the metadata may include, but is not limited to: literature title, abstract, author information, DOI, keywords, etc.

[0028] For example, in document retrieval, a set of logical operators (such as "and", "or", and "not") can be used to combine scattered keywords into complex logical search expressions. Since the search syntax varies across different databases, keyword expressions can be debugged independently for each database.

[0029] Subsequently, based on this metadata, combined with Python scripts and manual verification, the literature records from multiple databases were automatically merged and deduplicated, and PDF documents of the full text of the corresponding papers were obtained in batches. If the paper had corresponding supporting information (SI) files, they were also downloaded together to form a candidate literature set.

[0030] Step 120: Based on the prompt word engineering, using the metadata as a benchmark, the candidate document set is relevance-based and the target document set is obtained by using a large language model.

[0031] Among related technologies, the emergence of Large Language Models (LLMs) has revolutionized the way humans interact with text, bringing new opportunities to the field of data extraction. LLMs can understand and generate text, making them ideal for processing unstructured information and providing a powerful tool for handling unstructured and diverse document information in the field of environmental science.

[0032] Compared to traditional manual data processing and semi-automated processes for specific scenarios, this application improves the data mining process by proposing a simple and efficient automatic mining method for key environmental chemical pollution information. By combining LLM and Python, it achieves efficient extraction of environmental pollutant element information (also known as environmental chemical pollutant element information) from research papers. Specifically, this embodiment utilizes the powerful workflow of LLM such as DeepSeek to automatically mine key environmental chemical pollution information such as pollutant name, sampling site, sampling time, environmental concentration, and analysis method. (Refer to...) Figure 1 As shown, the specific implementation process of this embodiment includes candidate literature collection, target literature screening, text preprocessing, context window processing, information extraction, prompt word engineering, and effect evaluation. Taking DeepSeek as an example, its application programming interface (API) can be configured, and key parameters (such as temperature=0) can be set.

[0033] Considering that the initial candidate documents obtained through keyword screening are often quite mixed and may include documents unrelated to the target information, this embodiment constructs a prompt word engineering to more accurately extract environmental pollutant element information (i.e., target information). This prompt word engineering is applied to target document screening, as well as subsequent context window processing and information extraction.

[0034] Based on prompt word engineering, prompt words can be designed for different tasks (such as literature screening, differentiated segmentation, and information extraction). In literature relevance screening, screening prompt words can be designed, and LLM can be called to perform secondary filtering on the initial screening results to accurately locate relevant articles containing target information and form a target literature set.

[0035] In one optional embodiment, based on the prompt word engineering and using the metadata as a benchmark, a large language model is used to screen the candidate document set for relevance to obtain a target document set. This includes: designing a prompt word engineering project; designing screening prompt words based on the prompt word engineering and document screening criteria; independently calling the large language model at least three times for each document in the candidate document set based on the screening prompt words; analyzing the metadata corresponding to each document through the large language model in each call, judging the relevance of the abstracts of each document, and obtaining the classification label results corresponding to each document; summarizing the voting data corresponding to each document in the classification label results, and determining the final classification result based on the voting data; and screening target documents that meet the requirements based on the final classification result to form a target document set. The large language model is set according to preset parameters; when calling the large language model using different types of prompt words, the identity and task of the large language model are indicated based on the different types of prompt words, constraints and output format requirements are added, and check conditions and sample examples are introduced to guide the large language model to map and output structured results under the prompt word engineering.

[0036] To extract target information more accurately, this embodiment designs prompt words for LLM to screen literature. These prompt words can be designed based on prompt word engineering and literature screening criteria.

[0037] Cue word engineering: Cue word engineering has become a key technology for improving the practicality and accuracy of LLMs. A well-structured, detailed, and proactively error-detecting cue word can effectively mitigate the illusion of ineffectiveness in LLMs. Choosing Markdown to organize the cue information first requires defining the LLM's identity and task, then adding constraints and output format requirements, and finally adding check conditions and providing few-sample examples to allow the LLM to learn the mapping and formatted output, ensuring the model can accurately understand and execute the task.

[0038] By using prompt word engineering and screening prompt words, the identity of the LLM (e.g., environmental science literature screening expert) is clearly defined, as is the task of the LLM (e.g., determining whether a literature abstract falls within the target literature scope containing information on environmental pollution elements). Constraints (e.g., clearly defining inclusion and exclusion criteria with "retention scope" and "exclusion scope") and output format requirements (e.g., outputting only a single conclusion 'yes' or 'no') are added. Finally, by adding check conditions to the prompt words and providing a small number of sample examples, a secondary filtering of the initial screening results is performed to accurately identify relevant articles.

[0039] Specifically, the acquired metadata is converted to JSON format. Python code then calls the Deepseek API to assess the relevance of each document's abstract based on filtering prompts, retaining only documents containing the target information. The filtering prompts explicitly define the document selection criteria, requiring the LLM model to classify each paper as "yes" or "no" according to these criteria. Only papers marked "yes" are selected for the next step. In this step, a majority voting method is used to mitigate the instability of the model's output. This process is repeated three times, yielding three responses. The votes are then tallied. If two or more of the three responses are identical, the document selection is confirmed (e.g., if the three outputs are "yes," "no," and "no" (i.e., 2 no vs 1 yes), the result is "no"). Throughout the three runs, the settings remain the same: the temperature parameter is set to 0, and all other parameters retain their default values ​​to ensure output stability while mitigating model illusions. Similarly, the model parameters are kept at the same settings for subsequent block processing and information extraction.

[0040] Therefore, this embodiment uses the majority voting method to mitigate the instability of the model output and improve the reliability of the results.

[0041] Step 130: The target document set is converted and parsed using a parser, and preprocessed using regular expression matching and word segmentation tools to obtain the required text set.

[0042] The acquired documents are in PDF format. To facilitate the processing of PDF documents by a large language model, a commercial parser (such as Textin) can be used to parse the PDF documents and convert them into lightweight text, such as Markdown text, to obtain the text corresponding to each document. Then, considering the possible unnecessary information in the text, regular expression matching and word segmentation tools are introduced to further process the converted Markdown text for easier subsequent processing, retaining valid data to obtain the required text set.

[0043] In one optional embodiment, a parser is used to transform and parse the target document set, and preprocessing is performed using regular expression matching and word segmentation tools to obtain the desired text set. This includes: using a parser to parse each target document in the target document set and transforming it to obtain lightweight text corresponding to each target document; establishing cleaning rules and cleaning each of the lightweight texts based on regular expression matching to obtain cleaned text; and using word segmentation tools to remove non-target information from the cleaned text to obtain the desired text, thus forming the desired text set.

[0044] The parser can extract text blocks from each converted text in the order of reading, preserving the integrity of the document to the greatest extent. Then, regular expression matching is used to clean the text. Before text cleaning, cleaning rules can be pre-built for non-text elements, including removing HTML tags, headers, footers, special characters, and formatting control characters.

[0045] Finally, the nltk word segmentation tool was used to remove non-textual parts such as acknowledgments, references, and supplementary materials from the cleaned text system, which effectively reduced data redundancy and saved the cost of subsequent processing.

[0046] Step 140: Design a differentiated block segmentation strategy for text processing. Perform context window segmentation on the supporting information files in the required text set, but do not segment the main text. Then merge the results to obtain the target text set.

[0047] Since the context window of LLM has been significantly increased, and the contextual integrity of the main text of the literature is crucial for accurately understanding the source, measurement conditions and spatiotemporal background of environmental pollutant element information, the main text of the literature is not segmented and directly proceeds to the next stage of information extraction.

[0048] However, for supporting information documents commonly found in the field of environmental science, where environmental pollutant element information is stored in the form of tables or unstructured paragraphs, the text is usually long and loosely structured. Therefore, a special chunking strategy is used for processing. A differentiated chunking strategy is adopted, selecting supporting information documents from the required text set for chunking. By segmenting through a context window, a set of supporting information fragments corresponding to the supporting information documents is obtained. These fragments are then combined with the main text that does not require chunking to form the target text set.

[0049] It should be noted that, in the field of environmental science, a large amount of detailed data (such as sampling time, sampling point coordinates, concentration information, etc.) is often stored in the supporting information in the form of tables or unstructured paragraphs. It is necessary to process it into blocks, retaining only the content related to the environmental pollutant element information, so as to make the text more compact.

[0050] The text processing involves designing a differentiated segmentation strategy, segmenting the supporting information files in the required text set using a context window, without segmenting the main text, and merging them to obtain the target text set. This includes: constructing a differentiated processing strategy based on text type; selecting supporting information files from the required text set based on the differentiated processing strategy; initializing a Token window; segmenting the supporting information files using the Token window, defining dynamic semantic boundaries during segmentation, and adjusting the segmentation point to the end of the nearest complete semantic unit to obtain a segmented text set; designing content extraction prompts based on the prompt word project combined with the original fragment extraction task of environmental pollutant element information; analyzing each text in the segmented text set using the large language model based on the content extraction prompts, identifying and outputting text fragments related to environmental pollutant element information; sorting and concatenating the text fragments according to their context order in the original document's supporting information to obtain a processed set of supporting information fragments; and merging the set of supporting information fragments with the complete main text set in the required text set to form the target text set.

[0051] First, initialize a Token block window ("Token" (word, prefix / suffix or single character) is the basic processing unit of LLM), the size of which is the basic block threshold, but the threshold is not strictly fixed when block is divided, as long as the total number of tokens in each block does not exceed the preset upper limit.

[0052] During the segmentation process, it is necessary to dynamically locate semantic boundaries (such as the end of a paragraph, the end of a table, or before a chapter title) and adjust the cutting point to the end of the nearest complete semantic unit to ensure that the last semantic unit in the window is not cut off.

[0053] Subsequently, following the prompt word engineering approach, and combining the task of extracting original fragments of environmental pollutant element information, content extraction prompt words were designed. First, these prompt words identified the LLM's role (e.g., environmental science data extraction expert) and clarified its task (e.g., extracting original fragments of environmental pollutant element information from supporting text blocks, including pollutant name, sampling time, location, environmental medium, detection method, and concentration data). Then, constraints were added (e.g., requiring the retention of complete sentences or relevant columns from tables containing the above information, excluding irrelevant explanations), while specifying output format requirements (e.g., outputting only original text fragments, arranged in the original text order; outputting blanks if no relevant information is available). Finally, check conditions were added (e.g., confirming the output is an original text excerpt, concentration units are complete, table structure is intact, and there are no additional explanations), and a few sample examples were provided (e.g., providing several examples in the prompt words to show the LLM how to output in paragraph extraction, table extraction, and when no relevant information is available). Based on the content extraction prompt words, the LLM's API was called to extract the main information from each text block, requiring the LLM to retain the original text related to environmental pollutant element information, output strictly in the original text order, and without adding any explanations or additional content.

[0054] Step 150: Construct a multi-step information extraction process, break down the complex task of extracting environmental pollutant element information into sub-tasks, and introduce a multi-round dialogue verification mechanism to verify the intermediate extraction results. Extract environmental pollutant element information from the target text set to obtain a structured dataset.

[0055] The target environmental pollutant element information (such as pollutant name, sampling time, sampling location, concentration data, experimental conditions, etc.) is not only structurally complex but also interrelated. Existing technologies typically employ a single-step extraction method in information extraction; however, this method leads to overly complex tasks and affects extraction results. To address this technical problem, this embodiment designs a multi-step information extraction process. By breaking down the complex task into multiple steps, the accuracy of the large language model in extracting target environmental pollutant element information is improved. Furthermore, to enhance the reliability of model decision-making, a multi-turn dialogue verification mechanism is introduced to guide the LLM to rethink the output results, thereby enabling the model to accurately extract environmental pollutant element information from the text and form the structured dataset required for analysis.

[0056] In one optional embodiment, a multi-step information extraction process is constructed, decomposing the complex task of extracting environmental pollutant element information into sub-tasks, and introducing a multi-turn dialogue verification mechanism to verify intermediate extraction results. Environmental pollutant element information is extracted from the target text set to obtain a structured dataset, including: analyzing the complex structure and correlation between environmental pollutant element information; designing a multi-step information extraction process; decomposing the complex task of extracting environmental pollutant element information using the multi-step information extraction process to obtain a set of sub-tasks; executing each sub-task in the sub-task set; designing compound prompts based on the prompt word engineering and information extraction task, wherein the compound prompts describe the information of the target compound and the formatted output requirements; and based on the compound... The system uses prompts to invoke the large language model to extract target compounds from the target text set, creating a target substance list as an intermediate extraction result. A multi-turn dialogue verification mechanism prompts the large language model to re-evaluate the intermediate extraction results, resulting in an accurate target substance list. Compound names from this list are used as indexes to design element prompts. Based on these prompts, the system uses the large language model to extract environmental pollutant element information from the target text set as an intermediate element extraction result. Again, the multi-turn dialogue verification mechanism prompts the large language model to re-evaluate the intermediate extraction results, resulting in an accurate structured dataset.

[0057] This study analyzes the complex structure and relationships among environmental pollutant elements to clarify the necessary processes and tasks for extracting key data from massive datasets, and designs a multi-step information extraction process accordingly. Then, the complex task is broken down into several sub-tasks.

[0058] Next, execute each subtask:

[0059] First, based on the compound names, and considering the cue word engineering and compound information extraction task, cue words for extracting compound names are designed as compound cue words. (Similar to the aforementioned cue words, the cue words for extracting compound names describe the target compound information and formatted output requirements in detail, such as: the compound cue words indicate the LLM's identity and clarify the LLM's task; constraints and output format requirements are added, and check conditions are added, providing a small sample example). Based on the compound cue words, the LLM's API is called to extract the corresponding target compounds from the text and output them in JSON format, creating a target substance list as the intermediate extraction results of the compounds.

[0060] After obtaining the target substance list, based on cue word engineering, the compound names in the target substance list are used as indexes to design cue words for extracting all environmental pollutant element information, which serve as element cue words. (Similarly, the element cue words clearly define the definition of each element and the requirements for JSON format output, such as indicating the identity and task of the LLM; adding constraints and specifying output format requirements; and adding check conditions and providing a small sample example). Based on the element cue words, the LLM API is called to extract environmental pollutant element information from the text and output in JSON format for subsequent work using structured datasets.

[0061] To enhance the reliability of the output results, a multi-round dialogue verification mechanism is embedded in both the compound extraction and feature extraction processes of LLM. After the first round of dialogue, the input text, the initial question, and the first answer from LLM are used as new context to prompt LLM to rethink, verify, or adjust its initial decision, thereby improving the reliability of the decision and ultimately obtaining an accurate list of target substances and a structured dataset.

[0062] Therefore, this implementation breaks down a complex task into several sub-tasks through a multi-step information extraction process, enabling the accurate extraction of information on multiple environmental pollutant elements. Based on this multi-step information extraction, a multi-round dialogue verification mechanism is introduced to check the data extraction results through multiple rounds of dialogue verification, effectively mitigating the illusion of a large model and improving the reliability of the results.

[0063] After the LLM model completes the result parsing, it outputs structured data. It can receive the JSON format response returned by LLMAPI in a specified storage / display location, parse and extract the required structured data (such as pollutant name, concentration value, sampling time, location coordinates and analysis instrument, etc.). This structured data can be used to form a structured dataset, which can be applied to industries such as those related to environmental chemical pollution.

[0064] This application enables automated data mining execution. For example, it utilizes a written Python program to integrate and execute the aforementioned steps, such as:

[0065] (1) File import: Read the obtained target document metadata and Markdown text.

[0066] (2) Loading prompt words: Load the specific task prompt words designed in the process, build the prompt word project, and apply the prompt word project to each step to realize the automatic output of prompt words.

[0067] (3) API call and interaction: The configured LLM API is called programmatically, and the literature metadata, text and prompt words are used as input to execute a multi-round conversational information extraction process.

[0068] (4) Result parsing and structured output: Receive the JSON format response returned by the LLM API, parse and extract the required structured data (such as pollutant name, concentration value, sampling time, location coordinates and analysis instrument, etc.).

[0069] Furthermore, based on the above embodiments, this embodiment also introduces an effect evaluation process to evaluate the screening and extraction effect of the LLM model, as follows:

[0070] The accuracy of the text mining process is evaluated by calculating precision and recall. A validation set and a test set are constructed based on Cochrane's theorem. The validation set is used to debug the method, and the test set is used for performance evaluation. Both the validation and test sets require manual annotation. Performance evaluation is mainly used in the relevant literature screening and information extraction stages. The calculation of precision and recall rate is as follows: ; .

[0071] The criteria for evaluating the effectiveness are as follows:

[0072] During the relevant literature screening stage, it can be verified whether the literature has been correctly classified as relevant or irrelevant. The specific criteria are shown in Table 1 below: ; Table 1

[0073] During the information extraction stage, it can be verified whether the environmental pollutant element information in the literature has been extracted correctly. Each environmental pollutant element information is judged separately, and the specific judgment criteria are shown in Table 2 below: ; Table 2

[0074] As can be seen, this application embodiment collects metadata from the data source to obtain the corresponding candidate document set. Using prompt word engineering combined with specific task-specific prompt words, a large language model is invoked to filter candidate documents, and the filtered documents are preprocessed to obtain the required text set. A differentiated segmentation strategy is designed to segment the supporting information files in the required text set using context windows, extracting target text containing key information about environmental chemical pollution. This target text set is then combined with the unsegmented main text. Finally, a multi-step information extraction process is constructed for extracting key information, breaking down the complex task into several sub-tasks to achieve accurate extraction of information on different environmental pollutant elements. Multi-round dialogue verification is introduced to validate intermediate extraction results, effectively mitigating the illusion of a large model and improving the reliability of the results. This application embodiment has the following beneficial effects:

[0075] 1. Significantly improved efficiency: Compared with manual extraction, this invention greatly improves data extraction efficiency, shortening a task that originally required months to complete to a few days.

[0076] 2. User-friendly and highly generalizable: Compared with semi-automated methods for specific scenarios, this invention is user-friendly for environmental researchers, requires no professional programming background, and can be applied to different tasks simply by modifying the prompt words, making it highly generalizable.

[0077] 3. Low technical threshold: Unlike fine-tuning models that require a large amount of labeled data and specialized training, this invention directly calls the API of mature LLM and achieves high-precision extraction only through prompt word engineering. There is no need to train or adjust the model, which greatly reduces the technical threshold.

[0078] 4. High reliability of results: The multi-turn dialogue consistency check mechanism introduced in this invention can effectively alleviate the common illusion and context error problems in LLM, and significantly improve the reliability of extraction results.

[0079] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should know that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps may be performed in other orders or simultaneously.

[0080] This application also provides a system for extracting key information on environmental chemical pollution based on a large language model, including:

[0081] The candidate literature collection module is used to obtain metadata of relevant literature from the data source based on the keywords corresponding to the target pollutant and the target environmental element, and to obtain the candidate literature set corresponding to the metadata. The target pollutant is a specific pollutant related to environmental chemical pollution.

[0082] The target document filtering module is used to perform relevance filtering on the candidate document set based on the prompt word engineering and the metadata, and to obtain the target document set by using a large language model.

[0083] The text preprocessing module is used to convert and parse the target document set using a parser, and to preprocess it using regular expression matching and word segmentation tools to obtain the required text set.

[0084] The differentiated text processing module is used to design differentiated block-based strategies for text processing. It performs context window segmentation on the supporting information files in the required text set, does not segment the main text, and merges them to obtain the target text set.

[0085] The information extraction module is used to construct a multi-step information extraction process, which breaks down the complex task of extracting environmental pollutant element information into sub-tasks and introduces a multi-round dialogue verification mechanism to verify the intermediate extraction results. It extracts environmental pollutant element information from the target text set to obtain a structured dataset.

[0086] It should be noted that the environmental chemical pollution key information extraction system based on a large language model provided in this application can execute the environmental chemical pollution key information extraction method based on a large language model provided in any embodiment of this application, and has the corresponding functions and beneficial effects of the execution method.

[0087] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0088] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for extracting key information on environmental chemical pollution based on a large language model, characterized in that, include: Based on the keywords corresponding to the target pollutant and the target environmental element, metadata of relevant literature is obtained from the data source, and a candidate literature set corresponding to the metadata is obtained. The target pollutant is a specific pollutant related to environmental chemical pollution. Based on the prompt word engineering, using the aforementioned metadata as a benchmark, a large language model is used to perform relevance screening on the candidate document set to obtain the target document set; The target document set is transformed and parsed using a parser, and preprocessed using regular expression matching and word segmentation tools to obtain the required text set; A differentiated block-based strategy is designed for text processing. The supporting information files in the required text set are segmented using context windows, while the main text is not segmented. The resulting text sets are then merged to obtain the target text set. A multi-step information extraction process is constructed, which decomposes the complex task of extracting environmental pollutant element information into sub-tasks, and introduces a multi-round dialogue verification mechanism to verify the intermediate extraction results. Environmental pollutant element information is extracted from the target text set to obtain a structured dataset.

2. The method according to claim 1, characterized in that, Based on the prompt word engineering and using the aforementioned metadata as a benchmark, a large language model is used to perform relevance screening on the candidate document set to obtain the target document set, including: Design prompt word project, and design screening prompt words based on the prompt word project and literature screening criteria; Based on the filtering prompts, for each document in the candidate document set, the large language model is independently invoked at least three times; In each call, the metadata corresponding to each document is analyzed through the large language model, the relevance of the abstract of each document is judged, and the classification label results corresponding to each document are obtained. Summarize the voting data corresponding to each document in the classification labeling results, and determine the final classification result based on the voting data; Based on the final classification results, target documents that meet the requirements are selected to form a target document set; Specifically, the large language model is set according to preset parameters; when the large language model is invoked using different types of prompt words, the identity and task of the large language model are indicated based on the different types of prompt words, constraints and output format requirements are added, and inspection conditions and sample examples are introduced to guide the large language model to map and output structured results under the prompt word project.

3. The method according to claim 1, characterized in that, The target document set is transformed and parsed using a parser, and preprocessed using regular expression matching and word segmentation tools to obtain the desired text set, including: A parser is used to parse each target document in the target document set and convert it into a lightweight text corresponding to each target document. Establish cleaning rules and clean the lightweight texts based on regular expression matching to obtain cleaned texts. The non-target information is removed from the cleaned text using a word segmentation tool to obtain the desired text and form the desired text set.

4. The method according to claim 1, characterized in that, A differentiated chunking strategy is designed for text processing. The supporting information files in the required text set are segmented using context windows, while the main text is not segmented. The resulting merged text sets are the target text set, including: Construct a text type-based differential processing strategy; Based on the aforementioned differentiated processing strategy, supporting information files are selected from the required text set; Initialize the Token window; The supporting information file is segmented using the Token window, and during the segmentation process, dynamic semantic boundaries are defined, and the cutting point is adjusted to the end of the nearest complete semantic unit to obtain a segmented text set. Based on the original fragment extraction task combining the aforementioned prompt word engineering with environmental pollutant element information, content extraction prompt words are designed; Based on the content, prompt words are extracted, and the large language model is called to analyze each text in the segmented text set, identify and output text fragments related to environmental pollutant element information; Based on the contextual order of the text fragments in the supporting information of the original document, they are sorted and concatenated to obtain a processed set of supporting information fragments. The set of supporting information fragments is combined with the complete text set in the required text set to form the target text set.

5. The method according to claim 1, characterized in that, A multi-step information extraction process is constructed, breaking down the complex task of extracting environmental pollutant element information into sub-tasks. A multi-round dialogue verification mechanism is introduced to verify the intermediate extraction results. Environmental pollutant element information is extracted from the target text set to obtain a structured dataset, including: Analyze the complex structure and correlations among environmental pollutant element information, and design a multi-step information extraction process; A multi-step information extraction process is adopted to break down the complex task of extracting environmental pollutant element information, resulting in a set of subtasks; Each subtask in the execution subtask set is performed. Based on the prompt word engineering and information extraction task, compound prompt words are designed. The compound prompt words describe the information of the target compound and the formatted output requirements. Based on the compound prompt words, the large language model is invoked to extract target compounds from the target text set and a target substance list is established as an intermediate extraction result of the compounds; Through a multi-round dialogue verification mechanism, the large language model is prompted to think a second time to verify the intermediate extraction results of the compound and obtain an accurate list of target substances. Use the names of compounds in the target substance list as indexes to design element prompts; Based on the element prompt words, the large language model is invoked to extract environmental pollutant element information from the target text set as intermediate extraction results of the elements; Through the multi-turn dialogue verification mechanism, the large language model is prompted to perform secondary thinking, verify the intermediate extraction results of elements, and obtain an accurate structured dataset.

6. A system for extracting key information on environmental chemical pollution based on a large language model, characterized in that, include: The candidate literature collection module is used to obtain metadata of relevant literature from the data source based on the keywords corresponding to the target pollutant and the target environmental element, and to obtain the candidate literature set corresponding to the metadata. The target pollutant is a specific pollutant related to environmental chemical pollution. The target document filtering module is used to perform relevance filtering on the candidate document set based on the prompt word engineering and the metadata, and to obtain the target document set by using a large language model. The text preprocessing module is used to convert and parse the target document set using a parser, and to preprocess it using regular expression matching and word segmentation tools to obtain the required text set. The differentiated text processing module is used to design differentiated block-based strategies for text processing. It performs context window segmentation on the supporting information files in the required text set, does not segment the main text, and merges them to obtain the target text set. The information extraction module is used to construct a multi-step information extraction process, which breaks down the complex task of extracting environmental pollutant element information into sub-tasks and introduces a multi-round dialogue verification mechanism to verify the intermediate extraction results. It extracts environmental pollutant element information from the target text set to obtain a structured dataset.

Images