Large model-based environment document processing method and device, equipment and storage medium

Abstract

This application provides a method, apparatus, device, and storage medium for environmental document processing based on a large model, relating to the field of environmental data processing technology. The method includes: acquiring multi-source environmental event document data; preprocessing the multi-source environmental event document data; based on the task scenario, using a large language model to extract features from the preprocessed text data to obtain text modal features, and using the YOLO object detection algorithm to extract image modal features from the preprocessed image data; adaptively fusing the text modal features and image modal features using a dynamic weighting method to obtain a cross-modal fusion vector; constructing a knowledge graph based on the cross-modal fusion vector, and generating a multi-modal summary. This application can generate concise and logically clear summaries for different types of documents, enabling users to quickly obtain the core information of the document and meeting the needs for rapid understanding in multiple scenarios.

Description

Technical Field

[0001] This application relates to the field of environmental data processing technology, and in particular to an environmental document processing method, apparatus, device and storage medium based on a large model. Background Technology

[0002] Existing methods for processing environmental documents face several technical challenges. First, the diverse content and complex structure of documents make manual information extraction prone to omissions or errors, leading to difficulties in information retrieval. Second, traditional methods often require manual review and editing, which is time-consuming and labor-intensive, and fails to meet the needs of large-scale data processing, resulting in low processing efficiency. Furthermore, existing methods struggle to correlate document content from different sources, failing to effectively establish cross-document information connections, which affects the accuracy and timeliness of the overall analysis.

[0003] Currently, most contextual document processing relies on basic Natural Language Processing (NLP) techniques, such as keyword extraction and named entity recognition. However, traditional NLP models cannot handle large amounts of complex contextual information, limiting their application effectiveness. In recent years, large-scale models have made breakthroughs in text semantic understanding, better capturing deep semantic information and demonstrating powerful performance in question answering, text generation, and information extraction. Nevertheless, the application of large-scale models in contextual document processing is still in its early stages, and a systematic solution to address the complex needs of the contextual domain has not yet been developed. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for processing environment documents based on a large model, so as to extract effective information from environment documents and process them quickly and accurately.

[0005] Firstly, this application provides a method for processing environment documents based on a large model, including:

[0006] Acquire multi-source environmental event document data, which includes text data and image data;

[0007] The preprocessing of the multi-source environmental event document data includes: segmenting the text data using semantic paragraph slicing and dynamic semantic slicing strategies to obtain preprocessed text data; and detecting the image data using an object detection algorithm to obtain preprocessed image data.

[0008] Based on the task scenario, a large language model is used to extract features from the preprocessed text data to obtain text modal features, and the YOLO object detection algorithm is used to extract image modal features from the preprocessed image data.

[0009] The text modal features and image modal features are adaptively fused using a dynamic weighting method to obtain a cross-modal fusion vector;

[0010] A knowledge graph is constructed based on the cross-modal fusion vector, and a multimodal summary is generated;

[0011] The text modal features and image modal features are adaptively fused using a dynamic weighting method to obtain a cross-modal fusion vector, including:

[0012] Multiple modal features are concatenated to train a global machine learning model and obtain global baseline weights; wherein, the modal features include text modal features and image modal features;

[0013] Multiple modal features are concatenated, and similar event samples of the current event sample are found based on K-nearest neighbors; a machine learning model is trained based on each modal feature, and performance indicators are obtained; the performance indicators are normalized, and the normalized performance indicators are weighted and averaged with the global benchmark weights and then normalized to obtain the modal weights of the current event sample; all event samples are traversed and the weight label matrix is ​​output.

[0014] Multimodal features are concatenated and combined with a weight label matrix to establish a weighted prediction regression model, which outputs the weight of each modality.

[0015] Based on each modality weight, text modality features and image modality features are fused to obtain a cross-modality fusion vector.

[0016] In one possible design, text data is segmented using semantic paragraph slicing and dynamic semantic slicing strategies. Methods for obtaining preprocessed text data include:

[0017] Text data is divided based on semantic units, which include title-body pairs or list items;

[0018] Based on semantic unit segmentation, a window sliding and overlapping strategy is adopted to ensure contextual coherence;

[0019] Based on the RoBERTa semantic model, the semantic similarity between adjacent segments is determined, and semantically closely related paragraphs are automatically merged and logical boundaries are marked to obtain preprocessed text data.

[0020] In one possible design, the text modal features include keyword density and severity score; the image modal features include detection results, spatial features, and model detection confidence.

[0021] In one possible design, based on each modal weight, the text modal features and image modal features are fused to obtain the cross-modal fusion vector. The calculation formula is as follows:

[0022] ;

[0023] in For event samples i In modality j The weight value, For modal number, For event samples i In modality j eigenvectors, Let i be the cross-modal fusion vector of event sample i.

[0024] In one possible design, a knowledge graph is constructed based on the cross-modal fusion vector, and a multimodal summary is generated, including:

[0025] First, category labels are constructed to generate entity nodes. Cross-modal fusion vectors are used as entity attributes. Based on expert experience and business processes, causal and impact relationships, composition and attribution relationships between events are generated. At the same time, the identified entities are aligned and linked with external standard knowledge bases to complete the construction of the knowledge graph. The category labels include pollutant categories, pollution sources and equipment types.

[0026] Based on the constructed knowledge graph, the system determines the type of input document by document structure tags, title keywords, and semantic vectors, automatically matches the corresponding summary template and Prompt strategy, extracts the charts and graphs in the original text using the structure parsing module, identifies the chart titles, units, and data fields, normalizes the structured data, converts the structured table data into text descriptions, and calls the large model to generate a set interpretation. The system then encapsulates the summary content, image links, graph data, and corresponding paragraph citations to generate a multimodal summary.

[0027] In one possible design, after generating the multimodal summary, the method further includes:

[0028] Obtain user feedback data, which includes star ratings, annotations of missing key information, free text suggestions, and error correction annotations for summary paragraphs;

[0029] The user feedback data is formatted into training sample pairs, which include original document fragments and their corresponding correct summaries.

[0030] The training samples were used to fine-tune the summary generation model, and the summary generation model was then reinforced through training.

[0031] Set a confidence threshold. For any part of the generated digest with a confidence level lower than the threshold, send it to the user for verification and confirmation. After receiving confirmation, save the digest or use it for downstream tasks.

[0032] Secondly, this application provides an environment document processing device based on a large model, the device comprising:

[0033] The data acquisition module is configured to acquire multi-source environmental event document data, which includes text data and image data.

[0034] The data preprocessing module is configured to preprocess the multi-source environmental event document data, including: segmenting the text data using semantic paragraph slicing and dynamic semantic slicing strategies to obtain preprocessed text data; and detecting the image data using an object detection algorithm to obtain preprocessed image data.

[0035] The feature extraction module is configured to extract features from the preprocessed text data based on the task scenario using a large language model to obtain text modal features, and to extract image modal features from the preprocessed image data using the YOLO object detection algorithm.

[0036] The feature fusion module is configured to perform multimodal feature adaptive fusion of the text modal features and image modal features using a dynamic weighting method to obtain a cross-modal fusion vector;

[0037] The knowledge graph summarization generation module is configured to construct a knowledge graph based on the cross-modal fusion vector and generate a multimodal summary.

[0038] Thirdly, embodiments of this application provide an electronic device, including: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the large-model-based environment document processing method as described in the first aspect and various possible designs of the first aspect.

[0039] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the environment document processing method based on a large model as described in the first aspect and various possible designs of the first aspect.

[0040] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the environment document processing method based on a large model as described in the first aspect and various possible designs of the first aspect.

[0041] The environment document processing method, apparatus, device, and storage medium based on a large model provided in this application have at least the following beneficial effects:

[0042] This application collects multi-source environmental event document data and performs intelligent preprocessing on the multimodal data. Text data employs semantic paragraph slicing and dynamic semantic slicing strategies, while image data utilizes object detection algorithms. Key features of the multimodal data are intelligently extracted based on the task scenario. A dynamic weighting method is used for adaptive fusion of multimodal features to achieve information complementarity. During knowledge graph construction, alignment and linking with external knowledge bases are adopted to enhance the knowledge content of the graph. Intelligent summary generation leverages the language generation capabilities of a large model to generate concise and logically clear summaries for different types of documents, enabling users to quickly obtain the core information of documents and meeting the needs for rapid understanding in various scenarios. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] Figure 1 A flowchart illustrating an environment document processing method based on a large model, provided in an embodiment of this application;

[0045] Figure 2 A flowchart illustrating the cross-modal fusion vector acquisition process provided in this application embodiment;

[0046] Figure 3 A flowchart illustrating the establishment of the weighted label matrix provided in this application embodiment;

[0047] Figure 4 A flowchart illustrating the process of constructing a knowledge graph and generating a multimodal summary graph based on cross-modal fusion vectors, as provided in this application embodiment;

[0048] Figure 5 A flowchart for generating multimodal summaries provided in this application embodiment;

[0049] Figure 6 This is a structural diagram of the environment document processing device based on a large model provided in an embodiment of this application.

[0050] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0052] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0053] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0054] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0055] In the current field of environmental protection and ecological governance, the volume and complexity of various environmental documents are constantly increasing. These documents include environmental assessment reports, monitoring data, regulatory documents, and technical guidelines, covering multiple areas such as pollution emissions, climate change, and law enforcement inspections. Traditional manual methods are no longer sufficient to meet the demands for efficiency and accuracy in extracting effective information from these documents and processing them quickly and accurately. With the advancement of Natural Language Processing (NLP) technology, especially the application of Large Language Models (LLMs), new possibilities have been provided for automated environmental document processing. Based on this, this application provides an environmental document processing method based on a large model, which collects environmental event document data, preprocesses the collected environmental event data, mainly including document segmentation and feature extraction, and then performs feature fusion and constructs a knowledge graph to achieve business question-and-answer retrieval.

[0056] like Figure 1 As shown, the environment document processing method based on the large model includes the following steps S10 to S50.

[0057] S10: Obtain multi-source environmental event document data, which includes text data and image data.

[0058] In this embodiment, the collected multi-source environmental event document data includes formats such as pdf, doc, docx, and txt.

[0059] S20: Preprocess the multi-source environmental event document data, including: segmenting the text data using semantic paragraph slicing and dynamic semantic slicing strategies to obtain preprocessed text data; and detecting the image data using object detection algorithms to obtain preprocessed image data.

[0060] In some embodiments, semantic paragraph slicing and dynamic semantic slicing strategies are employed for text data. Based on the segmentation of semantic units (heading-body pairs, list items, etc.), a window sliding + overlapping strategy is used to ensure contextual coherence. Then, based on the RoBERTa semantic model, the semantic similarity between adjacent segments is judged. If the semantics between adjacent segments are close, they are automatically merged, and their logical boundaries are marked to finally obtain preprocessed text data.

[0061] For image data, target detection algorithms are used to detect smoke areas, vegetation coverage, etc., and the detected smoke areas and vegetation coverage are used as preprocessed image data.

[0062] S30: Based on the task scenario, use a large language model to extract features from preprocessed text data to obtain text modal features, and use the YOLO object detection algorithm to extract image modal features from preprocessed image data.

[0063] In some embodiments, environmental risk prediction and tracing tasks are used as an example, combined with business scenarios.

[0064] For text data, LLM is used to extract key features such as keyword density (frequency of words such as "exceeding the standard" and "leaking") and severity score.

[0065] For image data, target detection algorithms such as YOLO are used to extract key features such as detection results, spatial features, and model detection confidence.

[0066] The system employs adaptive fine-tuning of the large language model (LLM). Addressing the highly structured nature and concentrated terminology of professional texts such as environmental policies and industry technical documents, this system performs industry-specific adaptive fine-tuning on the general large language model. A high-quality training set is constructed by collecting a large amount of environmental domain corpus (such as laws and regulations, implementation plans, standards and specifications, industry guidelines, etc.). Combined with LoRA lightweight parameter efficient fine-tuning technology, the system enhances the model's ability to understand professional terminology, paragraph logic, and policy style without significantly increasing computational resource overhead. The fine-tuning process includes: cleaning, standardizing, and slicing the original corpus; designing prompt-based instructions for tasks such as policy clauses, defined terms, and indicator parameters; and using supervised fine-tuning (SFT) to improve the model's task generalization and semantic alignment capabilities.

[0067] S40: Adaptive fusion of text modal features and image modal features using a dynamic weighting method to obtain a cross-modal fusion vector.

[0068] Taking environmental risk prediction and tracing as an example, this paper adopts a multimodal dynamic weighting approach. Using expert-based risk values ​​as the true labels, global baseline weights for each modality are obtained, a weight label matrix is ​​established, and a weight prediction model is trained to obtain the weight for each modality. Multimodal feature fusion is then performed based on these dynamic weights. The basic idea is that a single-modal feature performs well on similar event samples of the current event sample, and therefore should receive a higher weight score.

[0069] In some embodiments, such as Figure 2 As shown, step S40 can be implemented through the following steps S401-S404.

[0070] S401: Obtain the global baseline weights for each modality.

[0071] Multiple modal features are concatenated to train a global machine learning model, resulting in global baseline weights, which serve as priors and constraints for dynamic weights. These multimodal features include text modal features and image modal features.

[0072] S402: Establish the weight label matrix. For example... Figure 3 As shown, step S402 is implemented through the following steps S4021-S4024.

[0073] S4021: Finding similar samples: Concatenate multiple modal features and find similar event samples of the current event sample based on K nearest neighbors.

[0074] S4022: Training a single-modal model: Based on each modality feature, train a machine learning model, evaluate the model performance, and obtain performance metrics. The machine learning model is not limited to XGBoost, Support Vector Machine, etc.

[0075] S4023: Normalize the performance metrics and perform a weighted average with the global baseline weights to obtain the modal weights of the current event sample after normalization.

[0076] S4024: Iterate through all event samples and output the weight label matrix, represented as:

[0077] ;

[0078] in For event samples i In modality j The weight value.

[0079] S403: Train the weight prediction model to obtain multimodal weights.

[0080] Multimodal features are concatenated and combined with a weight label matrix to establish a weighted prediction regression model, which then outputs the weight of each modality. The weighted prediction regression model is not limited to LGBM, MLP, etc.

[0081] S404: Multimodal feature fusion.

[0082] The text modality features and image modality features are fused using the following formula:

[0083] ;

[0084] in For event samples i In modality j The weight value, For modal number, For event samples i In modality j eigenvectors, Let i be the cross-modal fusion vector of event sample i.

[0085] S50: Construct a knowledge graph based on cross-modal fusion vectors and generate multimodal summaries.

[0086] In some embodiments, such as Figure 4 As shown, step S50 includes the following steps S501 to S503.

[0087] S501: Construct category labels to generate entity nodes, use cross-modal fusion vectors as entity attributes, and generate the association relationship from event s to event e based on expert experience and business processes; where the category labels include pollutant category, pollution source and equipment type, and the association relationship includes causal and impact relationship, composition and attribution relationship.

[0088] S502: Align and link the identified entities with external standard knowledge bases. These external standard knowledge bases include national pollutant databases and local environmental protection regulations databases.

[0089] S503: Generate multimodal summaries.

[0090] Based on different document types, corresponding summarization strategies are triggered to output concise and clearly structured text summaries, while also supporting the embedding of mixed text and images and a closed loop of user feedback.

[0091] like Figure 5 As shown, a multimodal summary can be generated through the following steps S5031 to S5033.

[0092] S5031: Determine the matching of the summary generation strategy.

[0093] In this embodiment, the system can determine whether an input document belongs to a policy document, analysis report, industry standard, or other similar type based on document structure tags, title keywords, and semantic vectors (BERT encoding + KNN classification). According to the recognition results, the system will automatically match the corresponding summary template and Prompt strategy. The Prompt construction adopts a template-based approach combined with a small amount of instruction learning to guide the model to generate summary content under the target structure.

[0094] S5032: Generate multimodal summaries.

[0095] To improve the information density and expression efficiency of the abstract, the system introduces a multimodal abstract generation mechanism, which integrates chart data with text logic to generate an image-text embedded abstract object.

[0096] For example, during chart extraction, the structure parsing module is used to extract charts from the original text (parsing table blocks and image blocks in PDF), and to identify their titles, units, and data fields; structured data (such as pollutant trend tables and ranking charts) are normalized.

[0097] When interpreting charts and graphs, structured table data is converted into text descriptions, such as: "The annual average concentration of PM2.5 in 2024 decreased by 18% year-on-year, continuing the downward trend"; a large model is called to generate bullet-point interpretations of the table questions and answers.

[0098] When the embedding is generated, the image and text summary is encapsulated in JSON format, containing fields such as summary content, image links / image data, and corresponding paragraph references.

[0099] S5033: User ratings and feedback.

[0100] This embodiment supports user rating of summaries by star rating, marking whether key information is missing, and providing free text suggestions. The front end supports error correction annotation for erroneous summary paragraphs, such as replacement, deletion, and editing. Specifically, during data collection and training, user feedback data is uniformly formatted into training sample pairs for subsequent fine-tuning of the summary model. An exemplary training sample pair is the original fragment plus a correct summary. RLHF (Reinforcement Learning with Human Feedback) technology is integrated for reinforcement training. Model version comparison testing and A / B experiments are supported to iteratively optimize the summary generation logic. Furthermore, a confidence threshold can be set; low-confidence summaries must be verified and confirmed by the user before being saved or used for downstream tasks.

[0101] Therefore, this embodiment, based on a knowledge graph structure, supports path queries such as forward prediction and backward tracing, assisting users in achieving intelligent retrieval and causal explanation. The completed knowledge graph also supports linkage with the question-answering module, enabling upstream and downstream data collaboration and improving the overall intelligence level of the system.

[0102] This application also provides an environment document processing device based on a large model, such as... Figure 6 As shown, the environment document processing device based on a large model includes:

[0103] The data acquisition module 601 is configured to acquire multi-source environmental event document data, which includes text data and image data.

[0104] The data preprocessing module 602 is configured to preprocess the multi-source environmental event document data, including: segmenting the text data using semantic paragraph slicing and dynamic semantic slicing strategies to obtain preprocessed text data; and detecting the image data using an object detection algorithm to obtain preprocessed image data.

[0105] The feature extraction module 603 is configured to extract features from the preprocessed text data based on the task scenario using a large language model to obtain text modal features, and to extract image modal features from the preprocessed image data using the YOLO object detection algorithm.

[0106] The feature fusion module 604 is configured to perform multimodal feature adaptive fusion of the text modal features and image modal features using a dynamic weighting method to obtain a cross-modal fusion vector;

[0107] The knowledge graph summarization generation module 605 is configured to construct a knowledge graph based on the cross-modal fusion vector and generate a multimodal summary.

[0108] In some embodiments, the data preprocessing module is further configured to:

[0109] Text data is divided based on semantic units, which include title-body pairs or list items;

[0110] Based on semantic unit segmentation, a window sliding and overlapping strategy is adopted to ensure contextual coherence;

[0111] Based on the RoBERTa semantic model, the semantic similarity between adjacent segments is determined, and semantically closely related paragraphs are automatically merged and logical boundaries are marked to obtain preprocessed text data.

[0112] In some embodiments, the text modal features include keyword density and severity score; the image modal features include detection results, spatial features, and model detection confidence.

[0113] In some embodiments, the feature fusion module is further configured to:

[0114] Multiple modal features are concatenated to train a global machine learning model and obtain global baseline weights; wherein, the modal features include text modal features and image modal features;

[0115] Multiple modal features are concatenated, and similar event samples of the current event sample are found based on K-nearest neighbors; a machine learning model is trained based on each modal feature, and performance indicators are obtained; the performance indicators are normalized, and the normalized performance indicators are weighted and averaged with the global benchmark weights and then normalized to obtain the modal weights of the current event sample; all event samples are traversed and the weight label matrix is ​​output.

[0116] Multimodal features are concatenated and combined with a weight label matrix to establish a weighted prediction regression model, which outputs the weight of each modality.

[0117] Based on each modality weight, text modality features and image modality features are fused to obtain a cross-modality fusion vector.

[0118] In some embodiments, based on each modality weight, the text modality features and image modality features are fused to obtain the cross-modality fusion vector using the following formula:

[0119] ;

[0120] in For event samples i In modality j The weight value, For modal number, For event samples i In modality j eigenvectors, Let i be the cross-modal fusion vector of event sample i.

[0121] In some embodiments, the map summarization generation module is further configured to:

[0122] First, category labels are constructed to generate entity nodes. Cross-modal fusion vectors are used as entity attributes. Based on expert experience and business processes, causal and impact relationships, composition and attribution relationships between events are generated. At the same time, the identified entities are aligned and linked with external standard knowledge bases to complete the construction of the knowledge graph. The category labels include pollutant categories, pollution sources and equipment types.

[0123] Based on the constructed knowledge graph, the system determines the type of input document by document structure tags, title keywords, and semantic vectors, automatically matches the corresponding summary template and Prompt strategy, extracts the charts and graphs in the original text using the structure parsing module, identifies the chart titles, units, and data fields, normalizes the structured data, converts the structured table data into text descriptions, and calls the large model to generate a set interpretation. The system then encapsulates the summary content, image links, graph data, and corresponding paragraph citations to generate a multimodal summary.

[0124] In some embodiments, the apparatus further includes a user feedback module, the user feedback module being configured to:

[0125] Obtain user feedback data, which includes star ratings, annotations of missing key information, free text suggestions, and error correction annotations for summary paragraphs;

[0126] The user feedback data is formatted into training sample pairs, which include original document fragments and their corresponding correct summaries.

[0127] The training samples were used to fine-tune the summary generation model, and the summary generation model was then reinforced through training.

[0128] Set a confidence threshold. For any part of the generated digest with a confidence level lower than the threshold, send it to the user for verification and confirmation. After receiving confirmation, save the digest or use it for downstream tasks.

[0129] This application provides an electronic device. The electronic device may include a processor and a memory, wherein the processor and the memory can communicate; exemplarily, the processor and the memory communicate via a communication bus.

[0130] The processor executes computer execution instructions stored in memory, causing the processor to perform the scheme in the above embodiments. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0131] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0132] The electronic device provided in this application embodiment can be the terminal device described in the above embodiments.

[0133] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer performs the technical solution of the environment document processing method based on a large model described in the above embodiments.

[0134] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the environment document processing method based on the large model described in the above embodiments.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0136] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0137] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0138] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0139] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0140] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0141] Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Architecture (EISA) buses, etc. Buses can be categorized into address buses, data buses, control buses, etc.

[0142] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0143] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0144] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for processing environment documents based on a large model, characterized in that, The method includes: Acquire multi-source environmental event document data, which includes text data and image data; The preprocessing of the multi-source environmental event document data includes: segmenting the text data using semantic paragraph slicing and dynamic semantic slicing strategies to obtain preprocessed text data; and detecting the image data using an object detection algorithm to obtain preprocessed image data. Based on the task scenario, a large language model is used to extract features from the preprocessed text data to obtain text modal features, and the YOLO object detection algorithm is used to extract image modal features from the preprocessed image data. The text modal features and image modal features are adaptively fused using a dynamic weighting method to obtain a cross-modal fusion vector; A knowledge graph is constructed based on the cross-modal fusion vector, and a multimodal summary is generated; The text modal features and image modal features are adaptively fused using a dynamic weighting method to obtain a cross-modal fusion vector, including: Multiple modal features are concatenated to train a global machine learning model and obtain global baseline weights; wherein, the modal features include text modal features and image modal features; Multiple modal features are concatenated, and similar event samples of the current event sample are found based on K-nearest neighbors; a machine learning model is trained based on each modal feature, and performance indicators are obtained; the performance indicators are normalized, and the normalized performance indicators are weighted and averaged with the global benchmark weights and then normalized to obtain the modal weights of the current event sample; all event samples are traversed and the weight label matrix is ​​output. Multimodal features are concatenated and combined with a weight label matrix to establish a weighted prediction regression model, which outputs the weight of each modality. Based on each modal weight, text modal features and image modal features are fused to obtain a cross-modal fusion vector; A knowledge graph is constructed based on the cross-modal fusion vector, and a multimodal summary is generated, including: First, category labels are constructed to generate entity nodes. Cross-modal fusion vectors are used as entity attributes. Based on expert experience and business processes, causal and impact relationships, composition and attribution relationships between events are generated. At the same time, the identified entities are aligned and linked with external standard knowledge bases to complete the construction of the knowledge graph. The category labels include pollutant categories, pollution sources and equipment types. Based on the constructed knowledge graph, the input document type is determined by document structure tags, title keywords and semantic vectors. The corresponding summary template and Prompt strategy are automatically matched. The structure parsing module extracts the charts in the original text, identifies the chart titles, units and data fields and normalizes the structured data. The structured table data is converted into text descriptions and the large model is called to generate the set interpretation. The summary content, image links, graph data and corresponding paragraph references are encapsulated to generate multimodal summaries. After generating the multimodal summary, the method further includes: Obtain user feedback data, which includes star ratings, annotations of missing key information, free text suggestions, and error correction annotations for summary paragraphs; The user feedback data is formatted into training sample pairs, which include original document fragments and their corresponding correct summaries. The training samples were used to fine-tune the summary generation model, and the summary generation model was then reinforced through training. Set a confidence threshold. For the part of the generated summary with a confidence level lower than the threshold, send it to the user terminal for verification and confirmation. After receiving confirmation, save the summary or use it for downstream tasks. The formula for calculating the cross-modal fusion vector by fusing text modal features and image modal features is as follows: ； in For event samples i In modality j The weight value, For modal number, For event samples i In modality j eigenvectors, Let i be the cross-modal fusion vector of event sample i.

2. The environment document processing method based on a large model according to claim 1, characterized in that, Methods for segmenting text data using semantic paragraph slicing and dynamic semantic slicing strategies to obtain preprocessed text data include: Text data is divided based on semantic units, which include title-body pairs or list items; Based on semantic unit segmentation, a window sliding and overlapping strategy is adopted to ensure contextual coherence; Based on the RoBERTa semantic model, the semantic similarity between adjacent segments is determined, and semantically closely related paragraphs are automatically merged and logical boundaries are marked to obtain preprocessed text data.

3. The environment document processing method based on a large model according to claim 1, characterized in that, The text modal features include keyword density and severity score; the image modal features include detection results, spatial features, and model detection confidence.

4. A large-model-based environment document processing apparatus for implementing the method as described in any one of claims 1 to 3, characterized in that, The device includes: The data acquisition module is configured to acquire multi-source environmental event document data, which includes text data and image data. The data preprocessing module is configured to preprocess the multi-source environmental event document data, including: segmenting the text data using semantic paragraph slicing and dynamic semantic slicing strategies to obtain preprocessed text data; and detecting the image data using an object detection algorithm to obtain preprocessed image data. The feature extraction module is configured to extract features from the preprocessed text data based on the task scenario using a large language model to obtain text modal features, and to extract image modal features from the preprocessed image data using the YOLO object detection algorithm. The feature fusion module is configured to perform multimodal feature adaptive fusion of the text modal features and image modal features using a dynamic weighting method to obtain a cross-modal fusion vector, including: Multiple modal features are concatenated to train a global machine learning model and obtain global baseline weights; wherein, the modal features include text modal features and image modal features; Multiple modal features are concatenated, and similar event samples of the current event sample are found based on K-nearest neighbors; a machine learning model is trained based on each modal feature, and performance indicators are obtained; the performance indicators are normalized, and the normalized performance indicators are weighted and averaged with the global benchmark weights and then normalized to obtain the modal weights of the current event sample; all event samples are traversed and the weight label matrix is ​​output. Multimodal features are concatenated and combined with a weight label matrix to establish a weighted prediction regression model, which outputs the weight of each modality. Based on each modal weight, text modal features and image modal features are fused to obtain a cross-modal fusion vector; The knowledge graph summarization generation module is configured to construct a knowledge graph based on the cross-modal fusion vector and generate a multimodal summary.

5. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the environment document processing method based on a large model as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the environment document processing method based on a large model as described in any one of claims 1-3.

Images