Electronic file processing method and device based on multi-modal data

Abstract

The invention provides an electronic archive processing method and device based on multi-modal data. The method comprises the following steps: acquiring normalized electronic archive data in a current period; performing layout analysis on the electronic archive data, and identifying text information of the electronic archive data; utilizing a natural language processing model to extract entities and corresponding business elements in the text information; matching the entity with an existing entity in a stored electronic archive knowledge graph, and determining a unique entity identifier; based on the semantic features of the electronic archive data, the business elements and the text information, determining semantic relationships among different electronic archives; updating the stored electronic archive knowledge graph by taking the unique entity identifier as a node and the semantic relationship as an edge to obtain a new electronic archive knowledge graph; and based on the business elements and the new electronic archive knowledge graph, classifying the electronic archive data to obtain a classification result and corresponding cataloguing information. According to the method, intelligent extraction, semantic association and automatic cataloguing of the electronic archive elements are realized.

Description

Technical Field

[0001] This application relates to the field of electronic information processing technology, and more specifically, to a method and apparatus for processing electronic archives based on multimodal data. Background Technology

[0002] As enterprises deepen their digital transformation, the management of electronic records has become increasingly complex. Existing Electronic Records Management Systems (EDMS) primarily handle diverse unstructured data sources, such as instant messaging (IM) tools like WeChat, DingTalk, and Lark, emails, scanned images, engineering drawings, and PDF documents. These data sources generate electronic records characterized by diverse origins, unstructured content, and massive quantities. Existing technologies typically employ simple Optical Character Recognition (OCR) engines to convert images into two-layer PDFs for full-text retrieval, or use rule / template-based extraction methods (such as regular expressions or coordinate templates) to extract specific fields. Furthermore, existing technologies for archiving IM and emails often simply export them to Excel or HTML formats, lacking contextual restructuring and semantic association. Records are often stored in isolation, failing to automatically link them to form a complete chain of business evidence. Existing solutions have poor generalization capabilities when handling non-standard documents (such as correspondence) and rely on manual cataloging, resulting in low efficiency and a large number of records becoming "dead files," difficult to retrieve and utilize. Summary of the Invention

[0003] The purpose of this application is to provide an electronic archive processing method and apparatus based on multimodal data, which realizes intelligent extraction, semantic association and automatic cataloging of electronic archive elements by constructing an automated processing pipeline from multi-source heterogeneous data to structured knowledge graphs, fundamentally solving the core problems of low efficiency of manual operation and lack of logical connection between archives in traditional archive management.

[0004] Firstly, a method for processing electronic archives based on multimodal data is provided, which may include: Based on multiple data sources monitored in real time, the standardized electronic archive data for the current period is obtained. The layout of the electronic archive data is analyzed to identify the text information in the electronic archive data; Using a natural language processing model, entities and the business elements involved in the corresponding entities are extracted from the text information; The extracted entities are matched with existing entities in the stored electronic archive knowledge graph to determine unique entity identifiers; the semantic relationships between different electronic archives are determined based on the semantic features of the electronic archive data, business elements, and text information; and the stored electronic archive knowledge graph is updated with unique entity identifiers as nodes and semantic relationships as edges to obtain a new electronic archive knowledge graph. Based on business elements and a new electronic archives knowledge graph, electronic archives data are classified to obtain classification results and corresponding cataloging information.

[0005] In one possible implementation, the formatted electronic archive data for the current period is obtained, including: Initial electronic archive data is received from multiple data sources, including mail servers, instant messaging systems, and document management systems, through predefined API interfaces or file listening services. The initial electronic archive data includes email data, instant messaging records, audio and video files, and image / PDF documents. The received initial electronic archive data is standardized according to type to obtain standardized electronic archive data, which transforms the initial electronic archive data into a unified intermediate representation format file, including text stream, image block coordinates, and metadata header. The standardization process includes: parsing the MIME structure of email data to separate the email header, body, and attachments; performing session aggregation on instant messaging records based on time windows and participant sets; performing noise reduction, tilt correction, and optical character recognition preprocessing on image / PDF documents; and converting audio and video files into time-stamped subtitle files.

[0006] In one possible implementation, layout analysis is performed on the electronic archive data to identify the textual information of the electronic archive data, including: Using a document layout analysis model trained based on deep learning, the page pixels of the electronic archive data are segmented to obtain different semantic regions. The semantic regions include the physical regions containing at least two of the following: title, body text, table, image, seal, header / footer, and signature. The content in different semantic regions is identified to obtain the text content of the identified electronic archive data and the coordinate position of the corresponding text content in the page layout of the electronic archive data.

[0007] In one possible implementation, a natural language processing model is used to extract entities and corresponding business elements from the text information, including: Entities are quickly extracted from text information using the first natural language processing model; The text information and related context information are input into the large language model, and the large language model is guided by prompting engineering to output the business elements involved in the corresponding entity.

[0008] In one possible implementation, the semantic relationships between different electronic archives are determined based on the semantic features of the electronic archive data, business elements, and text information, including: Establish explicit relationships based on metadata, including establishing inclusion relationships based on the inherent parent-child structure in electronic archive data; Establish logical relationships based on business elements, including establishing a project association relationship when the key entities of project numbers extracted from two electronic archive data match. Establish implicit relationships based on semantic feature similarity calculation, including establishing content similarity or version association when the semantic feature similarity of two electronic archive data exceeds a threshold; Parse the reference patterns in the text information, and when the text content of the first electronic archive data mentions the unique identifier of the second electronic archive data, establish a reference relationship between the first electronic archive data and the second electronic archive data.

[0009] In one possible implementation, electronic record data is classified based on business elements and a new electronic record knowledge graph, yielding classification results and corresponding cataloging information, including: The business elements and semantic features of electronic archive data are matched with a predefined archive classification tree, and the category to which it belongs is predicted through a hierarchical classification model. Based on the classification results and the pre-set retention period table, calculate the retention period for the electronic archive data; When the extracted business elements lack standardized titles, the large language model is invoked to generate standardized titles based on the context of the text information, and these titles are automatically filled into the cataloging information entries.

[0010] In one possible implementation, the method further includes: Obtain confidence scores for the extracted and classified results of business elements; When the confidence score is lower than the preset threshold, the corresponding data will be pushed to the manual verification interface. Receive feedback data from manual verification and use this feedback data to optimize the natural language processing model and classification model used.

[0011] Secondly, an electronic record processing device based on multimodal data is provided, the device may include: The acquisition unit is used to acquire standardized electronic archive data within the current period based on multiple data sources monitored in real time. The identification unit is used to perform layout analysis on the electronic archive data and identify the text information of the electronic archive data; The extraction unit is used to extract entities and corresponding business elements from the text information using a natural language processing model. The determining unit is used to match the extracted entities with existing entities in the stored electronic archive knowledge graph to determine a unique entity identifier; and to determine the semantic relationships between different electronic archives based on the semantic features of the electronic archive data, business elements and text information. The update unit is used to update the stored electronic archive knowledge graph with unique entity identifiers as nodes and semantic relationships as edges, so as to obtain a new electronic archive knowledge graph. The classification unit is used to classify electronic archive data based on business elements and a new electronic archive knowledge graph, and to obtain classification results and corresponding cataloging information.

[0012] Thirdly, an electronic device is provided, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements any of the steps described in the first aspect above.

[0013] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of any of the methods described in the first aspect above.

[0014] This application provides a method and apparatus for processing electronic archives based on multimodal data. The method acquires standardized electronic archive data for the current period from multiple real-time monitored data sources; performs layout analysis on the electronic archive data to identify textual information; uses a natural language processing model to extract entities and corresponding business elements from the textual information; matches the extracted entities with existing entities in a stored electronic archive knowledge graph to determine unique entity identifiers; determines semantic relationships between different electronic archives based on the semantic features of the electronic archive data, business elements, and textual information; updates the stored electronic archive knowledge graph using unique entity identifiers as nodes and semantic relationships as edges to obtain a new electronic archive knowledge graph; and classifies the electronic archive data based on business elements and the new electronic archive knowledge graph to obtain classification results and corresponding cataloging information. This method, by constructing an automated processing pipeline from multi-source heterogeneous data to a structured knowledge graph, achieves intelligent extraction, semantic association, and automatic cataloging of electronic archive elements, fundamentally solving the core problems of low efficiency in traditional archive management and lack of logical connections between archives. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating an electronic archive processing method based on multimodal data, provided for an embodiment of this application; Figure 2 A schematic diagram of the structure of an electronic archive processing device based on multimodal data provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application. Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by those skilled in the art. The terms "first," "second," and similar terms used in this application do not indicate any order, quantity, or importance, but are only used to distinguish different components. The terms "comprising" or "including," etc., mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, but does not exclude other elements or objects. The terms "connection," "coupled," or "linked," etc., are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0018] Existing technologies face challenges in recording unstructured archive metadata, resulting in disorganized archiving of multimodal content (such as IM and email), a lack of logical connections between archives, and poor generalization capabilities of template-based extraction methods. This leads to low archiving efficiency, insufficient retrieval accuracy, and an inability to support the complete chain of evidence required for auditing and compliance. Specifically, existing systems cannot automatically identify key business elements (such as amounts and dates) in documents, cannot reconstruct fragmented communication records into logical documents, and cannot build semantic networks between archives to achieve intelligent relational queries.

[0019] The electronic record processing method based on multimodal data provided in this application embodiment can be applied to an electronic record processing system, which may include: a data access module, an AI model engine, a knowledge graph engine, a business logic and workflow engine, and a record management application module.

[0020] Data access module: Provides API interfaces, RPA robots and batch import tools for connecting to OA, email servers, instant messaging (IM) systems such as WeChat / DingTalk / Lark, and data sources such as scanners.

[0021] AI Model Engine: Integrates multiple pre-trained models, including document layout analysis models (such as LayoutLMv3, DiT), OCR engine, dedicated NER models (such as BERT / BiLSTM-CRF), large language models (LLM, such as Qwen, Llama3), table recognition models (based on GNN), speech recognition (ASR) engine, etc.

[0022] Knowledge graph engines use graph databases (such as Neo4j or NebulaGraph) to store and manage archive entities and relationships.

[0023] Business logic and workflow engine: Controls the execution order of the entire pipeline and implements a human-in-the-loop mechanism.

[0024] The document management application module provides electronic archives, classification management, cataloging rule configuration, and a manual verification workbench.

[0025] This system aims to address the pain points of traditional archival documentation, such as time-consuming manual cataloging, difficulty in retrieving unstructured content, and lack of logical connections between documents. By integrating layout analysis, OCR, natural language processing (NLP), and large language modeling (LLM) technologies, it achieves automatic conversion from raw documents to structured knowledge graphs.

[0026] It runs as an infinite loop service, continuously listening for data input. This applies to each batch or each newly arriving archive data.

[0027] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0028] Figure 1 This is a flowchart illustrating an electronic record processing method based on multimodal data, provided as an embodiment of this application. Figure 1 As shown, the method may include: Step S110: Based on multiple data sources monitored in real time, obtain the standardized electronic archive data for the current period.

[0029] Initial electronic record data is received from multiple data sources through predefined API interfaces or file monitoring services. Data sources may include mail servers, instant messaging systems, and document management systems. Initial electronic record data may include email data, instant messaging records, audio and video files, and image / PDF documents. Specifically, new data input events from configured sources (such as OA systems, mail servers, IM archiving servers, and scanner output directories) can be monitored through predefined API interfaces, RPA robots, or file monitoring services. At least the following data can be obtained: 1) a scanned PDF copy of a purchase contract sent by a supplier; 2) a DingTalk group chat record (JSON format streaming data) regarding the approval of the contract; 3) an email confirming the final contract (EML format).

[0030] Subsequently, the received initial electronic archive data is standardized according to type to obtain standardized electronic archive data, which transforms the initial electronic archive data into a unified intermediate representation format file containing text streams, image block coordinates, and metadata headers. The standardization process includes: parsing the MIME structure of email data to separate the email header, body, and attachments; performing session aggregation on instant messaging records based on time windows and participant sets; performing noise reduction, tilt correction, and optical character recognition preprocessing on image / PDF documents; and converting audio and video files into timestamped subtitle files.

[0031] Specifically: Based on file extensions (e.g., .pdf, .jpg, .eml) and content, identify the type and corresponding format of the initial electronic archival data, and perform standardization processing, including: Image / PDF Processing: For contract PDFs, the system identifies them as image-based PDFs. First, it calls an image processing library to perform adaptive binarization and perspective transformation, removing background noise and distortion caused by scanning. Then, it rasterizes the image into a high-resolution image, ready to be fed into the OCR engine.

[0032] Instant Messaging (IM) Processing: For chat logs from WeChat / DingTalk / Lark, since IM data is fragmented and streaming, the system introduces and executes a "conversation window aggregation algorithm." Based on a sliding time window (default 30 minutes) and a fixed set of participants, the algorithm aggregates consecutive chat logs (including text, emoticons, and an attachment link) about "XX project procurement contract review" into a logical "logical document." Simultaneously, the system downloads attachment files (images, file links) from the chat and replaces the link placeholders in the original text, ultimately generating a long-term saved OFD or HTML rich text document.

[0033] Email data processing: Parse the EML file to extract sender, recipient, subject, and date from the email header as basic metadata. Parse the email body (HTML), remove CSS styles and advertising signatures, and extract clean text. Recursively extract all attachments from the email and record the parent-child hash association between attachments and the email body.

[0034] Audio and video processing: Integrates an ASR (Automatic Speech Recognition) engine to convert meeting recordings and video conference notes into text copies and generate timestamped subtitle files, which are then processed along with the original audio and video.

[0035] Output Intermediate Representation (IR): Encapsulate all the above processing results into a unified IR object. This object contains: a text content stream, a list of image patch coordinates, and file metadata (such as source, format, time, etc.). This IR object will serve as input for all subsequent steps.

[0036] Step S120: Perform layout analysis on the electronic archive data to identify the text information in the electronic archive data.

[0037] In practice, a document layout analysis model trained using deep learning is used to segment the page pixels of electronic archive data, obtaining different semantic regions. These semantic regions include the physical areas containing at least two of the following: title, body text, tables, images, seals, headers / footers, and signatures. Specifically, the contract PDF is input into a pre-trained LayoutLMv3 model for inference. The model segments the page pixels into different semantic regions and outputs coordinate labels, such as: Title, Text, Table (coordinates [x, y]), Signature (coordinates [x, y]), Figure, Header / Footer, and Stamp. The system integrates visual features (layout) with textual features (semantics) to accurately distinguish between "body content" and "interference information" (such as watermarks and page numbers).

[0038] When the identified semantic region is a table, a table structure recognition model is invoked to reconstruct the row and column relationships and merging relationships of the table cells based on visual features, outputting structured table data. Specifically, for the identified table region, the system invokes a table structure recognition model based on a graph neural network (GNN). The model adopts a "segmentation-merging" strategy: first, it detects table border lines or alignment features to identify the coordinates of each cell; then, it uses GNN to predict the row / column merging relationships between cells, finally outputting structured data (such as Excel format) that retains complete row and column logic, ensuring that "amount" and "item" correspond accurately and preventing misaligned rows.

[0039] The content in different semantic regions is identified to obtain the text content of the identified electronic archive data and the coordinate position of the corresponding text content in the page layout of the electronic archive data.

[0040] Specifically, high-precision OCR can be performed on different semantic regions of the entire page image, not only extracting text but also recording the bounding box coordinates of each character or term. The text stream output by OCR is aligned and fused with the layout structure output by LayoutLMv3, enabling the system to recognize that the "Total Contract Amount" number appears to the right or below the "Total Contract Amount" label. For the recognition of seals and handwriting, a dedicated seal detection model can be trained to identify the seal in the signature area and extract the text within the seal. For handwritten signatures, a specialized handwriting OCR engine is used for recognition.

[0041] Step S130: Use a natural language processing model to extract entities from the text information and the business elements involved in those entities.

[0042] Using the first natural language processing model, entities can be quickly extracted from text information. Entities can include dates, names of people, or names of organizations. The text information and related context information are input into the large language model, and the large language model is guided by prompting engineering to output the business elements involved in the corresponding entities.

[0043] Specifically, this step employs a cascaded collaborative architecture of "small model + large model" to achieve the transformation from text to structured data, including: Common Field Extraction (NER): Input the clean text in IR into a lightweight BiLSTM-CRF or BERT-NER model to quickly extract common entities, such as: time ("May 20, 2024" normalized to "2024-05-20"), person's name ("Wang Si"), organization name ("XXX Company").

[0044] Key Business Elements Extraction (KIE): For fixed-format documents such as contracts, invoices, and official documents, a multimodal pre-trained model (integrating text, coordinates, and visual features) can be used. The model can recognize that "amount" usually appears to the right of "total," thus accurately extracting fields such as "total contract amount" and "payment method."

[0045] Long Text Understanding (LLM Application): For DingTalk chat logs aggregated into logical documents, the system invokes the Large Language Model (LLM) through PromptEngineering. An example Prompt is: "Read the following chat log, extract the project name discussed, the core points of contention in the contract, the final consensus reached, and generate a summary. Please output in JSON format." After LLM analysis, structured key-value pair data is output.

[0046] Data cleaning and validation: Perform logical validation on the extracted results, such as verifying that the "end date" is later than the "start date".

[0047] In a specific example, lightweight analysis (such as keyword matching and regular expressions) is first performed on the text information to determine the type of the fields to be extracted. High-frequency, structured fields (such as dates and amounts) are routed to the fast channel and processed by an optimized BERT small model. The small model (such as TinyBERT trained by distillation) performs millisecond-level inference and outputs the entity and its confidence score. Low-frequency fields that require deep semantic understanding (such as "description of potential contract risks") are routed to the complex channel. The system dynamically constructs a Prompt containing task instructions and context, calls the API of a Large Language Model (LLM) (such as OpenAI GPT or locally deployed Llama), and parses the returned JSON results. Then, the results from the two channels are merged, and conflict detection is performed (e.g., if both the small and large models extract dates, the result with higher confidence is used).

[0048] Step S140: Match the extracted entities with existing entities in the stored electronic archive knowledge graph to determine unique entity identifiers, and determine the semantic relationships between different electronic archives based on the semantic features of electronic archive data, business elements and text information, so as to obtain a new electronic archive knowledge graph.

[0049] By matching the extracted entities with existing entities in the stored electronic archive knowledge graph, a unique entity identifier (ID) is determined. This implementation method can achieve entity alignment, i.e., solve the problem of inconsistent names. For example, it can identify that "Project A" and "Proj-A" are the same project, thereby establishing a unified entity database.

[0050] Based on the semantic features of electronic archive data, business elements, and textual information, semantic relationships are established between different electronic archives, specifically including: (1) Establish explicit relationships based on metadata, including establishing inclusion relationships based on the inherent parent-child structure in electronic archive data; (2) Establish logical relationships based on business elements, including establishing a project association relationship when the key entities of the project number extracted from two electronic archive data match; for example, the contract PDF, DingTalk chat records, and emails all mention "Project Number: PROJ-2024-001", and the system automatically associates them with the "PROJ-2024-001" project node in the graph.

[0051] (3) Establish implicit relationships based on semantic feature similarity calculation, including establishing content similarity or version association when the semantic feature similarity of two electronic archive data exceeds the threshold; (4) Parse the reference patterns in the text information. When the text content of the first electronic archive data mentions the unique identifier of the second electronic archive data, establish a reference relationship between the first electronic archive data and the second electronic archive data. For example, if "see DingTalk group chat record" is parsed in the email body, establish a reference relationship from the email node to the DingTalk logical document node.

[0052] (5) Establish process relationships and link chat records (requests) and emails (approvals) into a complete business chain based on the approval process ID.

[0053] Furthermore, the stored electronic archive knowledge graph is updated using unique entity identifiers (IDs) as nodes and semantic relationships as edges to obtain a new electronic archive knowledge graph. For example, the aforementioned archives are stored as nodes and relationships as edges in a graph database. At this point, a semantic network is formed in the graph, centered on "PROJ-2024-001 project" and connecting multiple nodes such as contracts, emails, and chat logs.

[0054] Step S150: Based on business elements and the new electronic archives knowledge graph, classify the electronic archives data to obtain classification results and corresponding cataloging information.

[0055] The business elements and semantic features of electronic archival data are matched with a predefined archival classification tree, and a hierarchical classification model is used to predict its category. In practice, the enterprise's archival classification tree (Taxonomy) is first pre-defined. Then, the feature vector of the contract text is input into the hierarchical classification model to calculate its similarity to each node in the classification tree (e.g., business category -> contract agreement -> procurement contract), obtaining the classification result. Based on the classification result, it is automatically assigned to the most matching category. The acquired business elements are then automatically populated into the corresponding bibliographical entries for that category.

[0056] Based on the classification results and the pre-set retention period table, calculate the retention period for the electronic archive data; When the extracted business elements lack standardized titles (the original title of the DingTalk chat record is meaningless, such as "Are you there?"), the large language model is invoked to generate standardized titles based on the context of the text information (such as chat record about the review of the procurement contract for project PROJ-2024-001 - 20241027), and these titles are automatically filled into the cataloging information.

[0057] In some embodiments, this application introduces a Human-in-the-loop mechanism to ensure quality and achieve self-evolution. The method further includes: Obtain confidence scores for the extracted and classified results of business elements; When the confidence score is lower than the preset threshold, the corresponding data will be pushed to the manual verification interface. Receive feedback data from manual verification and use this feedback data to optimize the natural language processing model and classification model used.

[0058] Furthermore, the Parameter Efficient Fine-Tuning (PEFT / LoRA) technique can be used to perform lightweight updates to the model daily or weekly, making the system increasingly intelligent with use and gradually adapting to the enterprise's unique document terminology and classification habits.

[0059] For example, if the confidence level of the LLM's extraction result for "contract dispute points" is 0.78 (below the threshold of 0.85), the task is automatically intercepted by the system. The task is then pushed to the "Manual Verification Workbench." In the "Left Image, Right Text" interface, the administrator sees the original chat history highlighted with relevant text on the left, and the LLM's extraction result on the right. The administrator confirms or corrects the data and submits it. The system collects the manually corrected data as reinforcement learning feedback (RLHF) samples. Through a scheduled task (e.g., every Sunday morning), when the task is triggered, the size of the feedback sample library is checked. If the sample size reaches a set value, the system automatically uses parameter efficient fine-tuning (PEFT / LoRA) techniques to incrementally train the LLM model, optimizing its performance when processing similar chat history in the future.

[0060] In some embodiments, after step S150 is performed, the behavioral data of each entity in the electronic archive knowledge graph can be continuously monitored and quantified to construct time-series features, including: access frequency, number of citations (linked by other archives in the graph), popularity of related discussions (number of associated IMs / emails), number of downloads, etc.

[0061] A time series forecasting model (such as the Transformer-based Informer model or a simplified LSTM) is used to learn the above-mentioned indicators for each archive, predicting its activity curve for a future period (such as the next quarter) to predict future activity. Simultaneously, a graph neural network (GNN) is used to analyze the archive's centrality in the knowledge graph (such as a variant of the PageRank algorithm) to quantify its structural influence score.

[0062] The predicted future activity level is weighted and combined with the structural influence score to obtain a comprehensive archive value score. Based on this score, archives are automatically classified into different levels, such as hot archives, active archives, dormant archives, and archives awaiting evaluation.

[0063] High-frequency files are stored in high-speed storage; dormant files can be migrated to low-cost object storage or cold storage, and administrators will be prompted to review their retention period. Warnings can also be issued for high-impact but low-activity files (such as foundational contracts), prompting for key backups or compliance reviews to prevent potential risks from being overlooked due to their dormancy.

[0064] The electronic record processing method of this application integrates multimodal analysis, large language model and knowledge graph technology to create an integrated intelligent processing system with self-learning capabilities for element extraction, structuring and cataloging. This not only automates and automates the entire record management process, but also transforms it from a passive repository into an active knowledge asset that can support semantic retrieval, business insights and risk management.

[0065] Corresponding to the above method, embodiments of this application also provide an electronic record processing device based on multimodal data, such as... Figure 2 As shown, the device includes: The acquisition unit 210 is used to acquire standardized electronic archive data within the current period based on multiple data sources monitored in real time. The identification unit 220 is used to perform layout analysis on the electronic archive data and identify the text information of the electronic archive data; Extraction unit 230 is used to extract entities and corresponding business elements from the text information using a natural language processing model; The determining unit 240 is used to match the extracted entities with existing entities in the stored electronic archive knowledge graph to determine a unique entity identifier; and to determine the semantic relationships between different electronic archives based on the semantic features of the electronic archive data, business elements and text information. The update unit 250 is used to update the stored electronic archive knowledge graph with unique entity identifiers as nodes and semantic relationships as edges, so as to obtain a new electronic archive knowledge graph. Classification unit 260 is used to classify electronic archive data based on business elements and a new electronic archive knowledge graph, and obtain classification results and corresponding cataloging information.

[0066] The functions of each functional unit of the electronic archive processing device based on multimodal data provided in the above embodiments of this application can be implemented through the above methods and steps. Therefore, the specific working process and beneficial effects of each unit in the electronic archive processing device based on multimodal data provided in the embodiments of this application will not be repeated here.

[0067] This application also provides an electronic device, such as... Figure 3 As shown, it includes a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340.

[0068] Memory 330 is used to store computer programs; When the processor 310 executes the program stored in the memory 330, it performs the following steps: Based on multiple data sources monitored in real time, the standardized electronic archive data for the current period is obtained. The layout of the electronic archive data is analyzed to identify the text information in the electronic archive data; Using a natural language processing model, entities and the business elements involved in the corresponding entities are extracted from the text information; The extracted entities are matched with existing entities in the stored electronic archive knowledge graph to determine unique entity identifiers; the semantic relationships between different electronic archives are determined based on the semantic features of the electronic archive data, business elements, and text information; and the stored electronic archive knowledge graph is updated with unique entity identifiers as nodes and semantic relationships as edges to obtain a new electronic archive knowledge graph. Based on business elements and a new electronic archives knowledge graph, electronic archives data are classified to obtain classification results and corresponding cataloging information.

[0069] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0070] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0071] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0072] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0073] The implementation methods and beneficial effects of the various components of the electronic device in the above embodiments for solving the problem can be found in [reference needed]. Figure 1 The steps in the illustrated embodiments are used to implement the electronic device. Therefore, the specific working process and beneficial effects of the electronic device provided in this application will not be repeated here.

[0074] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the electronic document processing methods based on multimodal data described in the above embodiments.

[0075] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the electronic document processing methods based on multimodal data described in the above embodiments.

[0076] Those skilled in the art will understand that the embodiments in this application can be provided as methods, systems, or computer program products. Therefore, the embodiments in this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments in this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0078] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0080] Although preferred embodiments have been described in this application, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0081] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims in this application and their equivalents, then this application also intends to include these modifications and variations.

Claims

1. A method for processing electronic archives based on multimodal data, characterized in that, The method includes: Based on multiple data sources monitored in real time, the standardized electronic archive data for the current period is obtained. The layout of the electronic archive data is analyzed to identify the text information in the electronic archive data; Using a natural language processing model, entities and the business elements involved in the corresponding entities are extracted from the text information; The extracted entities are matched with existing entities in the stored electronic archive knowledge graph to determine unique entity identifiers; the semantic relationships between different electronic archives are determined based on the semantic features of the electronic archive data, business elements, and text information; and the stored electronic archive knowledge graph is updated with unique entity identifiers as nodes and semantic relationships as edges to obtain a new electronic archive knowledge graph. Based on business elements and a new electronic archives knowledge graph, electronic archives data are classified to obtain classification results and corresponding cataloging information.

2. The method as described in claim 1, characterized in that, Retrieve the formatted electronic archive data for the current period, including: Initial electronic archive data is received from multiple data sources, including mail servers, instant messaging systems, and document management systems, through predefined API interfaces or file listening services. The initial electronic archive data includes email data, instant messaging records, audio and video files, and image / PDF documents. The received initial electronic archive data is standardized according to type to obtain standardized electronic archive data, which transforms the initial electronic archive data into a unified intermediate representation format file, including text stream, image block coordinates, and metadata header. The standardization process includes: parsing the MIME structure of email data to separate the email header, body, and attachments; performing session aggregation on instant messaging records based on time windows and participant sets; performing noise reduction, tilt correction, and optical character recognition preprocessing on image / PDF documents; and converting audio and video files into time-stamped subtitle files.

3. The method as described in claim 1, characterized in that, The layout of the electronic archive data is analyzed to identify the text information of the electronic archive data, including: Using a document layout analysis model trained based on deep learning, the page pixels of the electronic archive data are segmented to obtain different semantic regions. The semantic regions include the physical regions containing at least two of the following: title, body text, table, image, seal, header / footer, and signature. The content in different semantic regions is identified to obtain the text content of the identified electronic archive data and the coordinate position of the corresponding text content in the page layout of the electronic archive data.

4. The method as described in claim 1, characterized in that, Using a natural language processing model, entities and corresponding business elements related to the entities in the text information are extracted, including: Entities are quickly extracted from text information using the first natural language processing model; The text information and related context information are input into the large language model, and the large language model is guided by prompting engineering to output the business elements involved in the corresponding entity.

5. The method as described in claim 1, characterized in that, Based on the semantic features of the electronic archive data, business elements, and text information, the semantic relationships between different electronic archives are determined, including: Establish explicit relationships based on metadata, including establishing inclusion relationships based on the inherent parent-child structure in electronic archive data; Establish logical relationships based on business elements, including establishing a project association relationship when the key entities of project numbers extracted from two electronic archive data match. Establish implicit relationships based on semantic feature similarity calculation, including establishing content similarity or version association when the semantic feature similarity of two electronic archive data exceeds a threshold; Parse the reference patterns in the text information, and when the text content of the first electronic archive data mentions the unique identifier of the second electronic archive data, establish a reference relationship between the first electronic archive data and the second electronic archive data.

6. The method as described in claim 1, characterized in that, Based on business elements and a new electronic archives knowledge graph, electronic archives data is classified to obtain classification results and corresponding cataloging information, including: The business elements and semantic features of electronic archive data are matched with a predefined archive classification tree, and the category to which it belongs is predicted through a hierarchical classification model. Based on the classification results and the pre-set retention period table, calculate the retention period for the electronic archive data; When the extracted business elements lack standardized titles, the large language model is invoked to generate standardized titles based on the context of the text information, and these titles are automatically filled into the cataloging information entries.

7. The method as described in claim 1, characterized in that, The method further includes: Obtain confidence scores for the extracted and classified results of business elements; When the confidence score is lower than the preset threshold, the corresponding data will be pushed to the manual verification interface. Receive feedback data from manual verification and use this feedback data to optimize the natural language processing model and classification model used.

8. An electronic archive processing device based on multimodal data, characterized in that, The device includes: The acquisition unit is used to acquire standardized electronic archive data within the current period based on multiple data sources monitored in real time. The identification unit is used to perform layout analysis on the electronic archive data and identify the text information of the electronic archive data; The extraction unit is used to extract entities and corresponding business elements from the text information using a natural language processing model. The determining unit is used to match the extracted entities with existing entities in the stored electronic archive knowledge graph to determine a unique entity identifier; and to determine the semantic relationships between different electronic archives based on the semantic features of the electronic archive data, business elements and text information. The update unit is used to update the stored electronic archive knowledge graph with unique entity identifiers as nodes and semantic relationships as edges, so as to obtain a new electronic archive knowledge graph. The classification unit is used to classify electronic archive data based on business elements and a new electronic archive knowledge graph, and to obtain classification results and corresponding cataloging information.

9. An electronic device, characterized in that, The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images