A RAG multimodal document parsing method, device and medium

Abstract

The embodiments of this specification disclose a RAG multimodal document parsing method, device, and medium, which relate to the field of natural language processing technology and are used to solve the problem of low processing power of existing parsing methods. The method includes: identifying the document type of the multimodal document, and assigning the parsing task of the multimodal document to the corresponding Kubernetes task node based on the document type; adjusting the parsing process of each Kubernetes task node based on the parsing requirements of the multimodal document to execute the parsing task of the multimodal document based on the updated parsing process to obtain parsed data; preprocessing the parsed data, semantically slicing the processed parsed data in a dynamic text windowing method based on a preset large language model, and obtaining semantic blocks corresponding to the processed parsed data; converting the semantic blocks into embedded vectors and storing them in a preset vector database for rapid retrieval based on the preset vector database.

Description

Technical Field

[0001] This specification relates to the field of natural language processing technology, and in particular to a RAG multimodal document parsing method, device, and medium. Background Art

[0002] In the fields of artificial intelligence and natural language processing, Retrieval-Augmented Generation (RAG) technology significantly improves the processing capabilities of knowledge-intensive tasks by combining retrieval systems with generative models. With the advancement of natural language processing, existing RAG systems need to process multimodal documents such as PDFs, tables, images, audio, and video. Parsing multimodal documents has become a critical step in the processing process.

[0003] The current parsing complexity of multimodal documents is relatively high, and it is difficult to efficiently extract structured information such as text, tables, codes, and formulas based on traditional parsing methods. Secondly, existing parsing methods lack the ability to uniformly process multimodal data and cannot simultaneously support the parsing and retrieval of text, images, audio, and video, resulting in low data utilization. In addition, the scalability and resource utilization of the system in existing parsing methods are insufficient, making it difficult to cope with high-concurrency requests and large-scale document processing needs. Especially in the process of document parsing and segmentation, the computing resources are unevenly distributed, which can easily become a performance bottleneck. Moreover, traditional RAG systems lack dynamism in document updates and knowledge base maintenance. The update process usually requires downtime or manual intervention, affecting the availability and real-time performance of the system. Summary of the Invention

[0004] In order to solve the above technical problems, one or more embodiments of this specification provide a RAG multimodal document parsing method, device and medium.

[0005] One or more embodiments of this specification adopt the following technical solutions:

[0006] One or more embodiments of this specification provide a RAG multimodal document parsing method, the method comprising:

[0007] Acquire a multimodal document uploaded by a user terminal based on a multimodal document parsing system, and identify a document type of the multimodal document based on a classification model;

[0008] Allocating the parsing task of the multimodal document to a corresponding Kubernetes task node based on the document type; wherein each of the Kubernetes task nodes has a corresponding parser;

[0009] Based on the parsing requirements of the multimodal document, the parsing process of each Kubernetes task node is adjusted to execute the parsing task of the multimodal document based on the updated parsing process to obtain parsed data;

[0010] Preprocessing the parsed data, semantically slicing the processed parsed data in a dynamic text windowing manner based on a preset large language model, and obtaining semantic blocks corresponding to the processed parsed data;

[0011] The semantic blocks are converted into embedding vectors and stored in a preset vector database so as to be quickly retrieved based on the preset vector database.

[0012] Optionally, in one or more embodiments of this specification, before obtaining the multimodal document uploaded by the user, the method further includes:

[0013] Installing and configuring a container orchestration environment of a Kubernetes cluster on the multimodal document parsing system, and installing and initializing a pre-configured distributed storage system to implement environment initialization of the multimodal document parsing system;

[0014] Obtaining metadata attributes corresponding to the document upload task created by the user end, so as to automatically generate a storage path based on the metadata attributes;

[0015] The multimodal document is sharded based on a preset shard size, so that each shard is uploaded in parallel to the preset distributed storage system based on the storage path, and a unique document identifier corresponding to the multimodal document is generated; wherein the preset distributed storage system is a MinIO distributed storage system.

[0016] Optionally, in one or more embodiments of this specification, identifying the document type of the multimodal document based on a deep learning model specifically includes:

[0017] Determining the document domain corresponding to the multimodal document to collect public multimodal documents corresponding to the document domain, and fine-tuning an existing deep learning model based on the public multimodal documents and a ResNet-50 architecture to obtain a corresponding classification model;

[0018] If it is determined that the multimodal document has been stored in the preset distributed storage system, triggering a classification task for the multimodal document based on the preset distributed storage system to extract corresponding preamble content within the multimodal document, so as to use the preamble content as classification basis content for the classification task;

[0019] Performing format unification processing on the classification basis content, so as to send the processed classification basis content to the classification model based on a preset protocol, and outputting a type prediction result of the multimodal document;

[0020] If the confidence corresponding to the type prediction result is greater than the preset confidence threshold, the type prediction result is used as the document type; if the confidence is less than the preset confidence threshold, a manual review process is triggered to determine the document type of the multimodal document.

[0021] Optionally, in one or more embodiments of the present specification, extracting corresponding preceding content in the multimodal document to use the preceding content as classification basis content for the classification task specifically includes:

[0022] Determining constraints for extracting content from the multimodal document based on historical classification task records in the document domain; wherein the constraints include: delay budget and computing resource limitations;

[0023] Determining key information distribution information of different types of documents based on the historical parsing data corresponding to the historical classification task records;

[0024] Determining a key information concentration area corresponding to the multimodal document based on key information distribution information of the different types of documents;

[0025] Based on the constraint conditions and the key information concentration area, defining an objective function corresponding to the current extracted content;

[0026] The objective function is subjected to simulated annealing within the analyzable capacity of the classification model to iteratively determine the size of the front content that should be extracted from the multimodal document.

[0027] Optionally, in one or more embodiments of this specification, allocating the parsing task of the multimodal document to a corresponding Kubernetes task node based on the document type specifically includes:

[0028] Obtain a scalable scheduler framework for the Kubernetes cluster, perform secondary development on the scalable scheduler framework based on the corresponding parsing scenario on the user side, and build a scheduling decision engine for the parsing task;

[0029] Receive batch parsing tasks created by the user, generate parsing tasks for each multimodal document based on the document type and storage path of each multimodal document corresponding to the batch parsing task, and submit each parsing task to the task queue of Kubernetes;

[0030] Obtaining indicator data for each task node in the Kubernetes cluster to evaluate the resource status of each task node in the Kubernetes cluster based on the indicator data;

[0031] Based on the scheduling decision engine, each parsing task in the task queue is obtained, and the parsing tasks are scheduled based on the average parsing resources corresponding to each document type and the resource status of each node to determine the Kubernetes task node corresponding to each parsing task.

[0032] Optionally, in one or more embodiments of this specification, based on the parsing requirements of the multimodal document, adjusting the parsing process of each of the Kubernetes task nodes to perform the parsing task of the multimodal document based on the updated parsing process to obtain parsed data specifically includes:

[0033] Determining, based on the parsing task of the multimodal document uploaded by the user, whether the parsing task has custom parsing content;

[0034] If not, obtain the fixed parsing process corresponding to each document type in each Kubernetes task node, and perform the parsing task of the multimodal document based on the fixed parsing process to obtain parsed data;

[0035] If so, obtaining the parsing operation required for the custom parsing content, and determining whether the parsing operation conflicts with the fixed parsing process;

[0036] If yes, determining the conflict type of the conflict, and updating the fixed parsing process based on the conflict type to add the parsing operation to obtain an updated parsing process;

[0037] If not, determining a dependent operation corresponding to the parsing operation, and adding the parsing operation to the fixed parsing process based on the dependent operation to obtain an updated parsing process;

[0038] The parsing task of the multimodal document is executed based on the updated parsing process to obtain parsed data.

[0039] Optionally, in one or more embodiments of the present specification, the parsed data is preprocessed, and semantic slicing is performed on the processed parsed data in a dynamic text windowing manner based on a preset large language model to obtain semantic blocks corresponding to the processed parsed data, specifically including:

[0040] Performing encoding detection on the parsed data based on a preset encoding library, and executing a text cleaning pipeline to clean the parsed data after detection to obtain cleaned parsed data, and performing dependency syntactic analysis on the cleaned parsed data to achieve sentence segmentation to obtain processed parsed data;

[0041] Initialize the parameters of the dynamic text windowing method to window the processed parsed data based on the initialization parameters;

[0042] Determine the perplexity mutation points of each processed parsed data within the window through semantic boundary detection, determine the semantic boundary based on a preset number of consecutive perplexity mutation points, and determine the initial semantic block corresponding to the processed parsed data based on the semantic boundary;

[0043] Based on the similarity between adjacent initial semantic blocks, the initial semantic blocks are merged to obtain merged initial semantic blocks;

[0044] A semantic relationship graph corresponding to the merged initial semantic blocks is constructed to perform clustering optimization on the semantic relationship graph, and the semantic blocks corresponding to the processed parsed data are determined based on the optimized semantic relationship graph and a preset rule engine.

[0045] Optionally, in one or more embodiments of the present specification, the semantic blocks are converted into embedded vectors and stored in a preset vector database so as to enable fast retrieval based on the preset vector database, specifically including:

[0046] Preprocessing the semantic blocks based on a preset index module, converting the preprocessed semantic blocks into embedding vectors based on a preset large language model, and storing the embedded vectors in a preset vector database;

[0047] receiving user query content and converting the query content into a query embedding vector based on a preset large language model;

[0048] Based on the similarity between the query embedding vector and each embedding vector in the preset vector database, the closest semantic block is retrieved.

[0049] One or more embodiments of this specification provide a RAG multimodal document parsing device, including:

[0050] at least one processor; and,

[0051] a memory communicatively connected to the at least one processor; wherein,

[0052] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: perform any of the above methods.

[0053] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute any of the above-described methods.

[0054] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0055] The classification model is used to quickly and accurately identify the document type of multimodal documents, and based on this, the parsing tasks are assigned to the corresponding Kubernetes task nodes, realizing the automatic allocation and parallel processing of tasks, improving the efficiency of the entire parsing process, and being able to handle multiple different types of document parsing tasks at the same time, reducing processing time. The parsing process of the Kubernetes task node can be adjusted according to the parsing requirements of the multimodal documents. This flexibility enables the system to adapt to changes in different document types and parsing requirements. Using the dynamic text windowing method of the preset large language model for semantic slicing can better understand the semantic structure of the text and divide the parsed data into meaningful semantic blocks, which helps to deeply explore the semantic information in the document. The semantic blocks are converted into embedded vectors and stored in the preset vector database, which facilitates rapid retrieval based on the vector database and improves retrieval efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some of the embodiments described in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without inventive work. In the drawings:

[0057] Figure 1 A schematic diagram of a method flow of a RAG multimodal document parsing method provided in an embodiment of this specification;

[0058] Figure 2 A schematic diagram of the structure of a RAG multimodal document parsing device provided in an embodiment of this specification;

[0059] Figure 3 A schematic diagram of the structure of a non-volatile storage medium provided in an embodiment of this specification. DETAILED DESCRIPTION

[0060] The embodiments of this specification provide a RAG multimodal document parsing method, device, and medium.

[0061] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this specification without creative work should fall within the scope of protection of this specification.

[0062] like Figure 1 As shown, the embodiment of this specification provides a method flow diagram of a RAG multimodal document parsing method. Figure 1 It can be seen that in one or more embodiments of this specification, a RAG multimodal document parsing method includes:

[0063] S101: A multimodal document uploaded by a user is obtained based on a multimodal document parsing system to identify the document type of the multimodal document based on a classification model.

[0064] The multimodal document parsing system receives multimodal documents uploaded by users through the user interface. These documents may contain multiple data formats such as text, images, audio, and video. To facilitate the parsing of multimodal documents, it is necessary to identify the document type based on a classification model.

[0065] Furthermore, in one or more embodiments of this specification, before obtaining the multimodal document uploaded by the user, the method further includes the following process:

[0066] First, a Kubernetes cluster container orchestration environment is installed and configured on the multimodal document parsing system. A pre-configured distributed storage system is also installed and initialized to initialize the multimodal document parsing environment. Specifically, in certain scenarios, a Kubernetes cluster or other container orchestration environment is installed and configured, and the initialization system and other middleware, such as Minio, Elasticsearch, PostgreSQL, and Argo Workflow, are installed. Next, when a user creates a document upload task on the multimodal document parsing system, the system retrieves the metadata attributes corresponding to the task. Specifically, when a user creates a document upload task through the system, PostgreSQL records metadata attributes such as the file's MD5, size, and upload time. Based on these metadata attributes, a storage path, such as " / tenant_id / year / month / day / uuid_filename.ext," is automatically generated. The multimodal document is then sharded based on the pre-configured shard size. Each shard is then uploaded to the pre-configured distributed storage system in parallel according to the storage path, and a unique document identifier is generated for each multimodal document. The pre-configured distributed storage system is the MinIO distributed storage system. The use of the MinIO distributed storage system in this process ensures high data availability and fault tolerance. Even if some storage nodes fail, data is not lost and can still be accessed and processed normally, providing reliable protection for the long-term storage and subsequent use of multimodal documents. Furthermore, document sharding ensures stable and efficient system operation when large numbers of multimodal documents are uploaded, eliminating performance bottlenecks caused by excessive file numbers or file sizes.

[0067] Specifically, in one or more embodiments of this specification, identifying the document type of a multimodal document based on a deep learning model specifically includes:

[0068] Determine the document domain corresponding to the multimodal document to collect public multimodal documents corresponding to the document domain, so as to fine-tune the existing deep learning model according to the public multimodal documents and the ResNet-50 architecture to obtain a corresponding classification model. Through fine-tuning, the model learns the characteristics of different types of documents in a specific document domain, thereby obtaining a classification model for multimodal documents in this domain, making it more adaptable to specific classification task requirements. If it is determined that the multimodal document has been stored in a preset distributed storage system, the classification task of the multimodal document is triggered based on the preset distributed storage system to extract the corresponding pre-content in the multimodal document, for example, the first 1MB of document content will be extracted, and the pre-content will be used as the classification basis content for the classification task. Then, the classification basis content is formatted in a unified manner, so that the processed classification basis content is sent to the classification model based on the preset protocol, and the type prediction result of the multimodal document is output. If the confidence level of the prediction result is greater than the preset confidence level, the prediction result is used as the document type. If the confidence level is less than the preset confidence level, a manual review process is triggered to determine the document type of the multimodal document.

[0069] During this process, the model is fine-tuned using publicly available multimodal documents from specific fields, making the classification model more relevant to actual application scenarios, better capturing the subtle features of different document types, and improving classification accuracy. Compared to general models, models optimized for specific fields can more accurately identify document types. Furthermore, the system automatically triggers classification tasks and makes predictions based on the model, improving classification efficiency and automating document type identification. At the same time, manual review is introduced for cases where model confidence is low, compensating for potential model deficiencies, ensuring the reliability of classification results, and achieving a balance between efficiency and accuracy.

[0070] Furthermore, in one or more embodiments of this specification, extracting corresponding preceding content in a multimodal document to use the preceding content as classification basis for a classification task specifically includes:

[0071] First, we refer to historical classification task records in the document domain to analyze and identify the constraints that must be followed when extracting content from multimodal documents. It is important to note that these constraints include delay budgets and computational resource limits. The delay budget refers to the maximum time allowed from the start of content extraction to the completion of the classification task, ensuring system responsiveness and user experience. Computational resource limits refer to the available computing resources, such as CPU and memory, for content extraction and classification tasks, ensuring successful completion despite resource constraints. Based on the historical parsing data corresponding to historical classification task records, we conduct a detailed analysis of different document types to identify the distribution patterns of key information within each document. For example, in some technical documents, key information may be concentrated in the initial summary and the core technology section in the middle; in news reports, key information is often found in the initial introduction. Combining the previously obtained key information distribution information for different document types, we determine the key information concentration areas for the current multimodal document. Based on the constraints and the key information concentration area, define the objective function corresponding to the current extracted content. For example, define the objective function as follows: processing delay ≤ delay budget, resource consumption ≤ resource limit, key information coverage ≥ preset threshold. Within the analyzable capacity of the classification model, use the simulated annealing algorithm to optimize the objective function to iteratively determine the size of the front content that should be extracted from the multimodal document.

[0072] S102: Allocate the parsing task of the multimodal document to a corresponding Kubernetes task node based on the document type; wherein each of the Kubernetes task nodes has a corresponding parser.

[0073] Multimodal documents contain multiple data forms, and the parsing methods for different types of documents vary greatly. Without a targeted parsing mechanism, it is difficult for a general parser to process all types of documents efficiently and accurately. Therefore, in the multimodal document parsing system in the embodiments of this specification, the type of multimodal document is first identified using a deep learning model or other classification methods. The Kubernetes task node is the working unit of the multimodal document parsing system in the Kubernetes cluster, and each task node is equipped with a corresponding parser. These parsers are specially designed to process specific types of documents. When the system identifies the type of multimodal document, it will send the parsing task to the corresponding Kubernetes task node according to the preset allocation rules. If it is a PDF document, the parsing task will be assigned to the task node with a PDF parser; if it is an audio document, it will be assigned to a node equipped with an audio parser. In this way, each task node can focus on processing the document type it is good at, thereby improving the efficiency and accuracy of parsing.

[0074] Specifically, in one or more embodiments of this specification, assigning the parsing task of a multimodal document to a corresponding Kubernetes task node based on the document type specifically includes the following process:

[0075] Parsing different document types requires significantly different resources, and the general scheduling strategies of the native Kubernetes scheduler are insufficient to meet the specific needs of multimodal document parsing. Therefore, we first developed a scalable scheduler framework for the Kubernetes cluster, which provides the foundation for subsequent development. Based on the user's specific parsing scenarios, we further developed the framework to build a scheduling decision engine suitable for multimodal document parsing tasks. For example, if the user primarily processes multimodal documents in the financial sector, including large numbers of PDF contracts and Excel spreadsheets, the scheduling decision engine should be developed specifically for the parsing characteristics of these documents. After receiving batch parsing tasks created by the user, the system generates corresponding parsing tasks based on the document type and storage path of each multimodal document and submits each task to the Kubernetes task queue for scheduling. Next, we obtain metrics for each task node in the Kubernetes cluster, including CPU usage, memory usage, and disk I / O rate. By analyzing these metrics, the system can comprehensively assess the resource status of each task node, understanding the current workload of each node and the remaining resources available for new tasks.

[0076] Based on the scheduling decision engine, each parsing task in the task queue is obtained, and the parsing tasks are scheduled based on the average parsing resources corresponding to each document type and the resource status of each node, and the Kubernetes task node corresponding to each parsing task is determined. In this process, the extensible scheduler framework is secondary developed based on the user's parsing scenario, and the constructed scheduling decision engine can accurately match user needs. In different industries and different business scenarios, the types and parsing requirements of multimodal documents vary greatly. The customized scheduler can flexibly adjust the scheduling strategy to improve the adaptability and efficiency of scheduling. By obtaining node indicator data in real time to evaluate resource status and combining the average parsing resource requirements of the document type for task scheduling, tasks can be assigned to the most appropriate nodes, avoiding resource waste and node overload, and improving the resource utilization of the entire Kubernetes cluster.

[0077] In certain application scenarios, the process of assigning parsing tasks to corresponding Kubernetes task nodes can also be as follows: A scheduling decision engine is developed using the Kubernetes Scheduler Framework, queue management is implemented based on Kueue, and node metrics are collected in real time using Prometheus. The user clicks to create a batch document parsing task, which retrieves document information from PostgreSQL and enters the task queue. The scheduling decision engine retrieves the task from the task queue and then uses Prometheus to perform real-time resource evaluation. It also implements a multi-dimensional scheduling strategy for the task. For example, if sufficient resources are determined for PDF file parsing, five documents will be assigned to each node job. If resources are insufficient, resources are expanded through HPA.

[0078] S103: Based on the parsing requirements of the multimodal document, adjust the parsing process of each of the Kubernetes task nodes to execute the parsing task of the multimodal document based on the updated parsing process to obtain parsed data.

[0079] In a multimodal document parsing system, different multimodal documents and different application scenarios will generate diverse parsing requirements. However, the traditional fixed parsing process is difficult to adapt to the diverse parsing requirements of multimodal documents in different scenarios. Therefore, in the embodiments of this specification, the parsing process of each Kubernetes task node will be adjusted according to the parsing requirements of the multimodal document, so as to execute the parsing task of the multimodal document based on the updated parsing process and obtain parsed data.

[0080] Specifically, in one or more embodiments of this specification, based on the parsing requirements of multimodal documents, the parsing process of each Kubernetes task node is adjusted to execute the multimodal document parsing task based on the updated parsing process to obtain parsed data, specifically including:

[0081] First, based on the parsing task for the multimodal document uploaded by the user, the system determines whether the task has custom parsing content. If not, the system retrieves the fixed parsing processes pre-configured for different document types in each Kubernetes task node. The system then parses the multimodal document according to these fixed parsing processes, ultimately generating parsed data.

[0082] For example, when the Kubernetes task node is a PDF parser: First, extract the original text, use Apache PDFBox to parse the native text layer of the PDF, and retain metadata such as fonts and positions; then use the.DeepDoc text purification module: Repair garbled characters (such as "##" → "company") based on rules + BiLSTM model; finally, output text blocks with coordinates (each block contains: x / y coordinates, font size, semantic tags). After that, process scanned documents, use the Tesseract 5.0 OCR engine (LSTM + Attention model) to process image pages; then use the deepdoc DRF-based version for analysis to distinguish the main text, headers, footers, etc.; perform re-detection of table areas to avoid incorrect OCR recognition; finally, perform image content understanding, call the CLIP-ViT-L / 14 model to generate image descriptions, and use Faster R-CNN to detect key elements in the image (such as logos / charts); finally, match the descriptive text based on spatial coordinates to perform text-image association.

[0083] When the Kubernetes task node is a table parser: Perform table detection. First, use the CascadeTabNet model to locate the table area (output a rotated rectangle box); then use a sequence classifier based on Transformer (to determine whether it is a real table); finally, filter decorative borders (accuracy 99.2%). Structure reconstruction: First, use the Hough transform of OpenCV to detect lines and use morphological processing to repair broken borders; then use the deepdoc cell merging algorithm based on Graph Neural Network to identify跨行 / 列 cells; dynamically adjust the weights of dividing lines (to handle dotted / lighter borders). Finally, perform content extraction: First, use PDFBox to extract the native table text, and then use DeepDoc table semantic annotation: Identify table headers / data cells (BERT + CRF) and infer cell relationships (such as "same as above" → automatic filling).

[0084] When the Kubernetes task node is an audio parser: First, perform speech recognition, perform automatic language detection through the Whisper-large-v3 model (supporting 50+ languages), and separate speakers (based on voiceprint clustering); then perform domain term correction (medical / legal and other professional dictionaries) and timestamp calibration (MFCC feature alignment) based on deepdoc.

[0085] When the Kubernetes task node is a video parser: First, use the FFmpeg frame extraction strategy to extract frames, use the key frame priority strategy for dynamic sampling. Use DINOv2 feature extraction to generate 1024-dimensional visual embeddings for each frame, and then perform key frame screening based on DeepDoc (based on semantic importance scoring).

[0086] If the parsing task is determined to contain custom parsing content, the system obtains the parsing operations required for the custom parsing content and determines whether the parsing operations conflict with the fixed parsing process. Conflicts may manifest as inconsistencies in the order of parsing operations, resource usage, and other aspects. For example, a custom operation must be completed before a certain step, but that step is already completed after other operations in the fixed process. If a conflict exists, the system determines the type of conflict, such as order conflict or resource conflict. Based on the conflict type, the fixed parsing process is updated, and the custom parsing operations are appropriately added to the process to resolve the conflict and obtain an updated parsing process. For example, if the conflict is an order conflict, the execution order of certain steps may need to be adjusted. If the parsing operation does not conflict with the fixed parsing process, the system determines the dependent operations corresponding to the parsing operation, namely, other operations that must be completed before executing the parsing operation. Based on these dependent operations, the parsing operation is added to the fixed parsing process, forming an updated parsing process. The multimodal document parsing task is then executed based on the updated parsing process to obtain parsed data.

[0087] S104: Preprocessing the parsed data, semantically slicing the processed parsed data in a dynamic text windowing manner based on a preset large language model, and obtaining semantic blocks corresponding to the processed parsed data.

[0088] Traditional text segmentation methods often use fixed lengths or simple rules, making it difficult to accurately capture the semantic boundaries of text, resulting in incomplete or incoherent semantics after segmentation. Therefore, in the embodiments of this specification, the parsed data is preprocessed, and the processed parsed data is semantically sliced ​​using a dynamic text windowing method based on a pre-set large language model to obtain semantic segments corresponding to the processed parsed data.

[0089] Specifically, in one or more embodiments of the present specification, preprocessing the parsed data, semantically slicing the processed parsed data using a dynamic text windowing method based on a preset large language model, and obtaining semantic blocks corresponding to the processed parsed data specifically includes the following steps:

[0090] Use a pre-built encoding library (such as chardet) to perform encoding detection on the parsed data. Then, run a text cleaning pipeline on the parsed data to clean it. This cleansed data is then subjected to dependency parsing and sentence segmentation to obtain processed parsed data. The text cleaning pipeline includes operations such as removing invisible control characters (characters with ASCII values ​​less than 32), standardizing quotes and hyphens, and merging consecutive whitespace.

[0091] Initialize the parameters of the dynamic text windowing method, setting parameters such as the initial window size (e.g., 256 tokens), the minimum valid block size (e.g., 64 tokens), and the overlap size between adjacent windows (e.g., 64 tokens). Based on these initial parameters, each processed parsed data is windowed, dividing the text into multiple windows. Each window contains a certain amount of text content, and adjacent windows partially overlap to ensure the coherence of contextual information. Semantic boundary detection is then used to calculate the perplexity breakpoints of the processed parsed data within each window. Perplexity breakpoints reflect changes in the text's semantics. Semantic boundaries are identified when a preset number (e.g., three) of these perplexity breakpoints appear consecutively and meet certain significance criteria (e.g., P < 0.01). Based on these semantic boundaries, the processed parsed data is divided into initial semantic blocks, each with relatively independent semantics. The similarity between adjacent initial semantic blocks is then calculated. If the cosine similarity is greater than 0.85 and the temporal continuity is less than 5 seconds, the initial semantic blocks are merged to obtain merged initial semantic blocks.

[0092] A semantic relationship graph corresponding to the merged initial semantic segments is constructed to optimize clustering of the semantic relationship graph. Based on the optimized semantic relationship graph and the pre-configured rule engine, the semantic segments corresponding to the processed parsed data are determined. In certain scenarios, segmentation quality can be optimized by combining graph structure representation and the Louvain community discovery algorithm with rule-based post-processing. First, a semantic relationship graph is constructed, where nodes represent candidate segments and edge weights are determined by the coreference resolution scores calculated by the Coreferee tool. Subsequently, community segmentation is performed and clustering optimization is performed using the Louvain method. With the resolution parameter γ = 1.25, iterative optimization is performed until modularity Q > 0.6. Finally, post-processing is performed using the rule engine, including forced segmentation of long paragraphs (e.g., those with more than 512 tokens), merging of short paragraphs (e.g., those with less than 32 tokens and a similarity greater than 0.9), and processing of dialogue scenarios, to ensure segmentation consistency and rationality.

[0093] In this process, dynamic text windowing combined with semantic boundary detection can accurately capture the semantic changes of the text and divide the text into blocks with clear semantic boundaries. This method can better reflect the semantic structure of the text than traditional fixed-length blocks, and improves the accuracy of semantic understanding. In addition, the initial semantic block merging step merges semantically similar and coherent blocks through similarity calculation and temporal continuity judgment, avoiding the semantic incoherence problem caused by overly fragmented blocks, making the final semantic block more consistent with the logical structure of the text. In addition, the construction and clustering optimization of the semantic relationship graph, as well as the use of a preset rule engine, enable this process to adapt to different types of text and diverse analysis needs.

[0094] S105: Convert the semantic blocks into embedded vectors and store them in a preset vector database so as to enable fast retrieval based on the preset vector database.

[0095] Traditional text retrieval methods are primarily based on keyword matching, making it difficult to understand the semantics of the text. They are also less effective for synonyms, near-synonyms, and texts with similar semantics but different expressions. Therefore, in the embodiments of this specification, semantic segments are converted into embedded vectors and retrieved using a vector database, enabling matching at the semantic level and significantly improving retrieval accuracy and recall. Semantic segments are then converted into embedded vectors and stored in a pre-set vector database, allowing for rapid retrieval based on the pre-set vector database.

[0096] Specifically, in one or more embodiments of this specification, semantic blocks are converted into embedded vectors and stored in a preset vector database so as to enable fast retrieval based on the preset vector database, specifically including:

[0097] The semantic segments are preprocessed based on a preset index module, such as text standardization and adding domain prefixes. The preprocessed semantic segments are then converted into embedding vectors based on a preset large language model and stored in a preset vector database. During this process, batch requests can be constructed and then written to the repository in parallel using multiple threads. The user's query content is then received and converted into a query embedding vector based on the preset large language model. Based on the similarity between the query embedding vector and each embedding vector in the preset vector database, the closest semantic segment is retrieved. This vector-based retrieval method significantly improves the speed and accuracy of semantic retrieval.

[0098] like Figure 2 As shown in the embodiment of this specification, a schematic diagram of the structure of a RAG multimodal document parsing device is provided, which comprises Figure 2 It can be seen that in one or more embodiments of this specification, a RAG multimodal document parsing device includes:

[0099] at least one processor; and,

[0100] a memory communicatively connected to the at least one processor; wherein,

[0101] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: perform any of the above methods.

[0102] like Figure 3 As shown in FIG, the embodiment of this specification provides a structural diagram of a non-volatile storage medium. Figure 3It can be seen that in one or more embodiments of this specification, a non-volatile storage medium stores computer-executable instructions 301, and the computer-executable instructions 301 can execute any of the above-described methods.

[0103] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For relevant details, refer to the descriptions of the method embodiments.

[0104] The foregoing description is of specific embodiments of the present disclosure. Other embodiments are within the scope of the accompanying description. In some cases, the actions or steps described in the description may be performed in an order different from that described in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0105] The foregoing is merely one or more embodiments of this specification and is not intended to limit this specification. It will be apparent to those skilled in the art that various modifications and variations may be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of one or more embodiments of this specification are intended to be within the scope of this specification.

Claims

1. A RAG multimodal document parsing method, characterized in that: The method comprises: Acquire a multimodal document uploaded by a user terminal based on a multimodal document parsing system, and identify a document type of the multimodal document based on a classification model; Allocating the parsing task of the multimodal document to a corresponding Kubernetes task node based on the document type; wherein each of the Kubernetes task nodes has a corresponding parser; Based on the parsing requirements of the multimodal document, the parsing process of each Kubernetes task node is adjusted to execute the parsing task of the multimodal document based on the updated parsing process to obtain parsed data; Preprocessing the parsed data, semantically slicing the processed parsed data in a dynamic text windowing manner based on a preset large language model, and obtaining semantic blocks corresponding to the processed parsed data; Converting the semantic blocks into embedded vectors and storing them in a preset vector database for fast retrieval based on the preset vector database; Before obtaining the multimodal document uploaded by the user, the method further includes: Installing and configuring a container orchestration environment of a Kubernetes cluster on the multimodal document parsing system, and installing and initializing a pre-configured distributed storage system to implement environment initialization of the multimodal document parsing system; Obtaining metadata attributes corresponding to the document upload task created by the user end, so as to automatically generate a storage path based on the metadata attributes; Slicing the multimodal document based on a preset slicing size, uploading the slicing portions in parallel to the preset distributed storage system based on the storage path, and generating a unique document identifier corresponding to the multimodal document; wherein the preset distributed storage system is a MinIO distributed storage system; Allocating the parsing task of the multimodal document to the corresponding Kubernetes task node based on the document type specifically includes: Obtain a scalable scheduler framework for the Kubernetes cluster, perform secondary development on the scalable scheduler framework based on the corresponding parsing scenario on the user side, and build a scheduling decision engine for the parsing task; Receive batch parsing tasks created by the user, generate parsing tasks for each multimodal document based on the document type and storage path of each multimodal document corresponding to the batch parsing task, and submit each parsing task to the task queue of Kubernetes; Obtaining indicator data for each task node in the Kubernetes cluster to evaluate the resource status of each task node in the Kubernetes cluster based on the indicator data; Obtain each parsing task in the task queue based on the scheduling decision engine, schedule the parsing tasks based on the average parsing resources corresponding to each document type and the resource status of each node, and determine the Kubernetes task node corresponding to each parsing task; If it is determined that the multimodal document has been stored in the preset distributed storage system, triggering a classification task for the multimodal document based on the preset distributed storage system to extract corresponding preamble content within the multimodal document, so as to use the preamble content as classification basis content for the classification task; Extracting corresponding preceding content in the multimodal document to use the preceding content as classification basis content for the classification task specifically includes: Determining constraints for extracting content from the multimodal document based on historical classification task records in the document field; wherein the constraints include: delay budget and computing resource limitations; Determining key information distribution information of different types of documents based on the historical parsing data corresponding to the historical classification task records; Determining a key information concentration area corresponding to the multimodal document based on key information distribution information of the different types of documents; Based on the constraint conditions and the key information concentration area, defining an objective function corresponding to the current extracted content; The objective function is subjected to simulated annealing within the analyzable capacity of the classification model to iteratively determine the size of the front content that should be extracted from the multimodal document.

2. A RAG multimodal document parsing method according to claim 1, characterized in that: Identifying the document type of the multimodal document based on a deep learning model specifically includes: Determining the document domain corresponding to the multimodal document to collect public multimodal documents corresponding to the document domain, and fine-tuning an existing deep learning model based on the public multimodal documents and a ResNet-50 architecture to obtain a corresponding classification model; Performing format unification processing on the classification basis content, so as to send the processed classification basis content to the classification model based on a preset protocol, and outputting a type prediction result of the multimodal document; If the confidence corresponding to the type prediction result is greater than the preset confidence threshold, the type prediction result is used as the document type; if the confidence is less than the preset confidence threshold, a manual review process is triggered to determine the document type of the multimodal document.

3. A RAG multimodal document parsing method according to claim 1, characterized in that: Based on the parsing requirements of the multimodal document, the parsing process of each Kubernetes task node is adjusted to execute the parsing task of the multimodal document based on the updated parsing process to obtain parsed data, specifically including: Determining, based on the parsing task of the multimodal document uploaded by the user, whether the parsing task has custom parsing content; If not, obtain the fixed parsing process corresponding to each document type in each Kubernetes task node, and perform the parsing task of the multimodal document based on the fixed parsing process to obtain parsed data; If so, obtaining the parsing operation required for the custom parsing content, and determining whether the parsing operation conflicts with the fixed parsing process; If yes, determining the conflict type of the conflict, and updating the fixed parsing process based on the conflict type to add the parsing operation to obtain an updated parsing process; If not, determining a dependent operation corresponding to the parsing operation, and adding the parsing operation to the fixed parsing process based on the dependent operation to obtain an updated parsing process; The parsing task of the multimodal document is executed based on the updated parsing process to obtain parsed data.

4. A RAG multimodal document parsing method according to claim 1, characterized in that: Preprocessing the parsed data, semantically slicing the processed parsed data in a dynamic text windowing manner based on a preset large language model, and obtaining semantic blocks corresponding to the processed parsed data, specifically including: Performing encoding detection on the parsed data based on a preset encoding library, and executing a text cleaning pipeline to clean the parsed data after detection to obtain cleaned parsed data, and performing dependency syntactic analysis on the cleaned parsed data to achieve sentence segmentation to obtain processed parsed data; Initialize the parameters of the dynamic text windowing method to window the processed parsed data based on the initialization parameters; Determine the perplexity mutation points of each processed parsed data within the window through semantic boundary detection, determine the semantic boundary based on a preset number of consecutive perplexity mutation points, and determine the initial semantic block corresponding to the processed parsed data based on the semantic boundary; Based on the similarity between adjacent initial semantic blocks, the initial semantic blocks are merged to obtain merged initial semantic blocks; A semantic relationship graph corresponding to the merged initial semantic blocks is constructed to perform clustering optimization on the semantic relationship graph, and the semantic blocks corresponding to the processed parsed data are determined based on the optimized semantic relationship graph and a preset rule engine.

5. A RAG multimodal document parsing method according to claim 1, characterized in that: Converting the semantic blocks into embedded vectors and storing them in a preset vector database for fast retrieval based on the preset vector database, specifically including: Preprocessing the semantic blocks based on a preset index module, converting the preprocessed semantic blocks into embedding vectors based on a preset large language model, and storing the embedded vectors in a preset vector database; receiving user query content and converting the query content into a query embedding vector based on a preset large language model; Based on the similarity between the query embedding vector and each embedding vector in the preset vector database, the closest semantic block is retrieved.

6. A RAG multimodal document parsing device, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: execute the method according to any one of claims 1 to 5.

7. A non-volatile storage medium storing computer-executable instructions, characterized in that: The computer executable instructions can execute the method according to any one of claims 1 to 5.

Images