Document analysis method and device, storage medium and electronic device

Abstract

The invention provides a document analysis method and device, a storage medium and an electronic device.The method comprises the steps that an initial document is segmented into a plurality of file fragments according to the document type of the initial document to be analyzed, and the data capacity of the file fragments is smaller than the preset capacity; according to the resource historical utilization rate and the document type, analyzing resources of a plurality of file fragments are distributed, the analyzing resources comprise a plurality of analyzing nodes with document analyzing capacity, and each analyzing node comprises a plurality of assembly line pipelines which are connected in series; and extracting document content slices of the plurality of file fragments by adopting the analysis resources, quantitatively calculating vector data of the document content slices according to slice content formats of the document content slices, and storing the vector data according to the slice content formats of the document content slices. According to the embodiment of the invention, the analysis efficiency maximization and the resource utilization rate optimization are realized on the whole system, and meanwhile, the analysis efficiency and the storage granularity of the document are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a document parsing method and apparatus, storage medium, and electronic device. Background Technology

[0002] In related technologies, with the rapid development of information technology, a large amount of unstructured document data is constantly emerging. How to efficiently parse and structure this data has become crucial for improving enterprise knowledge utilization efficiency and reducing repetitive work. In recent years, the application of large-scale modeling technology has brought new breakthroughs to knowledge base construction, significantly improving the accuracy of knowledge integration and intelligent question answering.

[0003] Existing document knowledge parsing technologies largely rely on single-modal processing methods or layout analysis methods based on lightweight models. The parsing of text, tables, images, and formulas suffers from fragmentation and localization, making it difficult to efficiently segment, distribute, and quantize large documents. Furthermore, the lack of a unified multimodal vectorized storage strategy limits efficiency and accuracy in knowledge retrieval, content fusion, and downstream applications. In addition, existing technologies in large document processing and task scheduling often rely on sequential processing or fixed resource allocation, lacking dynamic distributed scheduling and load balancing mechanisms. This fails to fully utilize computing resources and guarantee the real-time performance and stability of parsing.

[0004] No efficient and accurate solution has yet been found to address the aforementioned issues in the relevant technologies. Summary of the Invention

[0005] This invention provides a document parsing method and apparatus, storage medium, and electronic device to solve technical problems in related technologies.

[0006] According to an embodiment of the present invention, a document parsing method is provided, comprising: dividing the initial document into several file segments according to the document type of the initial document to be parsed, wherein the data capacity of the file segments is less than a preset capacity; allocating parsing resources for the several file segments according to historical resource utilization and the document type, wherein the parsing resources include multiple parsing nodes with document parsing capabilities, each parsing node including multiple pipelines connected in series; extracting document content slices from the several file segments using the parsing resources, and quantizing and calculating vector data of the document content slices according to the slice content format of the document content slices, and storing the vector data according to the slice content format of the document content slices.

[0007] Optionally, dividing the initial document into several file fragments based on the document type of the initial document to be parsed includes: identifying the document type based on the suffix information of the initial document to be parsed, wherein the document type includes one of the following: programming code document, single-page text document, multi-page text document; if the document type is a programming code document, the programming code document is determined as a file fragment; if the document type is a single-page text document or a multi-page text document, the initial document is divided into multiple file fragments.

[0008] Optionally, dividing the initial document into multiple file segments includes: if the document type is a single-page text document, determining whether the total number of lines or the total number of data blocks in the initial document exceeds a preset threshold; if the total number of lines or the total number of data blocks in the initial document exceeds the preset threshold, dividing the initial document into multiple first file segments, wherein the number of document lines or data blocks in each first file segment is less than the preset threshold; if the document type is a multi-page text document, determining whether the total number of pages in the initial document exceeds a preset number of pages; if the total number of pages in the initial document exceeds the preset number of pages, dividing the initial document into multiple second file segments, wherein the number of document pages in each second file segment is less than the preset number of pages.

[0009] Optionally, allocating parsing resources for the plurality of file fragments based on historical resource utilization and the document type includes: calculating a first average utilization rate of a first resource pool and a second average utilization rate of a second resource pool in the previous historical period, wherein the total resource pool includes the first resource pool and the second resource pool, the first resource pool is used to perform parsing tasks for a first document type, the second resource pool is used to perform parsing tasks for a second document type, and the data capacity of the first document type is smaller than that of the second document type; calculating a first resource quantity of the first resource pool and a second resource quantity of the second resource pool in the current period based on the first average utilization rate and the second average utilization rate; and allocating parsing resources for the plurality of file fragments from the first resource pool and the second resource pool according to the document type.

[0010] Optionally, calculating the first resource quantity of the first resource pool and the second resource quantity of the second resource pool for the current period based on the first average utilization rate and the second average utilization rate includes: determining the total number of resources in the total resource pool; and calculating the proportion of the first resource quantity in the first resource pool for the current period using the following formula. The ratio of the second resource in the second resource pool : ; Where k represents the smoothing coefficient. This is the first average utilization rate. The second average utilization rate is used; the first resource quantity of the first resource pool and the second resource quantity of the second resource pool in the current period are calculated using the first resource quantity ratio, the second resource quantity ratio, and the total resource quantity, respectively.

[0011] Optionally, allocating parsing resources for the plurality of file fragments from the first resource pool and the second resource pool according to the document type includes: determining the document type of the initial document corresponding to the plurality of file fragments, wherein the document type includes the first document type and the second document type; determining, for the first resource pool and the second resource pool, the number of used resources in the current period, the number of users waiting for parsing, and the number of expected new users for parsing in the future; and calculating the first available resource quantity for the first document type using the following formula. and the second available resource quantity of the second document type : in, Take either the first resource quantity or the second resource quantity. This indicates the number of resources used in the corresponding resource pool. This indicates the number of users waiting to have their data resolved in the corresponding resource pool. This indicates the expected number of new users to be parsed in the corresponding resource pool; the first available resource quantity is allocated as parsing resources for the first document type file fragment among the plurality of file fragments, and the second available resource quantity is allocated as parsing resources for the second document type file fragment among the plurality of file fragments.

[0012] Optionally, extracting document content slices from the plurality of file fragments using the parsed resources includes: for each file fragment, extracting the initial document content from the file fragment according to the document type of the corresponding initial document; identifying regular content and special content in the initial document content, wherein the special content includes URLs in the title and body; finding a preset segmentation template that matches the document type; using the preset segmentation template to perform sentence-level segmentation on the regular content to obtain segmented fragments; and outputting the segmented fragments and the special content as document content slices of the file fragments.

[0013] Optionally, extracting the initial document content from the file fragment based on the document type of the corresponding initial document includes: if the document type of the corresponding initial document is a presentation document, generating snapshot images of the file fragment page by page; calling a pre-trained first network model to mark the text area of ​​the snapshot image; extracting the first initial document content from the text area, wherein the first initial document content includes text, tables, and image content; if the document type of the corresponding initial document is a table document, calling a pre-trained second network model to parse the table header and data type of the file fragment; extracting the second initial document content from the file fragment row by row; if the document type of the corresponding initial document is a docx or md document, extracting text paragraphs, table elements, and image elements from the file fragment, and using an OCR model to extract text elements from the image elements; determining the text paragraphs, table elements, and text elements as the third initial document content; if the document type of the corresponding initial document is a code document; finding a syntax tree matching the corresponding coding language; and using the syntax tree to extract the fourth initial document content from the file fragment.

[0014] Optionally, quantizing the vector data of the document content slice based on the slice content format includes: parsing the slice content format of the document content slice; and generating the vector data of the document content slice based on the slice content format.

[0015] Optionally, generating vector data for the document content slice based on the slice content format includes: if the slice content format of the document content slice is text or code, calling a text vector model to generate first vector data for the document content slice; if the slice content format of the document content slice is image, calling a multimodal large model to perform quantization calculations on the image and text in the image of the document content slice to obtain second vector data for the document content slice; if the slice content format of the document content slice is table, extracting the table header and data rows of the document content slice; concatenating the table header and data rows according to a preset specific format to obtain concatenated data; and generating third vector data for the concatenated data.

[0016] Optionally, storing the vector data according to the slice content format of the document content slice includes: if the slice content format of the document content slice is text, determining whether the vector data exceeds a threshold length; if the vector data exceeds the threshold length, generating summary data of the vector data, storing it in the database as vector data of the document content slice, and storing the summary data as first associated data of the vector data; if the slice content format of the document content slice is image, obtaining the text within the image and the image URL information of the document content slice; storing the vector data in the database, and storing the text within the image and the URL information as second associated data of the vector data; if the slice content format of the document content slice is table, obtaining the field name and field content of the document content slice by row unit; storing the vector data in the database, and storing the field name and the field content as third associated data of the vector data; if the slice content format of the document content slice is code, generating tagging information of the document content slice, wherein the tagging information is used to mark the programming language information of the code block; storing the vector data in the database, and storing the tagging information as fourth associated data of the vector data.

[0017] According to another embodiment of the present invention, a document parsing apparatus is provided, comprising: a segmentation module, configured to segment the initial document into several file fragments according to the document type of the initial document to be parsed, wherein the data capacity of the file fragments is less than a preset capacity; an allocation module, configured to allocate parsing resources for the several file fragments according to historical resource utilization and the document type, wherein the parsing resources include multiple parsing nodes with document parsing capabilities, each parsing node including multiple pipelines connected in series; and a parsing module, configured to extract document content slices from the several file fragments using the parsing resources, and to quantize and calculate vector data of the document content slices according to the slice content format of the document content slices, and to store the vector data according to the slice content format of the document content slices.

[0018] According to another aspect of the embodiments of this application, a storage medium is also provided, the storage medium including a stored program that executes the above steps when the program is run.

[0019] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the programs stored in the memory.

[0020] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the above-described method.

[0021] The beneficial effects of this invention are: 1. In terms of parsing implementation logic, a pipeline distribution and subscription structure is adopted to decompose the document parsing process, so that the various processes in the document parsing process do not block each other, realizing a highly efficient operation mode of parallel operation between stages and serial operation within stages. It can realize parsing and answering simultaneously. At the underlying knowledge processing level, it is refined to the sentence level, and at the same time, it makes adaptation processing for different knowledge types, which greatly improves the accuracy of knowledge retrieval. 2. Through the combined action of internal document parsing logic and external document collaboration mechanism, and by constructing a two-layer resource management system of user resource scheduling and document size resource scheduling, the timeliness of document parsing under different users and the overall stability of the system are effectively guaranteed. A fair, dynamic and efficient resource allocation strategy in a distributed environment is realized, which solves the problem of a single user task monopolizing resources, and avoids the situation where large documents and small documents compete for resources. Thus, the parsing efficiency and resource utilization are maximized in the overall system. 3. A vector data storage solution for different content types has been implemented, storing text, tables, images, and code in a semantic vector database or keyword matching database respectively, and supporting granular storage at the sentence level, table row level, and code block level, thereby improving the efficiency, accuracy, and scalability of document knowledge parsing. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a hardware structure block diagram of a server according to an embodiment of the present invention; Figure 2 This is a flowchart of a document parsing method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the pipeline of the parsing node in an embodiment of the present invention; Figure 4 This is a schematic diagram of the resource scheduling structure in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the allocation of parsing nodes in an embodiment of the present invention; Figure 6 This is a detailed flowchart of a distributed document parsing and quantization method according to an embodiment of the present invention; Figure 7 This is a structural block diagram of a document parsing apparatus according to an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present application can be combined with each other.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] Example 1 The method embodiment provided in Embodiment 1 of this application can be executed in a processing device such as a server or processor. Taking running on a server as an example, Figure 1 This is a hardware structure block diagram of a server according to an embodiment of the present invention. Figure 1 As shown, a server may include one or more ( Figure 1 Only one is shown in the image. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the server may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server described above. For example, the server may also include components that are more complex than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0026] The memory 104 can be used to store server programs, such as application software programs and modules, like the server program corresponding to a document parsing method for a server in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the server program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the server via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0027] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the server's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0028] This embodiment provides a document parsing method. Figure 2 This is a flowchart of a document parsing method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Divide the initial document into several file segments according to the document type of the initial document to be parsed, wherein the data capacity of the file segments is less than the preset capacity; The initial document can be any format of document to be archived and stored.

[0029] Optionally, before splitting the initial document into several file fragments according to the document type of the initial document to be parsed, document types that cannot be recognized or processed can be converted to PDF format, such as converting the initial documents to be parsed in doc and ppt types into PDF documents.

[0030] Step S202: Allocate parsing resources for the plurality of file fragments according to the historical resource utilization rate and the document type, wherein the parsing resources include a plurality of parsing nodes with document parsing capabilities, and each parsing node includes a plurality of pipelines connected in series. Figure 3This is a schematic diagram of the pipeline of the parsing node in an embodiment of the present invention. The parsing node consists of four functional modules: document format conversion module, document content extraction module, document content quantification module, and document content storage module.

[0031] The document format conversion module converts the input documents into a specified standard format that the subsequent model can recognize, such as Markdown (MD) format, for subsequent processing. The document content extraction module performs fine-grained segmentation of the document content based on the MD format and classifies and labels it according to content type. The document content quantization module performs vectorization calculation on the extracted knowledge content to support subsequent knowledge similarity matching and recall operations. The document content storage module is responsible for storing the quantized knowledge content and adopts differentiated storage methods according to different types.

[0032] Step S203: Extract document content slices from the plurality of file fragments using the parsing resources, and quantize and calculate the vector data of the document content slices according to the slice content format of the document content slices, and store the vector data according to the slice content format of the document content slices; The solution in this embodiment can be applied to scenarios such as knowledge base construction, retrieval, and knowledge question answering.

[0033] Through the above steps, the initial document to be parsed is divided into several file fragments according to its document type, wherein the data capacity of each file fragment is less than a preset capacity. Parsing resources for the file fragments are allocated based on historical resource utilization and the document type. These parsing resources include multiple parsing nodes with document parsing capabilities, each parsing node comprising multiple serially connected pipelines. Document content slices are extracted from the file fragments using these parsing resources, and vector data of the document content slices are quantized according to their slice content format. This vector data is then stored according to the slice content format. This process solves the technical problem of low document parsing efficiency in existing technologies, achieves a fair, dynamic, and efficient resource allocation strategy in a distributed environment, solves the problem of single-user tasks monopolizing resources, and avoids resource contention between large and small documents. Therefore, the overall system achieves maximum parsing efficiency and optimized resource utilization, while also improving document parsing efficiency and storage granularity.

[0034] In one embodiment of this example, dividing the initial document into several file fragments according to the document type of the initial document to be parsed includes: identifying the document type based on the suffix information of the initial document to be parsed, wherein the document type includes one of the following: programming code document, single-page text document, and multi-page text document; if the document type is a programming code document, the programming code document is determined as a file fragment; if the document type is a single-page text document or a multi-page text document, the initial document is divided into multiple file fragments.

[0035] Optionally, single-page text documents can be csv, xlsx, xls, docx, txt, or md documents; multi-page text documents can be pdf or pptx documents; and programming code documents can be documents with suffixes such as py, java, cpp, c, h, sh, bat, html, vue, js, jsx, css, scss, sass, ts, or tsx.

[0036] The document type to be parsed is determined. Based on the document type, a corresponding method is used to determine whether the document is a large document. Large documents are further segmented. Specifically: if the document type is csv, xlsx, xls, docx, txt, or md, it is further divided into large and small documents based on whether the total number of lines or data blocks exceeds a preset threshold. If it exceeds the preset number of lines, it is considered a large document; otherwise, it is a small document. Further, if it is a large document, it is segmented into multiple small documents of the same document type based on the preset number of lines. If the document type is pdf or pptx, it is further divided into large and small documents based on whether the total number of pages exceeds a preset number of pages. If it exceeds the preset number of pages, it is considered a large document; otherwise, it is a small document. Furthermore, if it is a large document, the document to be parsed will be divided into multiple smaller documents of the same document type according to the preset number of pages; if the document type is programming source code such as py, java, cpp, c, h, sh, bat, html, vue, js, jx, css, scss, sass, ts, tsx, etc., the document will be directly classified as a smaller document.

[0037] In one example, dividing the initial document into multiple file segments includes: if the document type is a single-page text document, determining whether the total number of lines or the total number of data blocks in the initial document exceeds a preset threshold; if the total number of lines or the total number of data blocks in the initial document exceeds the preset threshold, dividing the initial document into multiple first file segments, wherein the number of document lines or data blocks in each first file segment is less than the preset threshold; if the document type is a multi-page text document, determining whether the total number of pages in the initial document exceeds a preset number of pages; if the total number of pages in the initial document exceeds the preset number of pages, dividing the initial document into multiple second file segments, wherein the number of document pages in each second file segment is less than the preset number of pages.

[0038] In this example, pre-set size thresholds for various document types: 500 pages for PDF documents; 100 pages for PPTX documents; 5000 lines for TXT text files; 2000 blocks for DOCX documents; and 2000 lines for tabular documents such as CSV and XLSX. Documents exceeding these thresholds are marked as large documents, while those below are marked as small documents. Additionally, coded files such as .py and .java are directly marked as small documents. These thresholds can be adjusted as needed based on actual requirements.

[0039] During the process of determining whether each document meets the set threshold, the documents are simultaneously segmented and subdivided. This involves dividing the original document content into multiple file fragments of the same type according to the threshold, facilitating subsequent pipeline scheduling, increasing fault tolerance, and reducing internal waiting time. The file fragment segmentation process includes: for .txt text files, segmentation is done in units of 500 lines; for .docx files, segmentation is done in units of 50 structural blocks; for .pptx files, segmentation is done in units of 50 pages; for .pdf files, segmentation is done in units of 50 pages; for table files, segmentation is done in units of 50 lines; and for code files, segmentation is done in units of 10 code blocks. When processing tables, a large model is first used to intelligently analyze the table header range to avoid content extraction errors caused by the first row not being column names in special table formats. After segmentation, multiple file fragments containing all the information from the original document are obtained.

[0040] In this embodiment, allocating parsing resources for the plurality of file fragments based on historical resource utilization and the document type includes: calculating the first average utilization rate of the first resource pool and the second average utilization rate of the second resource pool in the previous historical period, wherein the total resource pool includes the first resource pool and the second resource pool, the first resource pool is used to perform parsing tasks for the first document type, the second resource pool is used to perform parsing tasks for the second document type, and the data capacity of the first document type is smaller than that of the second document type; calculating the first resource quantity of the first resource pool and the second resource quantity of the second resource pool in the current period based on the first average utilization rate and the second average utilization rate; and allocating parsing resources for the plurality of file fragments from the first resource pool and the second resource pool according to the document type.

[0041] The first resource quantity is the resource quantity of the parsing resources in the first resource pool, i.e., the number of parsing nodes. The second resource quantity is the resource quantity of the parsing resources in the second resource pool, i.e., the number of parsing nodes. By counting the number of processes of the parsing nodes (each parsing node corresponds to one process) in the first and second resource pools and reading the storage space occupied by the first and second resource pools, the number of parsing nodes can be determined.

[0042] In one example, calculating the first resource quantity of the first resource pool and the second resource quantity of the second resource pool for the current period based on the first average utilization rate and the second average utilization rate includes: determining the total number of resources in the total resource pool; and calculating the proportion of the first resource quantity in the first resource pool for the current period using the following formula. The ratio of the second resource in the second resource pool : ; Where k represents the smoothing coefficient. This is the first average utilization rate. The second average utilization rate is used; the first resource quantity of the first resource pool and the second resource quantity of the second resource pool in the current period are calculated using the first resource quantity ratio, the second resource quantity ratio, and the total resource quantity, respectively.

[0043] In this example, the smoothing coefficient k is used to describe the degree of attention paid to historical data during data processing or model prediction. In this embodiment, it is used to describe the degree of attention paid to the resource allocation of the first and second resource pools in the current period compared to the resource allocation of the first and second resource pools in the historical period. The value ranges from 0 to 1. The larger the value, the higher the degree of attention, and the greater the impact of the resource allocation of the first and second resource pools in the historical period on the current period.

[0044] The first resource ratio is the ratio between the first resource quantity in the first resource pool and the total resource quantity in the total resource pool. Similarly, the second resource ratio is the ratio between the second resource quantity in the second resource pool and the total resource quantity in the total resource pool. For example, if the total resource quantity (number of resolver nodes) in the total resource pool is 100, the first resource quantity in the first resource pool is 40, and the second resource quantity in the second resource pool is 60, then the first resource ratio is 40%, and the second resource ratio is 60%.

[0045] Using the solution in this example, when the system detects that there are significantly more small document (first document type) tasks than large document (second document type) tasks recently, the small document resource pool (first resource pool) will automatically expand to ensure the fast parsing capability of small documents; conversely, when the proportion of large document tasks increases, the system will dynamically increase the proportion of large document resource pool (second resource pool) to ensure that long-running tasks can obtain sufficient parsing resources.

[0046] In one example, allocating parsing resources for the plurality of file fragments from the first resource pool and the second resource pool according to the document type includes: determining the document type of the initial document corresponding to the plurality of file fragments, wherein the document type includes the first document type and the second document type; determining the number of used resources, the number of users waiting for parsing, and the number of expected new parsing users in the future for the first resource pool and the second resource pool, respectively; and calculating the first available resource quantity for the first document type using the following formula. and the second available resource quantity of the second document type : in, Take either the first resource quantity or the second resource quantity. This indicates the number of resources used in the corresponding resource pool. This indicates the number of users waiting to have their data resolved in the corresponding resource pool. This indicates the expected number of new users to be parsed in the corresponding resource pool; the first available resource quantity is allocated to the parsing resources of the first document type file fragments among the plurality of file fragments, and the second available resource quantity is allocated to the parsing resources of the second document type file fragments among the plurality of file fragments. Represents a positive integer.

[0047] Optional, It can be obtained directly through statistics. It can then be predicted based on historical data through pre-trained time series.

[0048] When the calculation result r=0, the task will automatically enter the parsing queue until the system releases idle resources and then reschedules it.

[0049] This embodiment achieves a fair, dynamic, and efficient resource allocation strategy in a distributed environment by combining internal document parsing logic and external document collaboration mechanisms, employing a two-layer resource management mechanism of user identity scheduling and document size scheduling. This mechanism solves the problem of a single user task monopolizing resources, while avoiding resource contention between large and small documents, thereby maximizing parsing efficiency and optimizing resource utilization across the entire system.

[0050] Figure 4 This is a schematic diagram of the resource scheduling structure in this embodiment of the invention, including a registration center, a master parsing node, and multiple slave parsing nodes. After all parsing nodes start, they automatically register with the registration center. The registration center provides system parsing node registration services and automatically elects a master node. If the current master node fails, the system will trigger a re-election of a new master node. The master node is responsible for the dynamic balancing of global small and large file parsing resources, task scheduling, and dynamic balancing of user parsing resources. The master node is mainly responsible for document parsing task allocation, task retries, and task migration. Slave nodes are responsible for pipeline work such as document conversion, extraction, quantization, and storage. When a slave node fails, the master node also implements failure retry and task migration strategies. During each step of the task—large document splitting, document content slicing, and slice content quantization—the slave node records the intermediate and final states. Intermediate states refer to temporary data generated at each step, while the final state indicates whether each step succeeded or failed. Failure retries include parsing exception retries and parsing timeout retries. Parsing exception retries include IO exceptions, network exceptions, etc., and will retry within a preset number of exception retries; after that, they will not retry. Parsing timeout retries occur when there are no error messages during the parsing process and the state has not been updated within a preset time; in such cases, a retry will be performed. Task migration means that when a document parsing task for a document is assigned to one or more parsing nodes, and one or more of these nodes fail or are unable to handle the document parsing task, the system will automatically migrate the task to other normal document parsing nodes. Task retries will resume execution from where it left off, avoiding the duplication of already executed steps.

[0051] At the user identity scheduling layer, the master node employs a global scheduler. This scheduler uses the user's workspace as a unique identifier, and each user's tasks are stored in an independent queue. The scheduler polls the global task queue, distributing tasks according to their submission order. When only a single user exists, that user is allowed to execute multiple tasks concurrently, up to a preset maximum parallelism (e.g., a maximum of 3 tasks per user). When multiple users exist, the scheduler automatically switches to a fair mode, ensuring that each user is assigned only one task to execute within a single polling cycle, thus preventing a single user from excessively consuming global resources. The scheduler controls the total concurrency of the system through a global semaphore, with an upper limit equal to the sum of the number of threads in the large document resource pool and the small document resource pool. If all resources are currently occupied, newly submitted tasks enter the user's pending parsing queue. Once an existing task completes and releases resources, the scheduler reallocates resources from the pending parsing queue according to a fair strategy. This mechanism ensures that the system maintains a stable and controllable load level even under large-scale concurrency. At the document size scheduling layer, the resource pool is divided into a small document resource pool and a large document resource pool based on the document size. These two pools are independent and do not interfere with each other. Specifically, after determining the document size in the aforementioned internal document parsing logic, the small and large document resource pools are scheduled in the pipeline parsing process. The small document resource pool is suitable for lightweight document types and has a larger number of threads to ensure fast response for small tasks. The large document resource pool is suitable for large document types with many pages, has a smaller number of threads, but resources are more concentrated for each task to ensure the stability of the parsing process. When scheduling tasks, the master node automatically allocates tasks to the corresponding resource pool based on the document size and type, thereby achieving isolation and optimized execution of tasks of different sizes.

[0052] The solution in this embodiment adopts a distributed document parsing strategy and establishes a two-layer mechanism of user resource scheduling manager and document size resource scheduling manager to work together, thereby ensuring the fairness and efficiency of the system when multiple users and multiple types of documents are submitted concurrently.

[0053] In this embodiment, extracting document content slices from the plurality of file fragments using the parsed resources includes: for each file fragment, extracting the initial document content from the file fragment according to the document type of the corresponding initial document; identifying regular content and special content in the initial document content, wherein the special content includes URLs in the title and body; finding a preset cutting template that matches the document type; using the preset cutting template to perform sentence-level cutting on the regular content to obtain cut segments; and outputting the cut segments and the special content as document content slices of the file fragments.

[0054] Optionally, extracting the initial document content from the file fragment based on the document type of the corresponding initial document includes: if the document type of the corresponding initial document is a presentation document, generating snapshot images of the file fragment page by page; calling a pre-trained first network model to mark the text area of ​​the snapshot image; extracting the first initial document content from the text area, wherein the first initial document content includes text, tables, and image content; if the document type of the corresponding initial document is a table document, calling a pre-trained second network model to parse the table header and data type of the file fragment; extracting the second initial document content from the file fragment row by row; if the document type of the corresponding initial document is a docx or md document, extracting text paragraphs, table elements, and image elements from the file fragment, and using an OCR model to extract text elements from the image elements; determining the text paragraphs, table elements, and text elements as the third initial document content; if the document type of the corresponding initial document is a code document; finding a syntax tree matching the corresponding coding language; and using the syntax tree to extract the fourth initial document content from the file fragment.

[0055] The first network model is a multimodal large model. It takes an image as input (such as a snapshot image in this embodiment), constrains the model's behavior through instructions to recognize the image, and outputs the content of each region in the image. The recognized text regions are then marked using text boxes or similar methods. The second network model is a general-purpose language large model. It reads the first few rows of data from a table, inputs them into the model, and constrains the model through instructions to recognize the table header and data type.

[0056] Optionally, the presentation document can be a PDF or PPTX document, the table document can be a CSV, XLSX, or XLS document, and the code document can be a PY, Java, C++, C, H, SH, BAT, HTML, Vue, JS, JSX, CSS, SCSS, Sass, TS, or TSX document.

[0057] If the document type of the corresponding initial document is a large document, the parsing tasks of the file segments are evenly distributed to the parsing nodes (slave nodes) according to their idle time and minimum waiting time; if the document type of the corresponding initial document is a small document, the document parsing task is randomly assigned to a parsing node according to its idle time and minimum waiting time.

[0058] After extracting the initial document content from the file fragment according to the document type of the corresponding initial document, invalid characters and invalid data can be removed according to preset rules to complete the splicing of cross-page paragraphs and tables; when using the preset cutting template to perform sentence-level cutting of the regular content: the text content is matched according to the rules and the preset cutting template to perform fine-grained cutting at the sentence level, and special forms of content such as titles and URLs are protected to form document content slices.

[0059] In this embodiment, the quantitative calculation of the vector data of the document content slice based on the slice content format includes: parsing the slice content format of the document content slice; and generating the vector data of the document content slice based on the slice content format.

[0060] In one example, generating vector data for the document content slice based on the slice content format includes: if the slice content format of the document content slice is text or code, calling a text vector model to generate the first vector data of the document content slice; if the slice content format of the document content slice is image, calling a multimodal large model to perform quantization calculations on the image and text in the image of the document content slice to obtain the second vector data of the document content slice; if the slice content format of the document content slice is table, extracting the table header and data rows of the document content slice; concatenating the table header and data rows according to a preset specific format to obtain concatenated data; and generating the third vector data of the concatenated data.

[0061] In this embodiment, storing the vector data according to the slice content format of the document content slice includes: if the slice content format of the document content slice is text format, determining whether the vector data exceeds a threshold length; if the vector data exceeds the threshold length, generating summary data of the vector data, storing it in the database as the vector data of the document content slice, and storing the summary data as the first associated data of the vector data; if the slice content format of the document content slice is image format, obtaining the text within the image and the image URL information of the document content slice; storing the vector data in the database, and storing the text within the image and the URL information as the second associated data of the vector data; if the slice content format of the document content slice is table format, obtaining the field name and field content of the document content slice by row unit; storing the vector data in the database, and storing the field name and the field content as the third associated data of the vector data; if the slice content format of the document content slice is code format, generating the tagging information of the document content slice, wherein the tagging information is used to mark the programming language information of the code block; storing the vector data in the database, and storing the tagging information as the fourth associated data of the vector data.

[0062] This embodiment includes a semantic vector database and a keyword database, which are used for retrieval using semantic vectors and natural language, respectively. During storage, the semantic vector database and keyword matching database are used to store the quantified content of document parsing separately, and further subdivided based on the content slice type. Specifically: if the current document content slice is text, sentence-level detail is maintained in the database storage; for sentences exceeding a threshold length, a summary information is generated using a large model and stored in the database. If the document content slice is an image, its OCR text recognition content is stored along with its file cloud URL information for later display. If the document content slice is a table, its row-by-row format is used in the database, along with its field names and content. If the document content slice is code, it is stored in the database as code blocks, with the code block's programming language information tagged.

[0063] After determining the document size, multiple self-information blocks are distributed to their respective pipelines for parsing and processing based on the file type. PDF, docx, pptx, doc, and ppt files are all sent to the Markdown style processing pipeline. TXT text files are sent to the text style processing pipeline, table files to the table style processing pipeline, and image files to the image style processing pipeline. Figure 5 This is a schematic diagram of the allocation of parsing nodes in an embodiment of the present invention. After the document is segmented, the file fragments are distributed to the parsing nodes for pipeline processing.

[0064] Taking Markdown (MD) style data as an example, the pipelined parsing nodes are divided into five separate pipeline combinations. Each pipeline is responsible for different processing tasks, and they send messages to each other through subscription. These pipelines are: document content to Markdown pipeline, Markdown data formatting pipeline, Markdown data cleaning pipeline, document knowledge extraction pipeline, knowledge vectorization pipeline, and database storage pipeline. The collaboration between pipelines is as follows: processing within a pipeline is serial, while processing between pipelines is parallel. That is, after each pipeline completes the processing of its batch of data, it immediately sends it to the next pipeline and processes the next batch of data immediately. The next pipeline, upon receiving the data, also processes it immediately. This pipelined structure transforms the originally blocking stages of single document processing into simultaneous processes, greatly shortening document processing time.

[0065] In the example above, the document content to Markdown pipeline serves several purposes: it converts PDF, doc, docx, ppt, and pptx files into Markdown format, which is easily understood by the large model, using third-party tools. Each step processes the self-information blocks resulting from the aforementioned document segmentation. Simultaneously, it generates snapshot images of the original document page by page, using the large model to mark headers, footers, footnotes, backgrounds, and text areas. This information is then combined with the subsequent document knowledge extraction pipeline to ensure the accuracy of information extraction and preserve correct structural information.

[0066] The Markdown Data Formatting Pipeline and the Markdown Data Cleaning Pipeline unify and clean MD format data obtained after converting different types of document sources. The document knowledge extraction pipeline transforms the MD data (converted from sub-information blocks) into paragraph-level content classification and sentence-level knowledge decomposition. Paragraph-level classification divides the MD content into one of the following: title, image, mathematical formula, text, table, or code. Sentence-level decomposition refines the paragraph classification. For titles, the original style tags are preserved; for images, OCR is used for text recognition; for mathematical formulas, LaTeX styles are adopted; and for text, rule-based sentence decomposition is performed, including: special coherence format protection, data cleaning, segmentation based on common ending punctuation marks, sentence segmentation based on a specified threshold length, and short sentence merging. For tables (in MD format), they are split into multiple sub-tables according to the MD format and a specified number of rows during storage; for code, it is stored in units of complete code functions based on the syntax tree.

[0067] After passing through the knowledge extraction pipeline, the knowledge content, originally organized into sub-information blocks, is further subdivided into sentence levels according to different paragraph types, resulting in a more granular division. Then, it is fed into the knowledge vectorization pipeline, processing a set batch size of knowledge at a time, and invoking different vectorization models based on the type of knowledge slice. If the slice content is text, a text vectorization model is used to quantize the text; if the slice content is an image, a multimodal large model is used to quantize the image and the text extracted by OCR; if the slice content is a table, the table header and data rows are concatenated according to a preset specific format and then quantized using a text vectorization model. Different quantization models ensure a certain degree of differentiation between different types of knowledge, facilitating subsequent retrieval.

[0068] The vectorized data is then stored in the database storage pipeline. After quantization, the sentence-level knowledge content slices are combined with the original content and the vectorized content, and stored in the semantic vector database and the keyword matching database, respectively. During storage, further subdivision is performed based on the content slice type, specifically: If the current document content slice is text, sentence-level detail is maintained in the database storage. For sentences exceeding a threshold length, a summary is generated using a large model and stored in the database. If the document content slice is an image, its OCR text recognition content is stored along with its file cloud URL information for later display. If the document content slice is a table, it is stored in the database row by row, along with its field names and content. If the document content slice is code, it is stored in the database as a block by block, with the code block's programming language information tagged. This embodiment provides a distributed document quantization parsing method and system, which can intelligently segment documents according to document type and size, and evenly distribute parsing tasks to distributed parsing nodes based on idle time and minimum waiting time. It also incorporates a large model to achieve quantization processing of multimodal content (text, tables, images, code) slices. Furthermore, this invention designs a vectorized storage scheme for different content types, storing text, tables, images, and code in a semantic vector database or keyword matching database respectively, and supporting granular storage at the sentence level, table row level, and code block level, thereby improving the efficiency, accuracy, and scalability of document knowledge parsing.

[0069] Figure 6 This is a detailed flowchart of a distributed document parsing and quantization method according to an embodiment of the present invention. The process includes the following distributed document parsing steps: obtaining the document to be parsed by the user; identifying the document format and performing necessary conversions to convert the target document into PDF format; determining the document size and classifying it, and splitting large files into smaller file fragments according to different file formats; obtaining document parsing resources available to the user; determining whether the user terminal has available resources; if not, entering the user's parsing waiting queue; if so, distributively extracting document content according to different document formats; distributively quantizing the content using different methods according to different content formats; and storing the original parsed document content and the corresponding quantized content.

[0070] This embodiment effectively ensures the timeliness of document parsing and the overall stability of the system by constructing a two-layer resource management system of user resource scheduling and document size resource scheduling. Taking a 6000-page PDF document as an example, compared with non-distributed parsing methods, the parsing time of this embodiment is reduced from 46 hours to 4 hours. In terms of parsing implementation logic, a pipelined distribution-subscription structure is adopted to decompose the document parsing process, so that the various processes in the document parsing process do not block each other, realizing a highly efficient operation mode of parallel operation between stages and serial operation within stages, which can realize parsing and question answering simultaneously. At the underlying knowledge processing level, it is refined to the sentence level, and adaptive processing is made for different knowledge types, which greatly improves the accuracy of knowledge retrieval.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0072] Example 2 This embodiment also provides a document parsing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0073] Figure 7 This is a structural block diagram of a document parsing apparatus according to an embodiment of the present invention, such as... Figure 7 As shown, the device includes: The segmentation module 71 is used to segment the initial document into several file fragments according to the document type of the initial document to be parsed, wherein the data capacity of the file fragments is less than a preset capacity; The allocation module 72 is used to allocate parsing resources for the plurality of file fragments according to the historical utilization rate of resources and the document type, wherein the parsing resources include a plurality of parsing nodes with document parsing capabilities, and each parsing node includes a plurality of pipelines connected in series. The parsing module 73 is used to extract document content slices from the plurality of file fragments using the parsing resources, and to quantize and calculate the vector data of the document content slices according to the slice content format of the document content slices, and to store the vector data according to the slice content format of the document content slices.

[0074] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0075] Example 3 Embodiments of the present invention also provide a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0076] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps: S1, the initial document is divided into several file segments according to the document type of the initial document to be parsed, wherein the data capacity of the file segments is less than the preset capacity; S2, allocate parsing resources for the plurality of file fragments according to the historical utilization rate of resources and the document type, wherein the parsing resources include a plurality of parsing nodes with document parsing capabilities, and each parsing node includes a plurality of pipelines connected in series. S3, using the parsing resource to extract document content slices from the plurality of file fragments, and quantizing and calculating the vector data of the document content slices according to the slice content format of the document content slices, and storing the vector data according to the slice content format of the document content slices.

[0077] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0078] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0079] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0080] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program: S1, the initial document is divided into several file segments according to the document type of the initial document to be parsed, wherein the data capacity of the file segments is less than the preset capacity; S2, allocate parsing resources for the plurality of file fragments according to the historical utilization rate of resources and the document type, wherein the parsing resources include a plurality of parsing nodes with document parsing capabilities, and each parsing node includes a plurality of pipelines connected in series. S3, using the parsing resource to extract document content slices from the plurality of file fragments, and quantizing and calculating the vector data of the document content slices according to the slice content format of the document content slices, and storing the vector data according to the slice content format of the document content slices.

[0081] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0082] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0084] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0085] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A document parsing method, characterized in that, include: The initial document is divided into several file segments according to the document type of the initial document to be parsed, wherein the data capacity of the file segments is less than the preset capacity; Parsing resources for the plurality of file fragments are allocated based on historical resource utilization and the document type, wherein the parsing resources include multiple parsing nodes with document parsing capabilities, and each parsing node includes multiple pipelines connected in series. The document content slices of the plurality of file fragments are extracted using the parsing resources, and the vector data of the document content slices are quantized according to the slice content format of the document content slices, and the vector data is stored according to the slice content format of the document content slices.

2. The method according to claim 1, characterized in that, The initial document is divided into several file fragments based on its document type, including: The document type is identified based on the suffix information of the initial document to be parsed, wherein the document type includes one of the following: programming code document, single-page text document, and multi-page text document; If the document type is a programming code document, the programming code document is determined as a file fragment; if the document type is a single-page text document or a multi-page text document, the initial document is divided into multiple file fragments.

3. The method according to claim 2, characterized in that, The initial document is divided into multiple file fragments, including: If the document type is a single-page text document, determine whether the total number of lines or the total number of data blocks in the initial document exceeds a preset threshold; if the total number of lines or the total number of data blocks in the initial document exceeds the preset threshold, divide the initial document into multiple first file segments, wherein the number of document lines or the number of data blocks in each first file segment is less than the preset threshold. If the document type is a multi-page text document, determine whether the total number of pages in the initial document exceeds the preset number of pages; if the total number of pages in the initial document exceeds the preset number of pages, divide the initial document into multiple second file segments, wherein the number of pages in each second file segment is less than the preset number of pages.

4. The method according to claim 1, characterized in that, The allocation of parsing resources for the aforementioned file fragments based on historical resource utilization and the document type includes: Calculate the first average utilization rate of the first resource pool and the second average utilization rate of the second resource pool in the previous historical period, wherein the total resource pool includes the first resource pool and the second resource pool, the first resource pool is used to perform parsing tasks of the first document type, the second resource pool is used to perform parsing tasks of the second document type, and the data capacity of the first document type is smaller than that of the second document type. Calculate the first resource quantity of the first resource pool and the second resource quantity of the second resource pool for the current period based on the first average utilization rate and the second average utilization rate; Parsing resources for the plurality of file fragments are allocated from the first resource pool and the second resource pool according to the document type.

5. The method according to claim 4, characterized in that, The calculation of the first resource quantity of the first resource pool and the second resource quantity of the second resource pool for the current period based on the first average utilization rate and the second average utilization rate includes: Determine the total number of resources in the total resource pool; The following formula is used to calculate the proportion of the first resource in the first resource pool for the current period. The ratio of the second resource in the second resource pool : ; Where k represents the smoothing coefficient. This is the first average utilization rate. This is the second average utilization rate; The first resource quantity of the first resource pool and the second resource quantity of the second resource pool in the current period are calculated using the first resource quantity ratio, the second resource quantity ratio, and the total resource quantity, respectively.

6. The method according to claim 4, characterized in that, The allocation of parsing resources for the plurality of file fragments from the first resource pool and the second resource pool according to the document type includes: Determine the document type of the initial document corresponding to the plurality of file fragments, wherein the document type includes the first document type and the second document type; For the first resource pool and the second resource pool, determine the number of resources used in the current period, the number of users waiting to be resolved, and the number of new users expected to be resolved in the future, respectively. The first available resource quantity for the first document type is calculated using the following formula. and the second available resource quantity of the second document type : in, Take either the first resource quantity or the second resource quantity. This indicates the number of resources used in the corresponding resource pool. This indicates the number of users waiting to have their data resolved in the corresponding resource pool. This indicates the expected number of new users accessing the corresponding resource pool in the future. The first available resource quantity is allocated as parsing resources for the file fragments of the first document type among the plurality of file fragments, and the second available resource quantity is allocated as parsing resources for the file fragments of the second document type among the plurality of file fragments.

7. The method according to claim 1, characterized in that, Extracting document content slices from the plurality of file fragments using the parsed resources includes: For each file segment, extract the initial document content from the file segment according to the document type of the corresponding initial document; Identify regular and special content in the initial document content, wherein the special content includes URLs in the title and body text; Find a preset cutting template that matches the document type; The preset cutting template is used to cut the regular content into sentence-level segments to obtain segmented fragments. The segmented fragments and the special content are then output as document content slices of the file fragment.

8. The method according to claim 7, characterized in that, Extracting the initial document content from the file fragment based on the document type of the corresponding initial document includes: If the document type of the corresponding initial document is a presentation document, generate snapshot images of the file fragments page by page; call the pre-trained first network model to mark the text area of ​​the snapshot image; extract the content of the first initial document in the text area, wherein the content of the first initial document includes text, tables and image content; If the document type of the corresponding initial document is a table document, call the pre-trained second network model to parse the header and data type of the file segment; extract the content of the second initial document from the file segment row by row; If the document type of the corresponding initial document is docx or md, extract the text paragraphs, table elements, and image elements from the file fragments, and use an OCR model to extract the text elements from the image elements; determine the text paragraphs, table elements, and text elements as the content of the third initial document; If the document type of the corresponding initial document is a code document; find the syntax tree that matches the corresponding encoding language; use the syntax tree to extract the content of the fourth initial document from the file fragment.

9. The method according to claim 1, characterized in that, The vector data of the document content slice is calculated based on the slice content format of the document content slice, including: Parse the slice content format of the document content slice; Generate vector data of the document content slices based on the slice content format.

10. The method according to claim 9, characterized in that, Generating vector data for the document content slices based on the slice content format includes: If the content format of the document content slice is text or code, the text vector model is called to generate the first vector data of the document content slice; If the content format of the document content slice is an image, the multimodal large model is called to perform quantization calculations on the image and text in the document content slice to obtain the second vector data of the document content slice. If the content of the document content slice is in table format, extract the table header and data rows of the document content slice; concatenate the table header and data rows according to a preset specific format to obtain concatenated data; and generate the third vector data of the concatenated data.

11. The method according to claim 1, characterized in that, Storing the vector data according to the slice content format of the document content slice includes: If the content format of the document content slice is text, determine whether the vector data exceeds the threshold length; if the vector data exceeds the threshold length, generate summary data of the vector data, store it in the database as the vector data of the document content slice, and store the summary data as the first associated data of the vector data; If the content format of the document content slice is an image, obtain the text within the image and the URL information of the Unified Resource Locator (URL) of the image; store the vector data in the database, and store the text within the image and the URL information as the second associated data of the vector data; If the document content slice is in table format, the field name and field content of the document content slice are obtained by row unit; the vector data is stored in the database, and the field name and field content are stored as the third associated data of the vector data; If the format of the document content slice is code format, generate the tag information of the document content slice, wherein the tag information is used to mark the programming language information of the code block; store the vector data in the database, and store the tag information as the fourth associated data of the vector data.

12. A document parsing apparatus, characterized in that, include: The segmentation module is used to segment the initial document into several file fragments according to the document type of the initial document to be parsed, wherein the data capacity of the file fragments is less than a preset capacity; The allocation module is used to allocate parsing resources for the plurality of file fragments according to the historical utilization rate of resources and the document type, wherein the parsing resources include multiple parsing nodes with document parsing capabilities, and each parsing node includes multiple pipelines connected in series. The parsing module is used to extract document content slices from the plurality of file fragments using the parsing resources, and to quantize and calculate the vector data of the document content slices according to the slice content format of the document content slices, and to store the vector data according to the slice content format of the document content slices.

13. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 11 when it is run.

14. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 11.

Images