Document parsing method and device, storage medium and electronic device

Abstract

The application provides a document parsing method and device, a storage medium and an electronic device, wherein the method comprises the following steps: cutting an initial document into a plurality of file segments according to the document type of the initial document to be parsed, wherein the data capacity of the file segments is less than a preset capacity; allocating parsing resources for the plurality of file segments according to resource historical utilization and the document type, wherein the parsing resources comprise a plurality of parsing nodes with document parsing capability, each parsing node comprises a plurality of pipeline channels connected in series; extracting document content slices of the plurality of file segments by using the parsing resources, and quantitatively calculating vector data of the document content slices according to the slice content format of the document content slices, and storing the vector data according to the slice content format of the document content slices. Through the embodiment, the maximum parsing efficiency and the optimization of resource utilization are realized on the whole system, and the parsing efficiency and the storage granularity of the document are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a document parsing method and device, a storage medium and an electronic device. BACKGROUND

[0002] In related technologies, with the rapid development of information technology, a large amount of unstructured document data is constantly emerging, and how to efficiently parse and structure store it has become the key to improving enterprise knowledge utilization efficiency and reducing repetitive work. In recent years, the application of large model technology has brought new breakthroughs to knowledge base construction, significantly improving the accuracy of knowledge integration and intelligent question answering.

[0003] The document knowledge parsing technology in related technologies relies on a single modal processing method or a layout analysis method based on a lightweight model, and there are problems of dispersion and localization in parsing text, tables, pictures and formulas and other information, which makes it difficult to efficiently segment, distribute and quantify large documents, and there is a lack of unified multi-modal vector storage strategy, which limits the efficiency and accuracy in knowledge retrieval, content fusion and downstream applications. In addition, the existing technology relies on sequential processing or fixed resource allocation in large document processing and task scheduling, lacks dynamic distributed scheduling and load balancing mechanism, and cannot fully utilize computing resources and guarantee the real-time and stability of parsing.

[0004] For the above problems in related technologies, there is no efficient and accurate solution. SUMMARY

[0005] The present application provides a document parsing method and device, a storage medium and an electronic device to solve the technical problems in related technologies.

[0006] According to one embodiment of the present application, a document parsing method is provided, comprising: dividing an initial document to be parsed into a plurality of file segments according to the document type of the initial document, wherein the data capacity of the file segment is less than a preset capacity; allocating parsing resources of the plurality of file segments according to resource historical utilization rate and the document type, wherein the parsing resources include a plurality of parsing nodes with document parsing capability, and each parsing node includes a plurality of pipeline pipes connected in series; extracting document content slices of the plurality of file segments using the parsing resources, and quantitatively calculating vector data of the document content slices according to the slice content format of the document content slices, and storing the vector data according to the slice content format of the document content slices.

[0007] Optionally, the splitting the initial document into a plurality of file segments according to the document type of the initial document to be parsed comprises: identifying the document type according to suffix information of the initial document to be parsed, wherein the document type comprises one of the following: a programming code document, a single-page text document, and a multi-page text document; if the document type is the programming code document, determining the programming code document as one file segment; if the document type is the single-page text document or the multi-page text document, splitting the initial document into a plurality of file segments.

[0008] Optionally, the splitting the initial document into a plurality of file segments comprises: if the document type is the single-page text document, judging whether a total number of document lines or a total number of data blocks of the initial document exceeds a preset threshold; if the total number of document lines or the total number of data blocks of the initial document exceeds the preset threshold, splitting the initial document into a plurality of first file segments, wherein a number of document lines or a number of data blocks of each first file segment is less than the preset threshold; if the document type is the multi-page text document, judging whether a total number of pages of the initial document exceeds a preset number of pages; if the total number of pages of the initial document exceeds the preset number of pages, splitting the initial document into a plurality of second file segments, wherein a number of document pages of each second file segment is less than the preset number of pages.

[0009] Optionally, the allocating the parsing resources of the plurality of file segments according to the historical resource utilization and the document type comprises: calculating a first average utilization of a first resource pool and a second average utilization of a second resource pool in a last historical period, wherein a total resource pool comprises the first resource pool and the second resource pool, the first resource pool is used for performing a parsing task of a first document type, the second resource pool is used for performing a parsing task of a second document type, and a data capacity of the first document type is less than that of the second document type; calculating a first resource amount of the first resource pool and a second resource amount of the second resource pool in a current period according to the first average utilization and the second average utilization; and allocating the parsing resources of the plurality of file segments from the first resource pool and the second resource pool according to the document type.

[0010] Optionally, the calculating the first resource amount of the first resource pool and the second resource amount of the second resource pool in the current period according to the first average utilization and the second average utilization comprises: determining a total number of resources of the total resource pool; and calculating a first resource number ratio of the first resource pool and a second resource number ratio of the second resource pool in the current period by using the following formula: wherein k represents a smoothing coefficient, is the first average utilization, ​​​The second average utilization rate is used to calculate the first resource amount of the first resource pool and the second resource amount of the second resource pool in the current period respectively.

[0011] Optionally, the allocating the parsing resources of the plurality of file segments from the first resource pool and the second resource pool according to the document type comprises: determining the document type of the initial document corresponding to the plurality of file segments, wherein the document type comprises the first document type and the second document type; determining the used resource number, the waiting parsing user number and the future predicted newly added parsing user number of the current period for the first resource pool and the second resource pool respectively; calculating the first available resource amount of the first document type and the second available resource amount of the second document type by using the following formula:

[0012] wherein, the first resource amount or the second resource amount is taken, the used resource number of the corresponding resource pool is represented, the waiting parsing user number of the corresponding resource pool is represented, the future predicted newly added parsing user number of the corresponding resource pool is represented; the first available resource amount is allocated as the parsing resources of the file segments of the first document type in the plurality of file segments, and the second available resource amount is allocated as the parsing resources of the file segments of the second document type in the plurality of file segments.

[0013] Optionally, the extracting the document content slices of the plurality of file segments by using the parsing resources comprises: for each file segment, extracting the initial document content in the file segment according to the document type of the corresponding initial document; identifying the regular content and the special content in the initial document content, wherein the special content comprises the title and the website in the text; finding a preset cutting template matched with the document type; cutting the regular content at the sentence level by using the preset cutting template to obtain cutting segments, and outputting the cutting segments and the special content as the document content slices of the file segment.

[0014] ​​Optionally, the extracting the initial document content in the file segment according to the document type of the corresponding initial document comprises: if the document type of the corresponding initial document is a presentation document, generating a snapshot picture of the file segment by page; calling a pre-trained first network model to mark a text area of the snapshot picture; extracting first initial document content in the text area, wherein the first initial document content comprises text, table, and picture content; if the document type of the corresponding initial document is a table document, calling a pre-trained second network model to parse a table header and a data type of the file segment; extracting second initial document content in the file segment by row; if the document type of the corresponding initial document is a docx or md document, extracting a text paragraph, a table element, and a picture element in the file segment, using an OCR model to extract a text element in the picture element; determining the text paragraph, the table element, and the text element as third initial document content; if the document type of the corresponding initial document is a code document; finding a syntax tree matching the corresponding coding language; and extracting fourth initial document content in the file segment using the syntax tree.

[0015] Optionally, the vector data of the document content slice is calculated according to the slice content format of the document content slice, which comprises: analyzing the slice content format of the document content slice; and generating the vector data of the document content slice according to the slice content format.

[0016] Optionally, the vector data of the document content slice is generated according to the slice content format, which comprises: if the slice content format of the document content slice is a text format or a code format, calling a text vector model to generate first vector data of the document content slice; if the slice content format of the document content slice is a picture format, calling a multi-modal large model to quantitatively calculate the picture and the text in the picture of the document content slice, to obtain second vector data of the document content slice; and if the slice content format of the document content slice is a table format, extracting a table header and a data row of the document content slice; splicing the table header and the data row according to a preset specific format to obtain spliced data; and generating third vector data of the spliced data.

[0017] Optionally, the storing the vector data according to the slice content format of the document content slice comprises: if the slice content format of the document content slice is a text format, judging whether the vector data exceeds a threshold length; if the vector data exceeds the threshold length, generating summary data of the vector data, storing the vector data as the vector data of the document content slice in a database, and storing the summary data as first associated data of the vector data; if the slice content format of the document content slice is a picture format, obtaining picture internal text and picture uniform resource locator (URL) information of the document content slice; storing the vector data in the database, and storing the picture internal text and the URL information as second associated data of the vector data; if the slice content format of the document content slice is a table format, obtaining a field name and field content of the document content slice as a row unit; storing the vector data in the database, and storing the field name and the field content as third associated data of the vector data; and if the slice content format of the document content slice is a code format, generating mark information of the document content slice, wherein the mark information is used for marking programming language information of a code block; storing the vector data in the database, and storing the mark information as fourth associated data of the vector data.

[0018] According to another embodiment of the present application, a document parsing device is provided, comprising: a splitting module configured to split an initial document to be parsed into a plurality of file segments according to a document type of the initial document, wherein a data capacity of the file segments is less than a preset capacity; an allocation module configured to allocate parsing resources of the plurality of file segments according to resource historical utilization and the document type, wherein the parsing resources comprise a plurality of parsing nodes with document parsing capability, and each parsing node comprises a plurality of pipeline pipes connected in series; and a parsing module configured to extract document content slices of the plurality of file segments using the parsing resources, and to quantitatively calculate vector data of the document content slices according to a slice content format of the document content slices, and to store the vector data according to the slice content format of the document content slices.

[0019] According to another aspect of the embodiments of the present application, a storage medium is also provided, which comprises a stored program, and the program performs the above steps when running.

[0020] According to another aspect of the embodiments of the present application, an electronic device is also provided, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; the memory is used to store a computer program; and the processor is used to execute the steps in the above method by running the program stored in the memory.

[0021] The embodiment of the application further provides a computer program product containing instructions, which, when running on a computer, causes the computer to execute the steps in the above method.

[0022] Advantages of the application:

[0023] 1. In terms of logical implementation, a pipeline distribution subscription structure is adopted to disassemble the document parsing process, so that the processes in the document parsing process do not block each other, and a high-efficiency working mode of parallel stages and serial stages is realized, which can realize parsing while asking questions, and the knowledge processing at the bottom layer is refined to the sentence level, and adaptive processing is performed on different knowledge types, so that the accuracy of knowledge recall is greatly improved.

[0024] 2. Through the joint action of internal document parsing logic and external document coordination mechanism, two-layer resource management of user resource scheduling and document size resource scheduling is constructed, the timeliness of document parsing under different users and the stability of the whole system are effectively guaranteed, a fair, dynamic and efficient resource allocation strategy in a distributed environment is realized, the problem of single user task monopolizing resources is solved, and the situation of large documents and small documents occupying resources is avoided, so that the maximum parsing efficiency and the optimization of resource utilization are realized on the whole system.

[0025] 3. A vector data storage scheme for different content types is realized, and text, table, picture and code are respectively stored in a semantic vector database or a keyword matching database, and sentence-level, table row-level and code block-level delicacy storage is supported, so that the efficiency, accuracy and scalability of document knowledge parsing are improved. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings, which are included to provide a further understanding of the application and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application. In the drawings:

[0027] Figure 1 is a hardware structure block diagram of a server in an embodiment of the application;

[0028] Figure 2 is a flowchart of a document parsing method according to an embodiment of the application;

[0029] Figure 3 is a pipeline pipeline schematic diagram of a parsing node in an embodiment of the application;

[0030] Figure 4 is a scheduling structure schematic diagram of a parsing resource in an embodiment of the application;

[0031] Figure 5 is a schematic diagram of a parsing node distribution in an embodiment of the application;

[0032] Figure 6 is a detailed flowchart of a distributed document parsing quantification method in an embodiment of the present application;

[0033] Figure 7 is a structural block diagram of a document parsing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to enable persons skilled in the art to better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0035] It should be noted that the terms “first”, “second”, and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms “include” and “have” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, product, or device including a series of steps or units does not necessarily have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product, or device.

[0036] Embodiment 1

[0037] The method embodiment provided in the embodiment of the present application can be executed in a server, a processor, or a similar processing device. Taking the case of running on a server as an example, Figure 1 is a hardware structural block diagram of a server according to an embodiment of the present application. As shown in Figure 1 , the server can include one or more (only one is shown in Figure 1 ) processor 102 (the processor 102 can include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the above-mentioned server can also include a transmission device 106 for communication function and an input and output device 108. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the above-mentioned server. For example, the server can also include more than one processor, more than one memory, more than one transmission device, and more than one input and output device.Figure 1 more or less components than those shown, or configured differently from those shown. Figure 1

[0038] The memory 104 can be used to store server programs, such as software programs of application software and modules, such as a server program corresponding to the document parsing method of a server in an embodiment of the present application. The processor 102 can execute various functional applications and data processing, i.e., implement the method described above, by running the server program stored in the memory 104. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely disposed relative to the processor 102, which can be connected to the server through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0039] The transmission device 106 is configured to receive or send data via a network. Examples of the network include, but are not limited to, a wireless network provided by a communication provider of the server. In one example, the transmission device 106 includes a network adapter (NIC), which can be connected to other network devices through a base station so as to be able to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is configured to communicate with the Internet in a wireless manner.

[0040] In the present embodiment, a document parsing method is provided, Figure 2 is a flowchart of a document parsing method according to an embodiment of the present application, as shown in Figure 2 The flowchart includes the following steps:

[0041] In step S201, the initial document is divided into a plurality of file segments according to the document type of the initial document to be parsed, wherein the data capacity of the file segment is less than a preset capacity.

[0042] The initial document can be a document of any format to be archived and stored.

[0043] Optionally, before the initial document is divided into a plurality of file segments according to the document type of the initial document to be parsed, the document type that cannot be recognized or processed can be converted into a PDF format, such as converting the initial document to be parsed of the doc and ppt type into a PDF document.

[0044] ​In step S202, parsing resources of the plurality of file segments are allocated according to the resource historical utilization rate and the document type, wherein the parsing resources include a plurality of parsing nodes with document parsing capability, and each parsing node includes a plurality of pipeline pipes connected in series.

[0045] Figure 3 FIG. 4 is a schematic diagram of a pipeline pipe of a parsing node in an embodiment of the present application. The parsing node is composed of four functional modules: a document format conversion module, a document content extraction module, a document content quantization module, and a document content storage module.

[0046] The document format conversion module is configured to uniformly convert input documents into a specified standard format, such as Markdown (MD) format, which can be recognized by subsequent models, so as to facilitate subsequent processing. The document content extraction module is configured to finely segment the document content on the basis of the MD format, and classify and label the document content according to the content type. The document content quantization module is configured to perform vectorization calculation on the extracted knowledge content, so as to support subsequent knowledge similarity matching and recall operations. The document content storage module is responsible for storing the quantized knowledge content, and adopts a differentiated storage mode according to different types.

[0047] In step S203, document content slices of the plurality of file segments are extracted using the parsing resources, vector data of the document content slices is calculated according to the slice content format of the document content slices, and the vector data is stored according to the slice content format of the document content slices.

[0048] The scheme of the embodiment can be applied in scenarios such as knowledge base construction, retrieval, and knowledge question answering.

[0049] Through the above steps, the initial document is segmented into a plurality of file segments according to the document type of the initial document to be parsed, wherein the data capacity of the file segments is less than a preset capacity; the parsing resources of the plurality of file segments are allocated according to the resource historical utilization rate and the document type, wherein the parsing resources include a plurality of parsing nodes with document parsing capability, and each parsing node includes a plurality of pipeline pipes connected in series; and the document content slices of the plurality of file segments are extracted using the parsing resources, vector data of the document content slices is calculated according to the slice content format of the document content slices, and the vector data is stored according to the slice content format of the document content slices. The technical problem of low document parsing efficiency in the prior art is solved, a fair, dynamic, and efficient resource allocation strategy in a distributed environment is implemented, the problem of single user task monopolizing resources is solved, the situation of large documents and small documents monopolizing resources is avoided, and thus the parsing efficiency maximization and resource utilization optimization of the whole system are achieved, and the document parsing efficiency and storage granularity are improved.

[0050] In one embodiment of the present embodiment, the step of splitting the initial document into several file segments according to the document type of the initial document to be parsed comprises: identifying the document type according to the suffix information of the initial document to be parsed, wherein the document type comprises one of the following: a programming code document, a single-page text document, and a multi-page text document; if the document type is a programming code document, determining the programming code document as one file segment; and if the document type is a single-page text document or a multi-page text document, splitting the initial document into multiple file segments.

[0051] Optionally, the single-page text document can be a csv, xlsx, xls, docx, txt, md document, the multi-page text document can be a pdf, pptx document, and the programming code document can be a py, java, cpp, c, h, sh, bat, html, vue, js, jsx, css, scss, sass, ts, tsx document with a suffix.

[0052] The document type to be parsed is obtained, and a corresponding method is used to determine whether the document to be parsed is a large document according to different document types. If the document is a large document, it is further split. Specifically: if the document type is csv, xlsx, xls, docx, txt, md, the large and small documents are further divided according to whether the total number of document content rows or the total number of data blocks exceeds a preset threshold. If the number of rows exceeds the preset number of rows, the document is determined to be a large document, otherwise it is a small document. Further, if it is a large document, the document to be parsed is split into multiple small documents of the same document type according to the preset number of rows. If the document type is pdf, pptx, the large and small documents are further divided according to whether the total number of pages of the document exceeds a preset number of pages. If the number of rows exceeds the preset number of rows, the document is determined to be a large document, otherwise it is a small document. Further, if it is a large document, the document to be parsed is split into multiple small documents of the same document type according to the preset number of pages. If the document type is py, java, cpp, c, h, sh, bat, html, vue, js, jsx, css, scss, sass, ts, tsx programming source code, the document is directly classified as a small document.

[0053] In one example, the splitting the initial document into multiple file segments includes: if the document type is a single-page text document, determining whether the total number of rows or the total number of data blocks of the initial document exceeds a preset threshold; if the total number of rows or the total number of data blocks of the initial document exceeds the preset threshold, splitting the initial document into multiple first file segments, wherein the number of document rows or the number of data blocks of each first file segment is less than the preset threshold; if the document type is a multi-page text document, determining whether the total number of pages of the initial document exceeds a preset number of pages; if the total number of pages of the initial document exceeds the preset number of pages, splitting the initial document into multiple second file segments, wherein the number of document pages of each second file segment is less than the preset number of pages.

[0054] In the present example, the size threshold of each type of document is preset, the size threshold of the PDF document is set to 500 pages in terms of the number of page codes, the size threshold of the PPTX document is set to 100 pages in terms of the number of page codes, the size threshold of the TXT text file is set to 5000 lines in terms of the number of lines, the size threshold of the DOCX document is set to 2000 blocks in terms of the number of document content structure blocks, and the size threshold of the table type document such as csv and xlsx is set to 2000 lines in terms of the number of lines. The document exceeding the threshold is marked as a large document, otherwise it is marked as a small document. In addition, the py, java and other coding type files are directly marked as small documents. The above threshold size can be appropriately adjusted according to actual conditions.

[0055] In the process of obtaining whether each document meets the set threshold, the document is simultaneously subjected to cutting and subdivision processing, i.e., the original document content is cut into multiple file segments of the same type according to the threshold, which facilitates subsequent pipeline scheduling, increases fault tolerance and reduces internal waiting time of the process. The file segment cutting process includes: for a TXT text file, cutting is performed in units of 500 lines; for a DOCX file, cutting is performed in units of 50 structure blocks; for a PPTX file, cutting is performed in units of 50 pages; for a PDF file, cutting is performed in units of 50 pages; for a table type file, cutting is performed in units of 50 lines; and for a code type file, cutting is performed in units of 10 code blocks. When processing the table, the header range is first analyzed intelligently using a large model to avoid content extraction errors due to the first row of a special form of table not being a column name. After cutting, multiple file segments containing all the original document information are obtained.

[0056] In the embodiment, the allocating the parsing resources of the plurality of file segments according to the resource historical utilization rate and the document type comprises: calculating a first average utilization rate of a first resource pool and a second average utilization rate of a second resource pool in a previous historical period, wherein the total resource pool comprises the first resource pool and the second resource pool, the first resource pool is used for performing a parsing task of a first document type, the second resource pool is used for performing a parsing task of a second document type, and the data capacity of the first document type is smaller than that of the second document type; calculating a first resource amount of the first resource pool and a second resource amount of the second resource pool in a current period according to the first average utilization rate and the second average utilization rate; and allocating the parsing resources of the plurality of file segments from the first resource pool and the second resource pool according to the document type.

[0057] The first resource amount is the resource amount of the parsing resources in the first resource pool, that is, the number of parsing nodes, and the second resource amount is the resource amount of the parsing resources in the second resource pool, that is, the number of parsing nodes. The number of parsing nodes can be determined by counting the number of processes of the parsing nodes (each parsing node corresponds to a process) in the first resource pool and the second resource pool and reading the storage space occupied by the first resource pool and the second resource pool.

[0058] In one example, the calculating the first resource amount of the first resource pool and the second resource amount of the second resource pool in the current period according to the first average utilization rate and the second average utilization rate comprises: determining the total resource number of the total resource pool; and calculating the first resource number proportion of the first resource pool in the current period and the second resource number proportion of the second resource pool in the current period by using the following formula wherein k represents a smoothing coefficient, the first average utilization rate is the second average utilization rate; and the first resource amount of the first resource pool and the second resource amount of the second resource pool in the current period are calculated respectively by using the first resource number proportion, the second resource number proportion and the total resource number.

[0059] In the example, the smoothing coefficient k is used to describe the degree of attention to historical data in the data processing or model prediction process, and is used to describe the degree of attention to the resource allocation of the first resource pool and the second resource pool in the current period to the resource allocation of the first resource pool and the second resource pool in the historical period in the embodiment. The value range of the smoothing coefficient k is 0-1, the greater the value, the higher the degree of attention, and the greater the influence of the resource allocation of the first resource pool and the second resource pool in the historical period on the current period.

[0060] ​​​The first resource quantity ratio is the ratio between the first resource quantity of the first resource pool and the total resource quantity of the total resource pool, and correspondingly, the second resource quantity ratio is the ratio between the second resource quantity of the second resource pool and the total resource quantity of the total resource pool. For example, the total resource quantity (the number of parsing nodes) in the total resource pool is 100, the first resource quantity of the first resource pool is 40, and the second resource quantity of the second resource pool is 60, and then the first resource quantity ratio is 40%, and the second resource quantity ratio is 60%.

[0061] When the system detects that the recent small document (the first document type) task is significantly more than the large document (the second document type) task, the small document resource pool (the first resource pool) is automatically expanded, so as to ensure the rapid parsing ability of the small document; on the contrary, when the proportion of the large document task increases, the system dynamically increases the proportion of the large document resource pool (the second resource pool), so as to ensure that the long-time task can obtain sufficient parsing resources.

[0062] In one example, the allocating the parsing resources of the plurality of file segments from the first resource pool and the second resource pool according to the document type includes: determining the document type of the initial document corresponding to the plurality of file segments, wherein the document type includes the first document type and the second document type; determining the used resource quantity, the waiting parsing user quantity, and the future predicted newly added parsing user quantity of the current period for the first resource pool and the second resource pool respectively; calculating the first available resource quantity of the first document type and the second available resource quantity of the second document type by using the following formula

[0063] Optionally,

[0064] When the calculation result r=0, the task will automatically enter the waiting parsing queue until the system releases the idle resource and then re-schedules.

[0065] ​​​​​​​​​​The embodiment realizes the fair, dynamic and efficient resource allocation strategy in the distributed environment by the internal document analysis logic and the external document cooperation mechanism, adopts the two-layer resource management mechanism of user identity scheduling and document size scheduling, solves the problem of single user task monopolizing resources, avoids the situation of large documents and small documents occupying resources, and thus realizes the maximum analysis efficiency and the optimization of resource utilization in the whole system.

[0066] Figure 4 is a scheduling structure diagram of analysis resources in the embodiment, including a registration center, a main analysis node and multiple slave analysis nodes, all analysis nodes are automatically registered to the registration center after being started, the registration center is used for providing system analysis node registration service and automatically electing a main node; after the current main node is hung up, the system triggers to elect a new main node; the main node is responsible for global small file and large file analysis resource dynamic balance and task scheduling, is responsible for user analysis resource dynamic balance, and mainly responsible for document analysis task allocation, task retry and task migration; the slave node is responsible for document conversion, extraction, quantization and storage pipeline work. When the slave node fails, the main node also executes the failure retry and task migration strategy; when the slave node executes large document splitting, document content slicing and slice content quantization, the intermediate state and the final state in the analysis process are recorded, the intermediate state refers to temporary data generated in each step, and the final state refers to whether each step is successful or failed; the failure retry includes analysis exception retry and analysis timeout retry, the analysis exception retry includes IO exception, network exception and the like, the exception retry is retried within a preset exception retry number, and will not be retried after exceeding; the analysis timeout retry refers to that the analysis process has no error prompt, and has no state update within a preset time, and will be retried. The task migration refers to that when the document analysis task of a document is allocated to one or more analysis nodes, the one or more analysis nodes are down or unable to process the document analysis task, the system will automatically migrate to other normal document analysis nodes. The task retry continues to execute from the place where the last task execution is interrupted, and avoids repeated execution of the steps that have been executed.

[0067] The master node is in the user identity scheduling layer, and the system builds a global scheduler, which takes the user workspace as the unique identifier of the user identity, and the tasks of each user are stored in independent queues. The scheduler polls through the global task queue and distributes tasks according to the task submission order. When there is only a single user in the system, the user is allowed to execute multiple tasks concurrently, and the upper limit is the preset maximum parallelism (such as a maximum of 3 tasks per user); when there are multiple users in the system, the scheduler automatically switches to a fair mode, ensuring that each user can only be assigned one execution task within one polling cycle, thereby avoiding excessive occupation of global resources by a single user. The scheduler controls the total concurrency of the system through a global semaphore, and the upper limit is the sum of the thread numbers of the large document resource pool and the small document resource pool. If all resources are currently occupied, the newly submitted task enters the user parsing queue. When an existing task is executed and the resource is released, the scheduler will redistribute the resource from the parsing queue according to the fair strategy. This mechanism ensures that the system always maintains a stable and controllable load level under large-scale concurrency. In the document size scheduling layer, the resource pool is divided into a small document resource pool and a large document resource pool according to the document size, and the two are independent and do not interfere with each other. Specifically, after determining the document size through the aforementioned internal document parsing logic, the size document resource linear pool is scheduled in the pipeline parsing. The small document resource pool is suitable for lightweight document types, with a large number of threads configured to ensure that small tasks can be quickly responded; the large document resource pool is suitable for large document types with a large number of pages, with a small number of threads configured but more concentrated resources allocated to a single task to ensure the stability of the parsing process. When scheduling tasks, the master node will automatically allocate tasks to the corresponding resource pool according to the size and type of the document, thereby realizing the isolation and optimized execution of tasks of different sizes.

[0068] The scheme of the embodiment adopts a distributed document parsing strategy, and sets up two layers of mechanisms, i.e., a user resource scheduling manager and a document size resource scheduling manager, to work cooperatively, so as to ensure the fairness and efficiency of the system under the condition of concurrent submission of multiple users and multiple types of documents.

[0069] In the embodiment, the extracting the document content slices of the plurality of file segments by using the parsing resource includes: for each file segment, extracting initial document content in the file segment according to a document type of a corresponding initial document; identifying regular content and special content in the initial document content, wherein the special content includes a title and a website in a body; finding a preset cutting template matched with the document type; cutting the regular content at a sentence level by using the preset cutting template to obtain cutting segments; and outputting the cutting segments and the special content as the document content slices of the file segment.

[0070] Optionally, extracting the initial document content in the file segment according to the document type corresponding to the initial document comprises: if the document type corresponding to the initial document is a presentation document, generating a snapshot picture of the file segment by page; calling a pre-trained first network model to mark a text area of the snapshot picture; extracting first initial document content in the text area, wherein the first initial document content comprises text, table, and picture content; if the document type corresponding to the initial document is a table document, calling a pre-trained second network model to parse a table header and data type of the file segment; extracting second initial document content in the file segment by row; if the document type corresponding to the initial document is a docx or md document, extracting a text paragraph, a table element, and a picture element in the file segment, using an OCR model to extract a text element in the picture element; determining the text paragraph, the table element, and the text element as third initial document content; if the document type corresponding to the initial document is a code document; finding a syntax tree matching the corresponding coding language; and extracting fourth initial document content in the file segment using the syntax tree.

[0071] The first network model is a multi-modal large model, which inputs a picture (such as the snapshot picture in the embodiment), identifies the picture by instructing the model to behave, and outputs the content of each region in the picture, and then marks the recognized text area in the form of a text box. The second network model is a general language large model, which reads the first few rows of data of a table, inputs the data into the model, and identifies the table header and data type by instructing the model.

[0072] Optionally, the presentation document can be a pdf or pptx document, the table document can be a csv, xlsx, or xls document, and the code document can be a py, java, cpp, c, h, sh, bat, html, vue, js, jsx, css, scss, sass, ts, or tsx document.

[0073] If the document type corresponding to the initial document is a large document, the parsing task of the file segment is distributed to the parsing nodes (slave nodes) according to the idle state and the minimum waiting time; if the document type corresponding to the initial document is a small document, the document parsing task is randomly assigned to a parsing node according to the idle state and the minimum waiting time.

[0074] After extracting the initial document content in the file segment according to the document type corresponding to the initial document, invalid characters and invalid data can be removed according to a preset rule, and cross-page paragraphs and table splicing can be completed; when the regular content is cut at the sentence level using the preset cutting template: the text content is matched with the preset cutting template according to the rule, and the sentence-level delicacy cutting is performed, and the special form content such as the title and the website is protected, to form a document content slice.

[0075] In the embodiment, the vector data of the document content slice is generated according to the slice content format of the document content slice, which includes: analyzing the slice content format of the document content slice; and generating the vector data of the document content slice according to the slice content format.

[0076] In one example, the vector data of the document content slice is generated according to the slice content format, which includes: if the slice content format of the document content slice is a text format or a code format, calling a text vector model to generate first vector data of the document content slice; if the slice content format of the document content slice is a picture format, calling a multi-modal large model to quantitatively calculate the picture and the text in the picture of the document content slice to obtain second vector data of the document content slice; and if the slice content format of the document content slice is a table format, extracting a table header and data rows of the document content slice; splicing the table header and the data rows according to a preset specific format to obtain spliced data; and generating third vector data of the spliced data.

[0077] In the embodiment, the vector data is stored according to the slice content format of the document content slice, which includes: if the slice content format of the document content slice is a text format, determining whether the vector data exceeds a threshold length; if the vector data exceeds the threshold length, generating summary data of the vector data, storing the vector data as the vector data of the document content slice in a database, and storing the summary data as first associated data of the vector data; if the slice content format of the document content slice is a picture format, obtaining text in the picture and picture uniform resource locator (URL) information of the document content slice; storing the vector data in the database, and storing the text in the picture and the URL information as second associated data of the vector data; if the slice content format of the document content slice is a table format, obtaining field names and field contents of the document content slice in a row unit; storing the vector data in the database, and storing the field names and the field contents as third associated data of the vector data; and if the slice content format of the document content slice is a code format, generating marking information of the document content slice, wherein the marking information is used to mark programming language information of a code block; storing the vector data in the database, and storing the marking information as fourth associated data of the vector data.

[0078] The embodiment includes a semantic vector database and a keyword database, which are respectively searched by semantic vectors and natural languages. When stored, the semantic vector database and the keyword matching database are used for respectively storing the parsed content of the document, and further subdivision is made in combination with the content slice type. Specifically, if the current document content slice is a text type, the sentence level delicacy is maintained in the database storage, and for the sentence exceeding the threshold length, the summary information thereof is generated by using a large model and stored in the database. If the document content slice is a picture type, the content of the OCR text recognition thereof is stored, and the file cloud URL information is additionally stored, so as to facilitate subsequent display. If the document content slice is a table type, the field name and the field content thereof are stored in the database in a line unit. If the document content slice is a code type, the code block is stored in the database in a unit, and the programming language information of the code block is marked.

[0079] After the document size is determined, the multiple self-information blocks are distributed to respective corresponding pipelines according to the type of the file itself for analysis processing. The files of the types of pdf, docx, pptx, doc and ppt are uniformly transported into a MarkDown style processing pipeline. The txt text type file is transported into a text style processing pipeline, the table type file is transported into a table style processing pipeline, and the picture type is transported into a picture style processing pipeline. Figure 5 FIG. 1 is a schematic diagram of distributing and analyzing nodes in the embodiment of the application. After the document is cut, the file fragments are distributed to the analysis nodes for pipeline processing.

[0080] Taking the MD style data as an example, the analysis nodes of the pipeline processing are divided into five separate pipe combinations, different pipes are respectively responsible for different processing tasks, and the pipes send messages through subscription. They are respectively: a document content to MarkDown pipe, a MarkDown data formatting pipe, a MarkDown data cleaning pipe, a document knowledge extraction pipe, a knowledge vectorization pipe and a database storage pipe. The cooperation mode between the pipes is: the processing in the pipe is serial, and the processing between the pipes is parallel. That is, after each pipe completes the data processing of the current batch, it is immediately sent to the next pipe, and the next batch of data is immediately processed. After receiving the data, the next pipe is also immediately processed. The pipeline structure changes the originally blocked single document processing stages to be performed at the same time, which greatly shortens the document processing time.

[0081] In the above examples, the role of the document content to MarkDown pipeline includes: converting pdf, doc, docx, ppt, pptx type files into MD format through a third-party tool for easy understanding by the large model, and processing each time is the self-information block after the aforementioned document segmentation. At the same time, for the original document, a snapshot picture is generated by page, and the large model is used to mark the page header, page footer, page notes, background, and text area, etc. information, combined with the subsequent document knowledge extraction pipeline to ensure the correctness of information extraction and retain the correct structure information.

[0082] The MarkDown data formatting pipeline and the MarkDown data cleaning pipeline are to unify and clean the MD format data obtained after converting different types of document sources;

[0083] The document knowledge extraction pipeline is to classify the content at the paragraph level and split the knowledge at the sentence level based on the MD data converted by the sub-information block: the paragraph classification is to divide the content of each paragraph of MD into one of the following: title, picture, mathematical formula, text, table, and code. The sentence-level splitting is a refinement based on paragraph classification. For titles, the knowledge is marked in the original style; for pictures, call OCR for text recognition; for mathematical formulas, use LaTex style; for text, perform sentence-level splitting based on rules, including: special coherent format protection, data cleaning, cutting based on common ending punctuation marks, sentence splitting based on specified threshold length, and short sentence merging. For tables (here, tables in MD format), store them as multiple word tables according to MD format and specified row number; for code, store it according to the syntax tree in complete code function units.

[0084] After the knowledge extraction pipeline, the original knowledge content in the form of sub-information block is further subdivided into sentence level according to different paragraph types, with more delicate division. Then, it is fed into the knowledge vectorization pipeline, which processes a set of BatchSize number of knowledge at a time, and calls different vectorization models according to the different types of knowledge slices. If the slice content is text, use the text vectorization mode to quantify the text; if the slice content is a picture, use a multi-modal large model to quantify the picture and the text extracted by OCR; if the slice content is a table, concatenate the table header and data rows according to a pre-set specific format and then use the text vectorization model to quantify. Different quantization models ensure that different types of knowledge have a certain degree of distinction, which is convenient for subsequent recall.

[0085] The vector data after vectorization enters the database storage pipeline for data storage. After the sentence-level knowledge content slice is quantized, the original content and the vectorized content are combined and stored in the semantic vector database and the keyword matching database. During storage, further subdivision is done in combination with the content slice type. Specifically:

[0086] If the current document content slice is of a text type, the sentence-level delicacy is maintained in the database storage, and for sentences exceeding a threshold length, the summary information thereof is generated using a large model and stored in the database. If the document content slice is of a picture type, the file cloud URL information thereof is additionally stored while storing the content thereof recognized by OCR, facilitating subsequent display; if the document content slice is of a table type, the field name and field content thereof are stored in the database in a row unit; if the document content slice is of a code type, the code block is stored in the database in a unit, and the programming language information of the code block is marked;

[0087] The embodiment provides a distributed document quantization analysis method and system, which can intelligently split a document according to a document type and size, distribute an analysis task to distributed analysis nodes according to idle and minimum waiting time balancing, and realize quantization processing of multi-modal content (text, table, picture and code) slices in combination with a large model. Meanwhile, the application designs a vectorization storage scheme for different content types, stores the text, table, picture and code in a semantic vector database or a keyword matching database, and supports sentence-level, table row-level and code block-level delicacy storage, thereby improving the efficiency, accuracy and scalability of document knowledge analysis.

[0088] Figure 6 is a detailed flowchart of a distributed document analysis quantization method in the embodiment of the application, and the flowchart includes the following document distributed analysis steps: obtaining a document to be analyzed by a user; identifying a document format and performing necessary conversion on the document format, converting the target document into a PDF format; judging a document size and performing classification, splitting a large file into small file segments according to different file formats; obtaining document analysis resources available to the user; judging whether the user terminal has available resources; if not, entering a user analysis waiting queue, and if yes, distributing and extracting document content according to different document formats; using different methods to quantize the content according to different content formats; and storing the original content of the analyzed document and the corresponding quantized content.

[0089] The embodiment effectively guarantees the timeliness of document analysis of different users and the stability of the whole system by constructing two layers of resource management, i.e., user resource scheduling and document size resource scheduling, and reduces the analysis time from 46 hours to 4 hours for a 6000-page PDF document compared with a non-distributed analysis method. In the analysis implementation logic, a pipeline distribution subscription structure is adopted to disassemble the document analysis process, so that the processes in the document analysis process do not block each other, a high-efficiency working mode of parallel stages and serial stages is realized, and the document analysis and question answering can be realized simultaneously. In the bottom knowledge processing, the accuracy of knowledge recall is greatly improved by refining to the sentence level and adapting to different knowledge types.

[0090] Those skilled in the art can clearly understand that the method according to the above-mentioned embodiments can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better implementation. Based on such understanding, the technical solutions of the present application or the part of the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, or network device, etc.) execute the method described in each embodiment of the present application.

[0091] Embodiment 2

[0092] In this embodiment, a document parsing device is also provided, which is used to realize the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that realizes a predetermined function. Although the device described in the following embodiments is preferably realized in software, hardware, or a combination of software and hardware is also possible and is contemplated.

[0093] Figure 7 is a structural block diagram of a document parsing device according to an embodiment of the present application, as shown in Figure 7 , the device comprises:

[0094] The splitting module 71 is configured to split the initial document into a plurality of file segments according to the document type of the initial document to be parsed, wherein the data capacity of the file segments is less than a preset capacity.

[0095] The allocation module 72 is configured to allocate parsing resources of the plurality of file segments according to resource historical utilization and the document type, wherein the parsing resources comprise a plurality of parsing nodes with document parsing capability, and each parsing node comprises a plurality of pipeline channels connected in series.

[0096] The parsing module 73 is configured to extract document content slices of the plurality of file segments by using the parsing resources, and to quantitatively calculate vector data of the document content slices according to the slice content format of the document content slices, and to store the vector data according to the slice content format of the document content slices.

[0097] It should be noted that each of the above-mentioned modules can be realized by software or hardware. For the latter, the following implementation methods can be used, but are not limited thereto: all the above-mentioned modules are located in the same processor; or each of the above-mentioned modules is located in different processors in any combination.

[0098] Embodiment 3

[0099] The embodiment of the present application also provides a storage medium, wherein the storage medium stores a computer program, and the computer program is configured to execute the steps in any of the method embodiments when running.

[0100] Optionally, in the embodiment, the storage medium is configured to store a computer program for executing the following steps:

[0101] S1, splitting an initial document to be parsed into a plurality of file segments according to a document type of the initial document, wherein a data capacity of the file segments is less than a preset capacity;

[0102] S2, allocating parsing resources of the plurality of file segments according to a resource historical utilization rate and the document type, wherein the parsing resources include a plurality of parsing nodes with document parsing capabilities, and each parsing node includes a plurality of pipeline channels connected in series;

[0103] S3, extracting document content slices of the plurality of file segments by using the parsing resources, and quantitatively calculating vector data of the document content slices according to a slice content format of the document content slices, and storing the vector data according to the slice content format of the document content slices.

[0104] Optionally, in the embodiment, the storage medium can include but is not limited to a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk and various storage media that can store a computer program.

[0105] The embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to execute the steps in any of the method embodiments.

[0106] Optionally, the electronic device can further include a transmission device and an input and output device, wherein the transmission device is connected with the processor, and the input and output device is connected with the processor.

[0107] Optionally, in the embodiment, the processor is configured to execute the following steps by using the computer program:

[0108] S1, splitting an initial document to be parsed into a plurality of file segments according to a document type of the initial document, wherein a data capacity of the file segments is less than a preset capacity;

[0109] S2, allocating parsing resources for the plurality of file segments according to resource historical utilization and the document type, wherein the parsing resources comprise a plurality of parsing nodes with document parsing capability, and each parsing node comprises a plurality of pipeline pipes connected in series;

[0110] S3, extracting document content slices of the plurality of file segments by using the parsing resources, and quantitatively calculating vector data of the document content slices according to slice content formats of the document content slices, and storing the vector data according to the slice content formats of the document content slices.

[0111] Optionally, specific examples in the embodiments can refer to the examples described in the foregoing embodiments and optional implementation manners, and the embodiments will not be described here again.

[0112] The device embodiments described above are only schematic, wherein the units illustrated as separate components may or may not be physically separate, and the components illustrated as units may or may not be physical units, that is, they may be located in one place or distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0113] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0114] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises", "comprising", "includes", "including" and "has" are inclusive and therefore specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order in which they are described, unless specifically indicated as such. It is also to be understood that additional or alternative steps can be employed.

[0115] The foregoing is considered as illustrative only of the principles of the application. Numerous modifications and changes will readily occur to those skilled in the art, and it is intended to embrace all such modifications and changes that fall within the scope of the application. Accordingly, the application is not to be restricted in scope to the specific embodiments disclosed herein but is to be accorded the full scope that the principles and novel features request appropriately granted.

Claims

1. A method of parsing a document, characterized by, include: The initial document is divided into several file segments according to the document type of the initial document to be parsed, wherein the data capacity of the file segments is less than the preset capacity; Parsing resources for the plurality of file fragments are allocated based on historical resource utilization and the document type, wherein the parsing resources include multiple parsing nodes with document parsing capabilities, and each parsing node includes multiple pipelines connected in series. The document content slices of the plurality of file fragments are extracted using the parsing resources, and the vector data of the document content slices are quantized according to the slice content format of the document content slices, and the vector data is stored according to the slice content format of the document content slices; The allocation of parsing resources for the plurality of file segments based on historical resource utilization and the document type includes: calculating the first average utilization rate of the first resource pool and the second average utilization rate of the second resource pool in the previous historical period, wherein the total resource pool includes the first resource pool and the second resource pool, the first resource pool is used to perform parsing tasks for the first document type, the second resource pool is used to perform parsing tasks for the second document type, and the data capacity of the first document type is smaller than that of the second document type; calculating the first resource quantity of the first resource pool and the second resource quantity of the second resource pool in the current period based on the first average utilization rate and the second average utilization rate; and allocating parsing resources for the plurality of file segments from the first resource pool and the second resource pool according to the document type.

2. The method of claim 1, wherein, The initial document is divided into several file fragments based on its document type, including: The document type is identified based on the suffix information of the initial document to be parsed, wherein the document type includes one of the following: programming code document, single-page text document, and multi-page text document; If the document type is a programming code document, the programming code document is determined as a file fragment; if the document type is a single-page text document or a multi-page text document, the initial document is divided into multiple file fragments.

3. The method of claim 2, wherein, The initial document is divided into multiple file fragments, including: If the document type is a single-page text document, determine whether the total number of lines or the total number of data blocks in the initial document exceeds a preset threshold; if the total number of lines or the total number of data blocks in the initial document exceeds the preset threshold, divide the initial document into multiple first file segments, wherein the number of document lines or the number of data blocks in each first file segment is less than the preset threshold. If the document type is a multi-page text document, determine whether the total number of pages in the initial document exceeds the preset number of pages; if the total number of pages in the initial document exceeds the preset number of pages, divide the initial document into multiple second file segments, wherein the number of pages in each second file segment is less than the preset number of pages.

4. The method of claim 1, wherein, The calculation of the first resource quantity of the first resource pool and the second resource quantity of the second resource pool for the current period based on the first average utilization rate and the second average utilization rate includes: Determine the total number of resources in the total resource pool; The first resource number proportion of the first resource pool in the current period is calculated by using the following formula and the second resource number proportion of the second resource pool : ; wherein k denotes a smoothing coefficient, is the first average utilization, is the second average utilization; The first resource quantity of the first resource pool and the second resource quantity of the second resource pool in a current period are calculated respectively by using the first resource quantity ratio, the second resource quantity ratio and the total resource quantity.

5. The method of claim 1, wherein, Allocating parsing resources of the plurality of file segments from the first resource pool and the second resource pool according to the document type includes: Determining the document type of the initial document corresponding to the plurality of file segments, wherein the document type includes the first document type and the second document type; For the first resource pool and the second resource pool, respectively determining the used resource quantity, the waiting parsing user quantity and the future predicted newly added parsing user quantity in a current period; The first available resource amount of the first document type is calculated using the following formula and the second available resource amount of the second document type : wherein, taking the first resource amount or the second resource amount, represents the used resource number of the corresponding resource pool, represents the number of users waiting for resolution of the corresponding resource pool, represents the future predicted number of newly added resolution users of the corresponding resource pool, represents a positive integer; The first available resource quantity is allocated as the parsing resource of the file segment of the first document type in the plurality of file segments, and the second available resource quantity is allocated as the parsing resource of the file segment of the second document type in the plurality of file segments.

6. The method of claim 1, wherein, Extracting the document content slice of the plurality of file segments by using the parsing resource includes: For each file segment, extracting the initial document content in the file segment according to the document type of the corresponding initial document; Identifying the regular content and the special content in the initial document content, wherein the special content includes the title and the website in the text; Finding a preset cutting template matching the document type; Cutting the regular content at the sentence level by using the preset cutting template to obtain a cutting segment, and outputting the cutting segment and the special content as the document content slice of the file segment.

7. The method of claim 6, wherein, Extracting the initial document content in the file segment according to the document type of the corresponding initial document includes: If the document type of the corresponding initial document is a demonstration document, generating a snapshot picture by page; calling a pre-trained first network model to mark a text area of the snapshot picture; extracting first initial document content in the text area, wherein the first initial document content includes text, table and picture content; If the document type of the corresponding initial document is a table document, calling a pre-trained second network model to parse the table header and data type of the file segment; extracting second initial document content in the file segment by row; If the document type of the corresponding initial document is a docx or md document, extracting text paragraphs, table elements and picture elements in the file segment, using an OCR model to extract text elements in the picture elements; determining the text paragraphs, table elements and the text elements as third initial document content; If the document type of the corresponding initial document is a code document; finding a syntax tree matching the corresponding coding language; extracting fourth initial document content in the file segment by using the syntax tree.

8. The method of claim 1, wherein, Quantitative calculation of vector data of the document content slice according to the slice content format of the document content slice includes: Parsing the slice content format of the document content slice; Generating vector data of the document content slice according to the slice content format.

9. The method of claim 8, wherein, Generating vector data of the document content slice according to the slice content format includes: If the slice content format of the document content slice is a text format or a code format, a text vector model is invoked to generate first vector data of the document content slice; If the slice content format of the document content slice is a picture format, a multi-modal large model is invoked to quantitatively calculate the picture and the text in the picture of the document content slice, to obtain second vector data of the document content slice; If the slice content format of the document content slice is a table format, a table header and a data row of the document content slice are extracted; the table header and the data row are spliced according to a preset specific format to obtain spliced data; third vector data of the spliced data is generated.

10. The method of claim 1, wherein, The vector data is stored according to the slice content format of the document content slice, including: If the slice content format of the document content slice is a text format, it is judged whether the vector data exceeds a threshold length; if the vector data exceeds the threshold length, summary data of the vector data is generated, stored as vector data of the document content slice in a database, and stored as first associated data of the vector data; If the slice content format of the document content slice is a picture format, text in the picture of the document content slice and picture uniform resource locator system URL information are obtained; the vector data is stored in the database, and the text in the picture and the URL information are stored as second associated data of the vector data; If the slice content format of the document content slice is a table format, field names and field contents of the document content slice are obtained as row units; the vector data is stored in the database, and the field names and the field contents are stored as third associated data of the vector data; If the slice content format of the document content slice is a code format, marking information of the document content slice is generated, wherein the marking information is used to mark programming language information of a code block; the vector data is stored in the database, and the marking information is stored as fourth associated data of the vector data.

11. A document parsing apparatus, characterized by comprising: including: The splitting module is configured to split the initial document into a plurality of file segments according to a document type of the initial document to be parsed, wherein a data capacity of the file segments is less than a preset capacity; The allocation module is configured to allocate parsing resources of the plurality of file segments according to resource historical utilization and the document type, wherein the parsing resources include a plurality of parsing nodes with document parsing capabilities, and each parsing node includes a plurality of pipeline pipes connected in series; The parsing module is configured to extract document content slices of the plurality of file segments by using the parsing resources, and quantitatively calculate vector data of the document content slices according to slice content formats of the document content slices, and store the vector data according to the slice content formats of the document content slices. The distribution module is further configured to: calculate a first average utilization rate of a first resource pool and a second average utilization rate of a second resource pool in a previous historical period, wherein the total resource pool comprises the first resource pool and the second resource pool, the first resource pool is used for performing a parsing task of a first document type, the second resource pool is used for performing a parsing task of a second document type, and a data capacity of the first document type is smaller than that of the second document type; calculate a first resource amount of the first resource pool and a second resource amount of the second resource pool in a current period according to the first average utilization rate and the second average utilization rate; and distribute parsing resources of the plurality of file segments from the first resource pool and the second resource pool according to the document type.

12. A storage medium, characterized by The storage medium stores a computer program, and the computer program is configured to execute the method in any one of claims 1 to 10 when running. 13.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to execute the method in any one of claims 1 to 10 by running the computer program.

Images