Document processing method, computing device and computer readable storage medium

Abstract

The invention discloses a document processing method, computing equipment and a computer readable storage medium. The method comprises the steps of obtaining a to-be-processed document in a target format; performing hierarchical structure extraction on the to-be-processed document based on the trained large language model to obtain hierarchical structure information of the to-be-processed document; the hierarchical structure information comprises multiple pieces of hierarchical information and starting page numbers and ending page numbers corresponding to all hierarchies; splitting the to-be-processed document according to the hierarchical structure information of the to-be-processed document and a preset splitting strategy to obtain a document splitting result; the document splitting result comprises at least one sub-document, and each sub-document is at least associated with one piece of hierarchical information. In this way, the document can be split flexibly, accurately and efficiently, the automation degree and the intelligence degree are high, and time and labor are saved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of document processing, in particular to a document processing method, a computing device and a computer readable storage medium. BACKGROUND

[0002] In order to improve the speed of online browsing and downloading of large-volume non-audio and video documents and avoid unnecessary waste of network traffic, a single large-volume document can be split into multiple small-volume documents, and then the small-volume documents are browsed and downloaded. However, in the existing document splitting methods, some need to manually split the document into multiple small documents according to fixed rules (such as the page range of chapters) through tools (websites, software, etc.), and some need to rely on specific conditions, such as bookmarks of PDF documents and directories with specific formats in the document. If these conditions are not met or the document structure is slightly complex, the splitting cannot be achieved. Therefore, the above-mentioned document splitting methods have the problems of time-consuming and laborious, lack of automation and intelligence, low accuracy and efficiency. SUMMARY

[0003] The purpose of the present application is to provide a document processing method, a computing device and a computer readable storage medium, which can flexibly, accurately and efficiently split documents, have high automation and intelligence, and save time and effort.

[0004] To achieve the above purpose:

[0005] In a first aspect, the embodiments of the present application provide a document processing method, comprising:

[0006] obtaining a target format of a to-be-processed document;

[0007] extracting a hierarchical structure of the to-be-processed document based on a trained large language model to obtain hierarchical structure information of the to-be-processed document; the hierarchical structure information comprises a plurality of hierarchical information and starting and ending page numbers corresponding to each hierarchical level;

[0008] splitting the to-be-processed document according to the hierarchical structure information of the to-be-processed document and a preset splitting strategy to obtain a document splitting result; the document splitting result comprises at least one sub-document, and each sub-document is associated with at least one hierarchical information.

[0009] Optionally, the obtaining of the target format of the to-be-processed document comprises:

[0010] obtaining the to-be-processed document;

[0011] when the format of the to-be-processed document is determined to be not the target format according to the file extension of the to-be-processed document, converting the format of the to-be-processed document into the target format.

[0012] Optionally, before the step of extracting the hierarchical structure information of the to-be-processed document based on the trained large language model, the method further comprises:

[0013] detecting whether the to-be-processed document meets a preset splitting condition;

[0014] if the to-be-processed document meets the preset splitting condition, triggering the step of extracting the hierarchical structure information of the to-be-processed document based on the trained large language model.

[0015] Optionally, the training process of the large language model comprises:

[0016] obtaining a plurality of training document samples; the training document samples comprise training documents and preset hierarchical structure information corresponding to the training documents;

[0017] respectively pre-processing the plurality of training document samples, the pre-processing comprising at least one of the following: text extraction according to the type of the training document, data cleaning on the extracted text content;

[0018] dividing the pre-processed plurality of training document samples into a training set and a test set;

[0019] training a large language model constructed based on the training set and aiming at a task of hierarchical structure element extraction, hierarchical relationship construction and page offset calibration, and obtaining predicted hierarchical structure information of the test set by the trained large language model;

[0020] when the loss between the preset hierarchical structure information corresponding to the test set and the predicted hierarchical structure information reaches a convergence condition, stopping training the large language model, and obtaining a trained large language model.

[0021] Optionally, the step of splitting the to-be-processed document according to the hierarchical structure information of the to-be-processed document and a preset splitting strategy to obtain a document splitting result comprises at least one of the following:

[0022] if the splitting strategy is hierarchical splitting, splitting the to-be-processed document according to the plurality of hierarchical information and the start page number and the end page number corresponding to each hierarchical level to determine the sub-documents corresponding to each hierarchical information;

[0023] If the splitting strategy is mixed splitting, a number of splitting levels is determined according to a size and / or a number of pages of the to-be-processed document and a preset level number division rule, and the to-be-processed document is split according to the number of splitting levels, the plurality of level information and a start page number and an end page number corresponding to each level, to determine a sub-document corresponding to each level information.

[0024] Optionally, the method further comprises:

[0025] The sub-documents are named according to a preset naming strategy.

[0026] Optionally, the method further comprises:

[0027] A start page number of the target level in the target sub-document is determined according to target level information corresponding to the target sub-document and a start page number corresponding to the target level; the target sub-document is any sub-document;

[0028] A corresponding bookmark in the target sub-document is generated according to the start page number of the target level in the target sub-document.

[0029] Optionally, the method further comprises:

[0030] The sub-documents are stored according to a preset storage strategy, and the document splitting result is output in a preset manner.

[0031] In a second aspect, an embodiment of the present application provides a computing device, comprising a processor and a memory storing a computer program, when the processor runs the computer program, the above-mentioned document processing method is implemented.

[0032] In a third aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is executed by a processor, the above-mentioned document processing method is implemented.

[0033] The document processing method, computing device and computer-readable storage medium provided by the embodiment of the present application include: obtaining a document to be processed in a target format; extracting a hierarchical structure of the document to be processed based on a trained large language model to obtain hierarchical structure information of the document to be processed; the hierarchical structure information includes multiple hierarchical information and the start page number and end page number corresponding to each hierarchical level; splitting the document to be processed according to the hierarchical structure information of the document to be processed and a preset splitting strategy to obtain a document splitting result; the document splitting result includes at least one sub-document and each sub-document is associated with at least one hierarchical information. In this way, the hierarchical structure information of the document to be processed is extracted by a large language model, and then the document to be processed is split according to the hierarchical structure information of the document to be processed and a preset splitting strategy to divide the document to be processed with a larger storage space into multiple sub-documents with smaller storage space, and each sub-document is associated with the corresponding hierarchical information, which can flexibly, accurately and efficiently split the document, with a high degree of automation and intelligence, and saves time and effort. At the same time, it can improve the speed of subsequent online browsing and downloading of documents to be processed, saving network traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of a flow chart of a document processing method provided by an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of the document processing method provided by an embodiment of the present invention;

[0036] Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different drawings represent identical or similar elements, unless otherwise indicated. It should be understood that the specific embodiments described herein are intended only to illustrate the present application and are not intended to limit the present application.

[0038] The following is an explanation of the terms that may be involved in this embodiment:

[0039] Large Language Models (LLMs): are artificial intelligence models based on deep learning, specifically designed to understand and generate human language, with core features including: (1) large-scale parameters: parameter quantity usually from hundreds of millions to tens of billions, trained through massive text data; (2) versatility: can complete multiple tasks (such as question answering, translation, writing), without the need for special design for a single task; (3) self-supervised learning: pre-training through predicting missing parts in text (such as masked language modeling), without the need for human-labeled data; (4) generation capability: can generate coherent and logically strong text, simulating human writing style.

[0040] Optical Character Recognition (OCR): refers to the process of electronic devices checking printed characters on paper, determining their shape by detecting light and dark patterns, and then translating the shape into computer text using character recognition methods; that is, for printed characters, using optical methods to convert the text in paper documents into black and white bitmap image files, and converting the text in the image into text format through recognition software, for further editing and processing by word processing software.

[0041] MD5 value: is the "digital fingerprint" of a file, which is unique. Even if the name and size of the file are the same, as long as the content is different, its MD5 value will be different. Therefore, the MD5 value can be used as a unique identifier for a file to distinguish different files. In some file management systems or databases, MD5 values are often used to identify files, making it easy to store, retrieve and manage files. By comparing MD5 values, you can quickly determine whether two files are the same, avoiding duplicate storage of the same file and saving storage space.

[0042] Referring to Figure 1 A document processing method is provided in the embodiments of the present application, which can be executed by a document processing device provided in the embodiments of the present application. The document processing device can be implemented in software and / or hardware, such as a computer or a server, etc. In the present embodiment, the execution subject of the document processing method is taken as an example of a server (such as a cloud server, etc.), and the document processing method provided in the present embodiment includes:

[0043] Step S101: obtaining a target format of a to-be-processed document.

[0044] The to-be-processed document is any non-audio / video document that needs to be split, such as an operation manual, a user manual, a maintenance manual, a parts atlas, a technical manual, or training materials of a construction machinery device, etc. The target format can be set according to actual needs, and in the present embodiment, the target format is taken as an example of a PDF format.

[0045] Specifically, the server can receive a to-be-processed document in a target format input or selected by a user, or receive a to-be-processed document in a non-target format input or selected by a user, and convert the format of the to-be-processed document into the target format.

[0046] In an embodiment, the to-be-processed document in the target format is obtained, including:

[0047] Obtaining the to-be-processed document;

[0048] Converting the format of the to-be-processed document into the target format when it is determined according to the file extension of the to-be-processed document that the format of the to-be-processed document is not the target format.

[0049] The file extension is the part after the dot at the end of the file name, used to identify the file format and type, and different file extensions correspond to different document formats and applicable software. Since the file extension of the to-be-processed document can represent the format of the to-be-processed document, the server can detect whether the format of the to-be-processed document is the target format according to the file extension of the to-be-processed document after obtaining the to-be-processed document. If it is determined according to the file extension of the to-be-processed document that the format of the to-be-processed document is not the target format, the format of the to-be-processed document needs to be converted into the target format. If it is determined according to the file extension of the to-be-processed document that the format of the to-be-processed document is the target format, the format of the to-be-processed document does not need to be processed. For example, taking the target format as the PDF format as an example, if the file extension of the to-be-processed document is not pdf, the to-be-processed document in the doc, docx, ppt, etc. format can be converted into the to-be-processed document in the pdf format using the Java programming language combined with Apache POI, iText, Apache FOP, etc. tools. In this way, by unifying the format of the document, the multi-format compatibility problem can be solved, and structured and standardized input data can be provided for subsequent large language models.

[0050] Step S102: performing hierarchical structure extraction on the to-be-processed document based on the trained large language model to obtain hierarchical structure information of the to-be-processed document; the hierarchical structure information includes a plurality of hierarchical information and starting and ending page numbers corresponding to each hierarchical level.

[0051] Among them, the hierarchical information includes both hierarchical content (such as the table of contents, the first chapter, etc.) and the corresponding levels (such as the table of contents is the first level, the first chapter is the first level, etc.). The multiple levels of the document to be processed may include the first level (such as the first level title), the second level (such as the second level title), the third level (such as the third level title), ..., the Nth level (such as the Nth level title), etc., and different levels can be numbered and / or written in different ways, for example, the first level can be numbered in uppercase Chinese characters or written in size 1 font, the second level can be numbered in lowercase numbers or written in size 2 font, the third level can be numbered in lowercase numbers with brackets or written in size 3 font, etc. In actual applications, the hierarchy of a document can be understood as a title. Taking the technical manual of a certain engineering machinery equipment as an example, the first level of the hierarchical structure information of the document to be processed may include the following titles: Overview, Safety and Compliance, Functional Usage Guide; the second level may include the following titles: Document Purpose, Product Introduction, Glossary under the Overview title, Safety Warning, Compliance Statement under the Safety and Compliance title, Core Function Module and Advanced Function under the Functional Usage Guide title; the third level may include the following titles: Function A and Function B under the Core Function Module title.

[0052] It should be noted that, when the document to be processed includes a table of contents page, the title (such as "Table of Contents" or "Contents") and chapter titles can be divided into the first level, the sub-chapter titles can be divided into the second level, and so on. In addition, the large language model in this embodiment can be obtained by training existing artificial intelligence models such as LayoutLMv3 and DeepSeek based on training document samples.

[0053] In one embodiment, the training process of the large language model includes:

[0054] Acquire multiple training document samples; the training document samples include training documents and preset hierarchical structure information of corresponding annotations of the training documents;

[0055] Preprocessing is performed on a plurality of training document samples respectively, wherein the preprocessing includes at least one of the following: extracting text from the training documents according to the types of the training documents, and performing data cleaning on the extracted text contents;

[0056] Divide the preprocessed multiple training document samples into training sets and test sets;

[0057] Based on the training set, a large language model is trained to extract hierarchical elements, build hierarchical relationships, and calibrate page offsets. The predicted hierarchical structure information of the test set is obtained by the trained large language model.

[0058] When the loss between the preset hierarchical structure information corresponding to the test set and the predicted hierarchical structure information reaches a convergence condition, the training of the large language model is stopped, and a trained large language model is obtained.

[0059] The training documents can be operation manuals, user manuals, maintenance manuals, part catalogs, technical manuals, training materials, and other types of documents. To ensure data diversity, these training documents include but are not limited to common languages, with directories, without directories, common extensions, scanned, different layout styles (such as multi-level directories, complex fonts), and the like. In addition, tools can also be used to generate a large number of simulated documents of various types as training documents to increase data diversity, such as using LaTeX tools to predefine various formats of documents, etc. After obtaining the training documents, the training documents can be labeled to determine the preset hierarchical structure information corresponding to the training documents, and the preset hierarchical structure information includes all hierarchical information in the training documents and the start and end page numbers corresponding to each level. Optionally, the training documents can be manually or semi-automatically added with hierarchical structure information to make the training documents more readable and understandable, so as to better identify, classify and analyze the training documents. For example, for non-scanned training documents with directories, a special labeling tool (such as Label Studio, CVAT) can be used for manual labeling, or a script can be written for semi-automatic labeling and manual correction to label the directory items (i.e., the titles of each level) and their corresponding start and end page numbers, while clearly indicating the hierarchical relationship between the directory items. For non-scanned training documents without directories, regular content modules such as chapter titles, paragraphs, topics, and fonts can be identified first, and their corresponding page numbers and hierarchical relationships between each other can be manually labeled. For scanned training documents, a special labeling tool can be used to manually label the directory items and their corresponding start and end page numbers.

[0060] Optionally, in order to facilitate the large language model to better understand the structure and hierarchical relationship of the document, the training document samples can be preprocessed respectively, and the preprocessing includes at least one of the following: text extraction according to the type of the training document, data cleaning of the extracted text content. The type of the training document is used to represent the generation mode of the training document, including scanning generation and non-scanning generation. Taking the target format PDF as an example, for the PDF document generated by scanning, the optical character recognition (OCR) technology can be used to convert the text in the document into editable text, and the parameters are adjusted according to the language and font characteristics of the document to improve the recognition accuracy; for the PDF document generated by non-scanning, the PDF processing library (such as PyMuPDF, PDFMiner) can be used to extract the text content from the document. Then, the extracted text content can be data cleaned, including removing noise information such as special characters, punctuation marks, and redundant spaces, to unify the text format. Optionally, the preprocessing can also include extracting the layout information of the document, such as the position, size, font, etc. of the text.

[0061] Optionally, the pre-processed training document samples can be divided into a training set for training the model and a test set for testing the performance of the model according to a preset ratio such as 7:3, 8:2, etc., the training set containing more training document samples than the test set. Optionally, the main tasks of the constructed large language model include hierarchical structure element (such as target item, chapter title, page number) extraction, hierarchical relationship construction (such as constructing a target tree), and page number offset calibration. In the specific implementation process of hierarchical structure element extraction, it can be detected first whether there is a "catalog" word or a special layout of "title... page number" in the first N pages of the document, such as the first 10 pages or the first 20 pages. If so, the entire directory structure is directly extracted, otherwise visual feature analysis is performed to identify large font, centering, and other special format text as candidate first-level titles, and identify regular numbered format text such as "1.1", "Chapter 2.2" as second-level titles, etc. In the specific implementation process of hierarchical relationship construction, the number of decimal points in the number can be used to determine the level (such as "2.3.1" is the third level), and the titles are arranged according to the rule that the parent title is in front of the child title and the vertical position in the page, and the content between the two titles is automatically detected to belong to which chapter or section. For example, it is detected that the 5th page has "4.1 Experimental Methods" and the 7th page has "4.2 Data Analysis", then it is determined that the content of the 5th-6th pages belongs to the "4.1 Experimental Methods" section. Optionally, since the table of contents page, preface page, etc. in the document occupy the physical page number, it may cause the page number of the actual content to be misaligned, at this time, the offset needs to be detected first, and then the offset is applied to correct the page number corresponding to different levels. For example, when the start page number of "Chapter 1" in the table of contents is the 5th page, but the content of Chapter 1 actually starts from the 7th page, the start page number of "Chapter 1" needs to be automatically increased by an offset of 2 pages, i.e. the start page number of "Chapter 1" is marked as the 7th page.

[0062] Optionally, the cross-entropy loss function can be used to evaluate the accuracy of the model in predicting the target items, chapter titles, and other elements, the mean square error loss function can be used to evaluate the accuracy of the model in predicting the page numbers, and a custom evaluation index such as tree structure similarity can be used to evaluate the accuracy of the model in predicting the upper and lower hierarchical relationships, i.e., the loss between the predicted results (i.e., the predicted hierarchical structure information) and the true labels (i.e., the preset hierarchical structure information) can be measured in the above-mentioned manner. When the loss between the preset hierarchical structure information and the predicted hierarchical structure information corresponding to the test set reaches the convergence condition, the training of the large language model is stopped, and the trained large language model is obtained. When the loss between the preset hierarchical structure information and the predicted hierarchical structure information corresponding to the test set does not reach the convergence condition, the parameters of the large language model are adjusted for further training. The loss reaching the convergence condition can be that the loss value is less than or equal to a preset value, etc. In this way, by training the large language model, the automation and accuracy of document splitting can be significantly improved.

[0063] In an embodiment, before extracting the hierarchical structure information of the to-be-processed document based on the trained large language model, the following steps are included:

[0064] Detecting whether the to-be-processed document meets the preset splitting condition;

[0065] If the to-be-processed document meets the preset splitting condition, triggering the step of extracting the hierarchical structure information of the to-be-processed document based on the trained large language model.

[0066] The preset splitting condition can be set according to actual needs, such as being greater than 1 KB in size, having more than 5 pages in total, having an extension belonging to doc, pdf, docx, or ppt, etc. It can be understood that before extracting the hierarchical structure information of the to-be-processed document based on the trained large language model, it is detected whether the to-be-processed document meets the preset splitting condition, so that when the to-be-processed document meets the preset splitting condition, the extraction of the hierarchical structure information of the to-be-processed document based on the trained large language model is triggered, thereby the to-be-processed document can not be split when it does not need to be split, and resource waste is reduced.

[0067] Optionally, the to-be-processed document in the target format or the original to-be-processed document can be stored in a file sharing server, and the splitting state of the to-be-processed document and other information can be stored in a search engine or a database. When the to-be-processed document needs to be split, the search engine or the database can be queried to determine whether the to-be-processed document has been split, and if it has been split, it is considered to meet the preset splitting condition.

[0068] In an embodiment, if the user feedback indicates that the document splitting result of the to-be-processed document is inaccurate, the to-be-processed document can be pushed as sample data to the large language model to retrain the large language model.

[0069] Step S103: splitting the to-be-processed document according to the hierarchical structure information of the to-be-processed document and the preset splitting strategy to obtain a document splitting result; the document splitting result includes at least one sub-document, and each sub-document is associated with at least one hierarchical information.

[0070] The splitting strategy is used to indicate the splitting manner of the document, including hierarchical splitting, mixed splitting, etc. It can be understood that, since the hierarchical structure information of the to-be-processed document can obtain the page range corresponding to different levels in the to-be-processed document, splitting the to-be-processed document according to the hierarchical structure information of the to-be-processed document and the preset splitting strategy can realize splitting each level in the to-be-processed document into one or more sub-documents, and each sub-document is associated with at least one hierarchical information. Alternatively, one sub-document can be associated with one hierarchical information or multiple hierarchical information. For example, assuming that the hierarchical structure information of the to-be-processed document includes "hierarchical content: table of contents, level: one, start page number: 1, end page number: 2", "hierarchical content: first chapter, level: one, start page number: 3, end page number: 15", "hierarchical content: 1.1 power system, level: two, start page number: 10, end page number: 13", "hierarchical content: 1.2 arm system, level: two, start page number: 14, end page number: 15", if splitting according to the first level, the content of the first page to the second page in the to-be-processed document generates a first sub-document, and the content of the third page to the fifteenth page in the to-be-processed document generates a second sub-document, at this time, the hierarchical information associated with the first sub-document includes "hierarchical content: table of contents, level: one", and the hierarchical information associated with the second sub-document includes "hierarchical content: first chapter, level: one", "hierarchical content: 1.1 power system, level: two", "hierarchical content: 1.2 arm system, level: two".

[0071] In an embodiment, splitting the to-be-processed document according to the hierarchical structure information of the to-be-processed document and the preset splitting strategy to obtain a document splitting result includes at least one of the following:

[0072] If the splitting strategy is hierarchical splitting, the to-be-processed document is split according to the multiple hierarchical information, the start page number and the end page number corresponding to each level, and the preset target level number to determine at least one sub-document;

[0073] If the splitting strategy is mixed splitting, the number of splitting levels is determined according to the size and / or the number of pages of the to-be-processed document and preset level division rules, and the to-be-processed document is split according to the number of splitting levels, the plurality of level information, and the start page number and the end page number corresponding to each level, to determine at least one sub-document.

[0074] The splitting by level refers to splitting the to-be-processed document by level in sequence, and the target level number is used to indicate the maximum level of splitting the to-be-processed document, which can be one level (also referred to as first level), two levels (also referred to as second level), three levels (also referred to as third level), etc. For example, assuming that the level structure information of the to-be-processed document includes "level content: directory, level: 1, start page number: 1, end page number: 2", "level content: first chapter, level: 1, start page number: 3, end page number: 15", "level content: 1.1 power system, level: 2, start page number: 10, end page number: 13", and "level content: 1.2 arm system, level: 2, start page number: 14, end page number: 15", if the splitting is by level and the preset target level number is one level, the content of page numbers 1 to 2 in the to-be-processed document can be split into one sub-document, and the content of page numbers 3 to 15 can be split into one sub-document; if the splitting is by level and the preset target level number is two levels, the content of page numbers 1 to 2 in the to-be-processed document can be split into one sub-document, the content of page numbers 3 to 9 can be split into one sub-document, the content of page numbers 10 to 13 can be split into one sub-document, and the content of page numbers 14 to 15 can be split into one sub-document.

[0075] The mixed splitting refers to splitting according to the size and / or the number of pages of the to-be-processed document, that is, first determining the number of levels of splitting according to the size and / or the number of pages of the to-be-processed document and a preset level number division rule, then splitting the to-be-processed document according to the number of levels of splitting, the plurality of level information and the start page number and the end page number corresponding to each level to determine at least one sub-document. For example, assuming that the level structure information of the to-be-processed document includes "level content: directory, level: 1, start page number: 1, end page number: 2", "level content: first chapter, level: 1, start page number: 3, end page number: 15", "level content: 1.1 power system, level: 2, start page number: 10, end page number: 13", "level content: 1.2 arm system, level: 2, start page number: 14, end page number: 15" and "level content: second chapter, level: 1, start page number: 16, end page number: 22", if the splitting strategy is mixed splitting and the preset level number division rule is to split the document with the number of pages greater than 10 and less than 50 according to the first level, according to the number of pages of the to-be-processed document and the preset level number division rule, it can be determined that the number of levels of splitting is one, and then the content of the to-be-processed document with the page number from 1 to 2 can be split into one sub-document, the content of the to-be-processed document with the page number from 3 to 15 can be split into another sub-document, and the content of the to-be-processed document with the page number from 16 to 22 can be split into another sub-document; if the splitting strategy is mixed splitting and the preset level number division rule is to split the document with the number of pages greater than 10 and less than 50 according to the second level, according to the number of pages of the to-be-processed document and the preset level number division rule, it can be determined that the number of levels of splitting is two, and then the content of the to-be-processed document with the page number from 1 to 2 can be split into one sub-document and the content of the to-be-processed document with the page number from 3 to 15 can be split into one sub-document; if the splitting strategy is according to the level and the preset target level number is two, the content of the to-be-processed document with the page number from 1 to 2 can be split into one sub-document, the content of the to-be-processed document with the page number from 3 to 9 can be split into one sub-document, the content of the to-be-processed document with the page number from 10 to 13 can be split into one sub-document, the content of the to-be-processed document with the page number from 14 to 15 can be split into one sub-document, and the content of the to-be-processed document with the page number from 16 to 22 can be split into one sub-document. In this way, the flexibility of document splitting is improved and the user experience is improved by providing a flexible splitting strategy for splitting the document.

[0076] In summary, the document processing method provided by the above embodiments can extract the level structure information of the to-be-processed document through a large language model, and then split the to-be-processed document according to the level structure information of the to-be-processed document and a preset splitting strategy, so as to divide the to-be-processed document with a large required storage space into a plurality of sub-documents with a small required storage space, while each sub-document is associated with corresponding level information, which can flexibly, accurately and efficiently split the document, has high automation and intelligence, saves time and effort. At the same time, the speed of subsequent online browsing and downloading of the to-be-processed document can be improved, and network traffic can be saved.

[0077] In an embodiment, the method further comprises:

[0078] naming the sub-documents according to a preset naming strategy.

[0079] The naming strategy is used to represent the naming manner of the document, including a random number strategy (such as naming according to a timestamp, naming according to a random function, etc.), a file name self-increment strategy, etc. Taking the file name self-increment strategy as an example, assuming that the file name of the to-be-processed document is A.pdf, the first sub-document split out can be named as A1.pdf, the second sub-document split out can be named as A2.pdf, and so on, until the to-be-processed document is split completely. In this way, the user can conveniently distinguish different sub-documents, and the user experience is improved.

[0080] In an embodiment, the method further comprises:

[0081] determining a start page number of the target level in the target sub-document according to target level information corresponding to the target sub-document and a start page number corresponding to the target level in the to-be-processed document; the target sub-document is any sub-document;

[0082] generating a corresponding bookmark in the target sub-document according to the start page number of the target level in the target sub-document.

[0083] Wherein, for any sub-document obtained by splitting, i.e. a target sub-document, since the page number marked in the target sub-document matches the page number in the document to be processed, and the starting page number in the target sub-document can not be 1, the page numbers in the bookmarks of different levels in the target sub-document need to be recalculated, and then corresponding bookmarks can be generated in the target sub-document. For example, assuming that the hierarchical structure information of the document to be processed includes "hierarchical content: table of contents, level: 1, starting page number: 1, ending page number: 2", "hierarchical content: first chapter, level: 1, starting page number: 3, ending page number: 15", "hierarchical content: 1.1 power system, level: 2, starting page number: 10, ending page number: 13", "hierarchical content: 1.2 arm system, level: 2, starting page number: 14, ending page number: 15", if the target sub-document is a sub-document generated from the content of the document to be processed with page numbers from 3 to 15, the target sub-document marks the page number range as 3 to 15, but in fact the target sub-document has only 13 pages in total, therefore when generating the bookmarks of the target sub-document, the page numbers in the bookmarks of different levels need to be recalculated, i.e. the corresponding page number of "hierarchical content: first chapter, level: 1" in the bookmarks is determined as 1, the corresponding page number of "hierarchical content: 1.1 power system, level: 2" in the bookmarks is determined as 8, and the corresponding page number of "hierarchical content: 1.2 arm system, level: 2" in the bookmarks is determined as 12, and then corresponding bookmarks can be generated in the target sub-document. In this way, the labels of the sub-document obtained by splitting can be automatically and accurately generated, and the user experience is improved.

[0084] In an embodiment, the method further comprises:

[0085] storing each sub-document according to a preset storage strategy, and outputting the document splitting result through a preset manner.

[0086] Wherein, the storage strategy is used to indicate the storage configuration information of the document, such as the storage manner (local storage, cloud storage), storage account and password, storage path, etc. The preset manner is used to indicate the notification manner of the document splitting result, including in-site message, short message, email, social application, etc. Optionally, when the document splitting result is output through the preset manner, the document splitting result and the storage path and other information can be output together. Optionally, each sub-document can be stored to a file sharing server, so as to facilitate the user to call. In this way, by storing the sub-document obtained by splitting and outputting the document splitting result, the user can view and use conveniently, and the user experience is improved.

[0087] Based on the same inventive concept of the foregoing embodiments, the method provided by the foregoing embodiments is exemplified by a specific example, and the example is taken as an example in the following.

[0088] Referring to Figure 2The document processing method provided by the embodiment mainly includes the following processes: configuration management, document preprocessing, task scheduling, model management, document processing, and feedback processing.

[0089] 1) Configuration management

[0090] Mainly responsible for managing configurations and strategies, mainly including splitting strategies, naming strategies, bookmark strategies, storage configurations, and notification strategies. Among them, all strategies can be set with a default value.

[0091] Splitting strategy refers to the splitting method of large-volume files, mainly including ignoring strategies (for example, documents with a volume less than 1 KB are not split; documents with a total number of pages less than 5 pages are not split; documents with extensions other than doc, pdf, docx, and ppt are not split), and documents that meet the ignoring strategy do not need to be split; splitting by level (for example, first-level, second-level, and third-level titles); splitting by page number (for example, splitting every 5 pages into a small document); mixed splitting (for example, splitting by first-level if the volume is less than 1M, and splitting by third-level if the total number of pages is greater than 500); and splitting every page (splitting every page of the document into a small document).

[0092] Naming strategy refers to the naming method of split files, mainly including random number strategies (such as timestamps and random functions), and original file name increment strategies (when the first file is split, the original file name is concatenated with the number 1; when the second file is split, the original file name is concatenated with the number 2, and so on until the splitting is completed).

[0093] Bookmark strategy refers to whether to generate PDF bookmarks, generate first-level bookmarks, generate second-level bookmarks, etc. The split documents are all in PDF format.

[0094] Storage configuration refers to the configuration related to the storage of split files, such as storage path, storage method (local storage, cloud storage), storage account and password, CDN domain name, etc.

[0095] Notification strategy refers to the notification method of the splitting result, mainly including in-site messages, short messages, emails, and enterprise WeChat.

[0096] 2) Document preprocessing

[0097] Responsible for receiving documents, converting documents, determining scanning, saving documents and their metadata.

[0098] When the original document is received, an asynchronous thread is started to extract the file extension, file size, and determine whether to support splitting according to the pre-configured ignore strategy. If not, the splitting result of not supporting splitting is returned. If supporting splitting, the MD5 value of the document is calculated, and then the MD5 value is used to query the status of the document in Elasticsearch. If the document has been correctly split, the calling party is informed that the document has been split and the related data of the split document through the notification strategy. If the document is being split, the calling party is informed that the document is being split. If the splitting fails or no related data is queried, the splitting operation can be performed.

[0099] Next, according to the extracted file extension, it is determined whether the document is a PDF file. If not, the non-PDF format document is converted into a PDF format document. For example, Java programming language can be used in combination with Apache POI, iText, Apache FOP, etc. to convert files in doc, docx, ppt, etc. format into PDF format files.

[0100] Using Java programming tools (iText or Apache PDFBox) or other programming tools, the text content of the first 20 pages of the PDF document is read. If there are more than 15 pages of text content that are empty, it is determined that the document is generated by scanning, and the document is marked as a scanned version.

[0101] The original PDF file or the PDF file converted from other formats is saved to the file storage server according to the storage configuration. At the same time, a document metadata object is created, and the id, upload time, uploader, modification time, file name, business id, system, MD5 value, storage address, size, status, extension, file page object, etc. attributes of the object are assigned values, and then stored in Elasticsearch. The status attribute value is "to be split", and the file page object contains page text content and page number.

[0102] 3) Task scheduling

[0103] Through the timing task scheduler, the large language model is scheduled to split the "to be split" document data every 1 minute (other intervals can also be defined). Then, the status of the "to be split" document data is modified to "splitting".

[0104] 4) Model management

[0105] Responsible for training large language models and providing services to external through API after training, including data preparation, model fine-tuning, model evaluation, inference output, and model deployment. In this embodiment, the LayoutLMv3 large language model is selected, which combines text, image, and layout information and can better handle multi-modal data. The input of LayoutLMv3 needs the following information: text sequence (text content of each page), bounding box (coordinates of each text segment), image.

[0106] 4.1) Data preparation: mainly includes data collection, data labeling, data preprocessing, and data set division.

[0107] 4.1.1) Data collection: Collect a large number of operation manuals, user manuals, maintenance manuals, parts catalogs, technical manuals, training materials, and other types of documents. To ensure data diversity, these documents include but are not limited to various common languages, with and without directories, common extensions, scanned, different layout styles (such as multi-level directories, complex fonts), etc. In addition, use tools to generate a large number of various types of simulation documents to increase data diversity, for example, you can use LaTeX tools to predefine various formats of documents; use StyleGAN to generate realistic scanned documents with wrinkles, shadows, and moire, and randomly inject OCR noise.

[0108] 4.1.2) Data labeling: Add labels, annotations, or other descriptive information to data through manual or semi-automatic methods to make data more readable and understandable, so that data can be better identified, classified, and analyzed. For non-scanned, directory-based documents, use specialized labeling tools (such as Label Studio, CVAT) to manually label or write scripts for semi-automatic labeling, and then manually correct the directory items and their corresponding page numbers, while clearly indicating the hierarchical relationship between directory items. For non-scanned, directory-free documents, identify regular content modules such as chapter titles, paragraphs, topics, fonts, positions, styles, etc., and manually label their page numbers and hierarchical relationships. For scanned documents, use specialized labeling tools to manually label the directory items and their corresponding page numbers.

[0109] 4.1.3) Data Preprocessing: For scanned-generated PDF documents, use Optical Character Recognition (OCR) techniques to convert the text in images into editable text, and adjust parameters according to the language and font characteristics of the document to improve recognition accuracy. OCR tools can be open source (such as Tesseract) or commercial OCR services (such as Ali Cloud OCR). For non-scanned-generated PDF documents, use PDF processing libraries (such as PyMuPDF, PDFMiner) to extract text content from PDF documents. Then, clean the extracted text to remove special characters, punctuation marks, extra spaces, and other noise information, and unify the text format. In addition to the text content, the layout information of the document, such as the position, size, and font of the text, should also be extracted. These layout information helps LayoutLMv3 better understand the structure and hierarchical relationship of the document.

[0110] 4.1.4) Dataset Division: Divide the preprocessed and labeled data into training set, validation set and test set, usually in the ratio of 7:2:1. The training set is used for model training, the validation set is used to monitor the performance of the model during training to prevent overfitting, and the test set is used to evaluate the performance of the model.

[0111] 4.2) Model Fine-tuning, responsible for training LayoutLMv3 model.

[0112] 4.2.1) Define tasks

[0113] There are three main tasks: extracting hierarchical structure elements (such as directory items, chapter titles, page numbers, etc.), building hierarchical relationship (establishing directory tree), and page number offset calibration.

[0114] Extract hierarchical structure elements: First, detect whether there is a "directory" word or a special layout of "title... page number" in the first 20 pages of the document, if there is, directly extract the entire directory structure, otherwise perform visual feature analysis, identify large font, centering, and other special format text as candidate first-level titles, and detect regular numbered format text such as "1.1", "Chapter 2.2" as second-level titles.

[0115] Hierarchical relationship construction: The number of decimal points in the number can be used to determine the level (such as "2.3.1" for three-level title), arrange the titles according to the page number and vertical position in the page, ensure that the parent title is in front and the child title is in back, and automatically detect which chapter the content between two titles belongs to. For example, if it is detected that the 5th page has "4.1 Experimental Methods" and the 7th page has "4.2 Data Analysis", then it is determined that the content of pages 5-6 belongs to "4.1" subsection.

[0116] Page offset calibration: Due to the occupation of physical page numbers by table of contents pages and preface pages, the actual content page numbers are misaligned. At this time, the offset amount needs to be detected first (for example, when the "first chapter" in the table of contents is marked on page 5, but the actual content starts from page 7, automatically calculate an offset of 2 pages), and then apply the offset to all page numbers for correction.

[0117] 4.2.2) Define loss function

[0118] The cross-entropy loss function is used to handle category prediction, and the mean square error loss function is used to handle page number prediction. For the prediction of hierarchical relationships and page offset calibration, a custom loss function can be designed to measure the difference between the predicted results and the true labels.

[0119] 4.2.3) Model adjustment

[0120] Load the pre-trained LayoutLMv3 model using a deep learning framework such as PyTorch. Convert the preprocessed and labeled data (which includes text sequences, bounding boxes, images, etc.) into a format suitable for model input, build a dataset and data loader. The data loader is responsible for loading data in batches during training, improving training efficiency. Fine-tune the pre-trained LayoutLMv3 model using the training set. During fine-tuning, continuously adjust the model's parameters to minimize the loss function. At the same time, use the validation set to monitor the performance of the model, and adjust training parameters such as learning rate, batch size, etc. according to the results of the validation set to prevent model overfitting.

[0121] 4.3) Model evaluation

[0122] Category prediction evaluation includes accuracy, recall, F1 value, etc., used to evaluate the accuracy of the model's prediction of directory items, chapter titles, etc. Page number prediction evaluation includes mean absolute error (MAE), root mean square error (RMSE), etc., used to evaluate the accuracy of the model's prediction of page numbers. For hierarchical relationship evaluation, custom evaluation indicators such as tree structure similarity can be designed to measure the consistency between the model's predicted hierarchical relationships and the true labels. Use the test set to evaluate the trained model and calculate the values of each evaluation indicator. Then, according to the evaluation results, analyze the strengths and weaknesses of the model, find out the problems of the model, such as inaccurate recognition of some types of documents or layouts, etc., in order to further optimize the model.

[0123] 4.4) Model inference

[0124] Inference on new PDF documents using the trained model. First, preprocess the input PDF document, extract text, layout information, images, and then input them into the model to get the model's prediction results. Next, process the model's prediction results to build a tree structure with hierarchical relationships. Then, generate JSON data based on the tree structure, where each node contains text content, page number, and hierarchical relationship information. Finally, return the JSON data to the caller.

[0125] 4.5) Model deployment

[0126] After packaging the model, deploy the trained model to multiple servers and provide RESTful APIs or other forms of interfaces for external calls, and these interfaces support batch processing and asynchronous result callbacks.

[0127] 5) Document processing

[0128] Responsible for document segmentation, bookmark generation, file saving, exception alarm, and result return.

[0129] Call the trained large language model to get the document segmentation results extracted by the large language model, combine the segmentation strategy and naming strategy, and use Java programming tools (iText or Apache PDFBox) to split the large volume document into different small documents with specific file names. As shown in Table 1, assuming the large language model returns hierarchical content, level, start page number, and end page number, and the segmentation strategy is to split by the first level, and the naming strategy is to increment the original file name, then the original document named "A.pdf" will be split into three sub-documents with file names "A1.pdf", "A2.pdf", and "A3.pdf".

[0130] Table 1

[0131] Hierarchical content Hierarchy Start page End page Corresponding file Directory 1 1 2 A1.pdf Chapter 1 1 3 15 A2.pdf 1.1 Power system 2 10 13 A2.pdf 1.2 Boom system 2 14 15 A2.pdf Chapter 2 1 16 22 A3.pdf

[0132] In addition, PDF bookmarks can be generated according to the bookmark strategy using Java programming tools (iText or Apache PDFBox). If the splitting strategy is "split by page", no bookmark needs to be generated. After document segmentation, the actual page numbers are offset, and the page numbers for bookmark jumping need to be recalculated. For example, the "A2.pdf" sub-document generated in the previous example, the page number calculation method for its bookmarks is shown in Table 2.

[0133] Table 2

[0134]

[0135] Then, according to the storage configuration, the small documents are stored on the document sharing server in batches. Here, if an exception occurs in the document splitting process, the calling party is informed through the notification strategy.

[0136] If the large-volume document has been correctly split, its state is modified to "correctly split", and the calling party is informed through the notification strategy. The calling party calls the document data after splitting through the encapsulated API interface as needed, wherein the data returned to the calling party can include: title, level, start page number, end page number, file path, etc.

[0137] 6) Feedback processing

[0138] The split inaccurate document data collected by the user feedback is pushed to the large language model as sample data, and the large language model is retrained, so that the large language model can be dynamically updated.

[0139] In summary, the document processing method provided by the above embodiments can flexibly and automatically split scanned documents, non-scanned documents, documents with directories, documents without directories, documents with complex styles, and documents with different extensions. The single large-volume document can be split into multiple small-volume documents and saved to the corresponding server, so that the user can browse and download the corresponding document faster. The method has the advantages of high automation and intelligence, flexible and accurate document splitting, and high efficiency.

[0140] Based on the same inventive concept as the foregoing embodiments, the embodiments of the present application provide a computing device, as shown in Figure 3 , the computing device comprises a processor 310 and a memory 311 storing a computer program; wherein, Figure 3 The processor 310 in the above embodiment is not used to refer to the number of processors 310 being one, but is only used to refer to the positional relationship of the processor 310 relative to other devices. In actual application, the number of processors 310 can be one or more. Similarly, Figure 3 The memory 311 in the above embodiment also has the same meaning, i.e., it is only used to refer to the positional relationship of the memory 311 relative to other devices. In actual application, the number of memories 311 can be one or more. When the processor 310 runs the computer program, the document processing method described above is implemented.

[0141] The computing device can further comprise at least one network interface 312. The various components in the computing device are coupled together through a bus system 313. It can be understood that the bus system 313 is used to realize the connection and communication between the components. The bus system 313 includes not only a data bus, but also a power bus, a control bus and a status signal bus. However, for the sake of clear illustration, all kinds of buses are marked as the bus system 313 in Figure 3 .

[0142] Based on the same inventive concept as the foregoing embodiments, the present embodiment also provides a computer readable storage medium, in which a computer program is stored, and the computer readable storage medium can be a ferromagnetic random access memory (FRAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, a compact disc read-only memory (CD-ROM), or the like memory; or can be various devices including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, and the like. The computer program stored in the computer readable storage medium is run by a processor, and the document processing method described above is implemented. The specific step flow implemented by the computer program run by the processor will not be described here again in the description of the embodiments shown in the drawings. Figure 1 The specific step flow implemented by the computer program run by the processor will not be described here again in the description of the embodiments shown in the drawings.

[0143] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A document processing method, characterized in that: include: Get the document to be processed in the target format; Extracting the hierarchical structure of the document to be processed based on the trained large language model to obtain hierarchical structure information of the document to be processed; The hierarchical structure information includes multiple hierarchical information and the start page number and end page number corresponding to each hierarchical level; The document to be processed is split according to the hierarchical structure information of the document to be processed and a preset splitting strategy to obtain a document splitting result; the document splitting result includes at least one sub-document and each sub-document is associated with at least one hierarchical information.

2. The method according to claim 1, characterized in that The step of obtaining the document to be processed in the target format includes: Obtaining the document to be processed; When it is determined according to the file extension of the document to be processed that the format of the document to be processed is not the target format, the format of the document to be processed is converted into the target format.

3. The method according to claim 1, characterized in that The step of extracting the hierarchical structure of the document to be processed based on the trained large language model and obtaining the hierarchical structure information of the document to be processed includes: Detecting whether the document to be processed meets the preset splitting conditions; If the document to be processed meets the preset splitting condition, the step of extracting the hierarchical structure of the document to be processed based on the trained large language model to obtain the hierarchical structure information of the document to be processed is triggered.

4. The method according to claim 1, wherein The training process of the large language model includes: Acquire multiple training document samples; the training document samples include training documents and preset hierarchical structure information corresponding to the annotations of the training documents; Preprocessing the plurality of training document samples respectively, wherein the preprocessing includes at least one of the following: extracting text from the training documents according to the types of the training documents, and performing data cleaning on the extracted text contents; Dividing the preprocessed multiple training document samples into a training set and a test set; Training a large language model constructed with hierarchical structure element extraction, hierarchical relationship construction, and page number offset calibration as target tasks based on the training set, and obtaining predicted hierarchical structure information of the test set by the trained large language model; When the loss between the preset hierarchical structure information and the predicted hierarchical structure information corresponding to the test set reaches a convergence condition, the training of the large language model is stopped to obtain a trained large language model.

5. The method according to any one of claims 1 to 4, characterized in that The step of splitting the document to be processed according to the hierarchical structure information of the document to be processed and a preset splitting strategy to obtain a document splitting result includes at least one of the following: If the splitting strategy is splitting by level, the document to be processed is split according to the multiple level information, the start page number and the end page number corresponding to each level, and the preset target number of levels to determine at least one sub-document; If the splitting strategy is a mixed splitting, the number of split levels is determined based on the size and / or number of pages of the document to be processed and the preset level division rules, and the document to be processed is split according to the number of split levels, the multiple level information and the starting page numbers and ending page numbers corresponding to each level to determine at least one sub-document.

6. The method according to claim 5, characterized in that The method further comprises: Each sub-document is named according to a preset naming strategy.

7. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Determining the starting page number of the target level in the target sub-document according to target level information associated with the target sub-document and the starting page number of the target level in the document to be processed; the target sub-document is any sub-document; According to the starting page number of the target level in the target sub-document, a corresponding bookmark is generated in the target sub-document.

8. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The sub-documents are stored according to a preset storage strategy, and the document splitting results are output in a preset manner.

9. A computing device, characterized in that include: A processor and a memory storing a computer program, wherein when the processor runs the computer program, the document processing method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that A computer program is stored, and when the computer program is executed by a processor, the document processing method according to any one of claims 1 to 8 is implemented.

Images