Document classification splitting method and device and storage medium

Abstract

The invention relates to the field of document processing, and discloses a document classification splitting method and device and a storage medium. The method comprises the following steps: receiving a document vectorization instruction, and receiving document data corresponding to the document vectorization instruction; when the document type is an Excel type, performing analysis processing on the document data by utilizing a preset Apache POI (Point of Interest) component to obtain a document character string; when the document type is a PDF (Portable Document Format) type, analyzing and processing the document data by utilizing a preset pdfplug component to obtain a document character string; when the document type is a general document, analyzing the document data by using a LangChain4j component to obtain a document character string; and receiving a storage instruction of a user, and storing the document fragment data into a preset vector database according to the storage instruction. In the embodiment of the invention, the document data is classified and split based on the types of the document data, so that the document accuracy of the document splitting is improved, and the effect that the AI large model replies more accurately after the corpus is input into the AI large model is realized.

Description

Technical Field

[0001] The present invention relates to the field of document processing, and in particular to a document classification and splitting method, device and storage medium. Background Art

[0002] With the development of artificial intelligence and Internet information technology, intelligent chat systems have attracted more and more attention from researchers and practitioners due to their huge potential and development value. The main types of chats include general, tool, FAQ, reasoning, etc. At present, AI has been widely used in many scenarios such as intelligent customer service, intelligent medical care, intelligent finance, intelligent education, intelligent energy, and intelligent agriculture.

[0003] Whether intelligent chat can accurately answer questions raised by users requires attention to whether the knowledge stored in the AI ​​intelligent model is complete and accurate. In order to achieve accurate answers, it is necessary to ensure that the content in the vector library is accurate. The current existing technology cannot meet the requirements of correct document segmentation, and cannot meet the requirements of splitting by paragraphs or chapters. This will lead to many sentence breaks, which will lead to incorrect content of the fragments or incorrect segmentation positions, resulting in incomplete content of the fragments, which in turn makes the AI ​​large model answer incomplete or incorrect. Therefore, in order to solve the technical problem that the current inaccurate segmentation of document data affects the data processing of the AI ​​large model, a new technology is needed to solve the current problem. Summary of the invention

[0004] The main purpose of the present invention is to solve the technical problem that inaccurate segmentation of current document data affects the data processing of AI large models.

[0005] A first aspect of the present invention provides a document classification and splitting method, the document classification and splitting method comprising: Receiving a document vectorization instruction and receiving document data corresponding to the document vectorization instruction, wherein the document vectorization instruction includes: a general splitting mode, and the document data includes: a document type; When the document type is Excel type, the document data is parsed using a preset Apache POI component to obtain a document string, and the document string is split according to a preset regular expression to obtain document fragment data; When the document type is PDF type, the document data is parsed and fragmented using the preset pdfplumber component to obtain document fragment data; When the document type is a general document, the document data is parsed using the LangChain4j component to obtain a document string, and the document string is split according to the general splitting mode to obtain document fragment data; Visually display the document segment data; A storage instruction from a user is received, and according to the storage instruction, the document segment data is stored in a preset vector database.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the step of splitting the document string according to the general splitting pattern to obtain document fragment data includes: When the general splitting mode is character splitting, the document string is split according to the preset DocumentByCharacterSplitter splitting component to obtain document fragment data.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the step of splitting the document string according to the general splitting mode to obtain document fragment data further includes: When the general splitting mode is paragraph splitting, the document string is split according to a preset regular expression to obtain document fragment data.

[0008] Optionally, in a third implementation of the first aspect of the present invention, the step of splitting the document string according to the general splitting mode to obtain document fragment data further includes: When the general splitting mode is splitting by line, the document string is split based on a preset regular expression to obtain document fragment data.

[0009] Optionally, in a fourth implementation manner of the first aspect of the present invention, the visual display of the document fragment data includes: The document segment data is visually displayed to obtain an editable segment interface corresponding to the document segment data.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, after the visual display of the document fragment data and before the receiving of a storage instruction from a user and storing the document fragment data in a preset vector database according to the storage instruction, the method further includes: Receive adjustment instructions from users; Based on the adjustment instruction, content adjustment processing is performed on the editable paging interface to generate an adjusted paging interface.

[0011] Optionally, in a sixth implementation of the first aspect of the present invention, the receiving a storage instruction from a user, and storing the document fragment data in a preset vector database according to the storage instruction includes: Receive a storage instruction from a user, wherein the storage instruction includes: a page ID; The display data of the paging interface corresponding to the page ID is stored in a preset vector database.

[0012] Optionally, in a seventh implementation manner of the first aspect of the present invention, storing the document fragment data in a preset vector database according to the storage instruction includes: Cache the document shard data in a preset MongoDB database; According to the storage instruction, the document fragment data corresponding to the storage instruction is synchronously stored from the MongoDB database to the preset vector database.

[0013] The second aspect of the present invention provides a document classification and splitting device, comprising: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected via lines; the at least one processor calls the instructions in the memory so that the document classification and splitting device executes the above-mentioned document classification and splitting method.

[0014] A third aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned document classification and splitting method.

[0015] In an embodiment of the present invention, classification analysis is performed based on the type of document data, and character strings are accurately parsed out of the document data in Excel type, PDF type, and general type, and the character strings are split using regular expressions, thereby avoiding unclear file contents after segmentation caused by incorrect document data segmentation, and achieving segmentation of document data into fragments according to content relevance, thereby improving the document accuracy of document segmentation and achieving a more accurate response from the AI ​​big model after the corpus is input into the AI ​​big model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of an embodiment of a method for classifying and splitting documents in an embodiment of the present invention; Figure 2 It is a schematic diagram of a first embodiment of step 106 of the document classification and splitting method in an embodiment of the present invention; Figure 3 It is a schematic diagram of a second embodiment of step 106 of the document classification and splitting method in an embodiment of the present invention; Figure 4 The figure is a schematic diagram of an embodiment of a document classification and splitting device in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The embodiments of the present invention provide a document classification and splitting method, device and storage medium.

[0018] The embodiments disclosed in the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments disclosed in the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments disclosed in the present invention are only for exemplary purposes and are not intended to limit the scope of protection disclosed in the present invention.

[0019] In the description of the embodiments disclosed in the present invention, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0020] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the document classification and splitting method in the embodiment of the present invention includes: 101. Receive a document vectorization instruction, and receive document data corresponding to the document vectorization instruction, wherein the document vectorization instruction includes: a general splitting mode, and the document data includes: a document type; In this embodiment, it is first necessary to create a new vector set, set the vector set name, and receive the document vectorization instruction issued by the user. The document vectorization instruction has a general split mode set by the user, and the user needs to upload the document data corresponding to the document vectorization instruction. The document data contains the corresponding document type data, such as a series of document data such as PDF document, Excel document, Word document, TXT document, Markdown document, etc.

[0021] 102. When the document type is Excel type, the document data is parsed using a preset Apache POI component to obtain a document string, and the document string is split according to a preset regular expression to obtain document fragment data; In this embodiment, since the Excel document of the default document parser is parsed by row, some useless information such as sheet name will be parsed out at the same time. Secondly, if you only want to keep the content in the Excel document starting from a certain row, a certain column, and a certain number of columns to the right, the original method cannot meet this specific requirement. Therefore, the Apache POI component is used to parse and process the Excel type document data sheet by sheet to obtain the document string. Apache POI is an open source Java library developed by the Apache Software Foundation for processing files in Microsoft Office format, including Excel, Word, and PowerPoint. Then use the relevant regular expressions to filter specific rows and columns according to requirements and segment the document string to obtain document segmentation data.

[0022] 103. When the document type is a PDF type, the document data is parsed and fragmented using a preset pdfplumber component to obtain document fragment data; In this embodiment, since the default document parser parses PDF content by line, each line of content is a separate fragment, which will cause multi-line paragraphs to be truncated, resulting in semantic changes and increased manual adjustment workload. Therefore, a python script is called to complete the parsing of PDF type documents by paragraph, and the pdfplumber component is specifically used to parse the document data. pdfplumber is a Python library for processing PDF files, and is particularly good at extracting text and table data from PDF files. Using the preset pdfplumber component, the document data of the PDF type is parsed to obtain the document fragmentation data fragmented by paragraph.

[0023] 104. When the document type is a general document, the document data is parsed using the LangChain4j component to obtain a document string, and the document string is split according to the general splitting mode to obtain document fragment data; In this embodiment, the document type is a general document, which can be an editable document such as a Word document, a TXT document, or a Markdown document, and the general document data can be directly extracted. First, the document data is parsed using the LangChain4j component to obtain a document string. LangChain4j is a library designed to simplify the integration of large language models (LLMs) into Java applications. LangChain4j provides a unified API so that developers do not need to learn and implement specific APIs for each LLM provider (such as OpenAI or Google Vertex AI) and embedding (vector) storage (such as Pinecone or Milvus).

[0024] The general splitting mode is the splitting logic for general documents. The splitting logic can be divided into three types: splitting by paragraph, splitting by character, and splitting by line. Based on different splitting logics, the parsed document string is split to obtain document fragment data.

[0025] Specifically, in one embodiment, step 104 includes the following specific implementations: 1041. When the general splitting mode is character splitting, the document string is split according to the preset DocumentByCharacterSplitter splitting component to obtain document fragment data.

[0026] In step 1041, when the general splitting mode is character splitting, the DocumentByCharacterSplitter splitting component is used to split the document string to obtain document fragmentation data. DocumentByCharacterSplitter is a document splitter in the LangChain4j library. Its main function is to split the provided Document based on characters and put as many characters as possible into a single TextSegment while complying with the limit set by maxSegmentSize.

[0027] Specifically, in one embodiment, step 104 also includes the following specific implementation methods: 1042. When the general splitting mode is paragraph splitting, the document string is split according to a preset regular expression to obtain document segmentation data.

[0028] In step 1042, when the general split mode is paragraph splitting, the TXT, MD and other type documents are parsed using the TextDocumentParser in the LangChain4j library to obtain the document string, and the DOCX type document is parsed using the ApachePoiDocumentParser in the LangChain4j library to obtain the document string. Then, the document string is filtered and split by a regular expression to obtain document fragment data.

[0029] Specifically, in one embodiment, step 104 also includes the following specific implementation methods: 1043. When the general splitting mode is splitting by line, the document string is split based on a preset regular expression to obtain document fragment data.

[0030] In step 1043, when the general splitting mode is to split by line, the content of the corresponding document type is first parsed according to different document parsers of langchain4j, and then the document string can be filtered and split according to the set regular expression to obtain document fragment data split by line.

[0031] 105. Visually display the document segment data; In this embodiment, the document segment data is displayed visually. After the user checks and confirms the segment content on the page, the file segment content is saved, the segment content is passed to the server, and a save request is initiated to the server to complete the save operation.

[0032] Specifically, step 105 includes the following specific implementation methods: 1051. Visually display the document segment data to obtain an editable segment interface corresponding to the document segment data.

[0033] In step 1051, when the document segment data is visually displayed, the displayed interface can be edited so that the user can check, add, modify, delete, replace, etc. the segment content of each document on the page.

[0034] Further, after step 1051 and before step 106, the following specific implementations are also included: 1052. Receive adjustment instructions from a user; 1053. Based on the adjustment instruction, perform content adjustment processing on the editable paging interface to generate an adjusted paging interface.

[0035] In steps 1052-1053, the user can select a certain slice content in the client browser for inspection or adjustment. Based on the user's adjustment instructions, the editable slice page is adjusted to obtain an adjustment paging interface, and the slice data corresponding to the adjustment paging interface is used as vector data in the vector set.

[0036] 106. Receive a storage instruction from a user, and store the document segment data into a preset vector database according to the storage instruction.

[0037] In this embodiment, a storage instruction from a user is received, and based on the storage instruction issued by the user, the document fragment data is stored in the knowledge base corresponding to the LLM large model, and the knowledge base of the LLM large model is designed as a vector database. The vector database converts unstructured data (such as text and images) into vector form for storage, provides efficient similarity search and retrieval functions, and thus enhances the intelligent service capabilities of the LLM large model in vertical fields.

[0038] For details, please refer to Figure 2 , Figure 2 This is the first embodiment of step 106 of the document classification and splitting method in the embodiment of the present invention, and step 106 includes the following specific implementation methods: 1061. Receive a storage instruction from a user, wherein the storage instruction includes: a page ID; 1062. Store the display data of the paging interface corresponding to the page ID into a preset vector database.

[0039] In steps 1061-1062, the user can view the shard content on the shard management page, select one or more shard records and select the enable button to send the shard page IDs to the server. After receiving the request, the server can query the corresponding shard content of the adjusted paging interface based on the ID and save the shard content synchronously to the vector collection.

[0040] For details, please refer to Figure 3 , Figure 3 This is a second embodiment of step 106 of the document classification and splitting method in an embodiment of the present invention, and step 106 includes the following specific implementation methods: 1063. Cache the document fragment data into a preset MongoDB database; 1064. Receive a storage instruction from the user, and according to the storage instruction, synchronously store the document fragment data corresponding to the storage instruction from the MongoDB database into the preset vector database.

[0041] In steps 1063-1064, after the user visualizes the document fragment data, the document fragment data can be cached in the MongoDB database. After receiving the user's storage instruction, the corresponding document fragment data stored in the MongoDB database is synchronously saved in the vector database.

[0042] Enabling shard synchronization to vector collection mainly provides the function of vectorizing the document shard content stored in the MongoDB database and storing it in the vector collection. Subsequently, you only need to bind the AI ​​large model to this knowledge base to use the content of this document.

[0043] In an embodiment of the present invention, classification analysis is performed based on the type of document data, and character strings are accurately parsed out of the document data in Excel type, PDF type, and general type, and the character strings are split using regular expressions, thereby avoiding unclear file contents after segmentation caused by incorrect document data segmentation, and achieving segmentation and vectorization of document data according to content-related fragments. By improving the document accuracy of document segmentation, a more accurate response from the AI ​​big model can be achieved after the corpus is input into the AI ​​big model.

[0044] Figure 4 It is a structural diagram of a document classification and splitting device provided by an embodiment of the present invention. The document classification and splitting device 400 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 410 (for example, one or more processors) and a memory 420, and one or more storage media 430 (for example, one or more mass storage devices) storing application programs 433 or data 432. Among them, the memory 420 and the storage medium 430 can be short-term storage or permanent storage. The program stored in the storage medium 430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the document classification and splitting device 400. Furthermore, the processor 410 can be configured to communicate with the storage medium 430 to execute a series of instruction operations in the storage medium 430 on the document classification and splitting device 400.

[0045] The document-based classification and splitting device 400 may also include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input and output interfaces 460, and / or one or more operating systems 431, such as Windows Serve, Mac OS X, Unix, Linux, Free BSD, etc. It can be understood by those skilled in the art that Figure 4The document classification and splitting device structure shown does not constitute a limitation on the document-based classification and splitting device, and may include more or less components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0046] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the document classification and splitting method.

[0047] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0048] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0049] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A document classification and splitting method, characterized in that: Includes steps: Receiving a document vectorization instruction and receiving document data corresponding to the document vectorization instruction, wherein the document vectorization instruction includes: a general splitting mode, and the document data includes: a document type; When the document type is Excel type, the document data is parsed using a preset Apache POI component to obtain a document string, and the document string is split according to a preset regular expression to obtain document fragment data; When the document type is PDF type, the document data is parsed and fragmented using the preset pdfplumber component to obtain document fragment data; When the document type is a general document, the document data is parsed using the LangChain4j component to obtain a document string, and the document string is split according to the general splitting mode to obtain document fragment data; Visually display the document segment data; A storage instruction from a user is received, and according to the storage instruction, the document segment data is stored in a preset vector database.

2. The document classification and splitting method according to claim 1, characterized in that: The document character string is split according to the general splitting mode to obtain document fragment data, including: When the general splitting mode is character splitting, the document string is split according to the preset DocumentByCharacterSplitter splitting component to obtain document fragment data.

3. The document classification and splitting method according to claim 1, characterized in that: The step of splitting the document string according to the general splitting mode to obtain document fragment data further comprises: When the general splitting mode is paragraph splitting, the document string is split according to a preset regular expression to obtain document fragment data.

4. The document classification and splitting method according to claim 1, characterized in that: The step of splitting the document string according to the general splitting mode to obtain document fragment data further comprises: When the general splitting mode is splitting by line, the document string is split based on a preset regular expression to obtain document fragment data.

5. The document classification and splitting method according to claim 1, characterized in that: The visual display of the document segment data includes: The document segment data is visually displayed to obtain an editable segment interface corresponding to the document segment data.

6. The document classification and splitting method according to claim 5, characterized in that: After visually displaying the document segment data and before receiving a storage instruction from a user and storing the document segment data in a preset vector database according to the storage instruction, the method further includes: Receive adjustment instructions from users; Based on the adjustment instruction, content adjustment processing is performed on the editable paging interface to generate an adjusted paging interface.

7. The document classification and splitting method according to claim 6, characterized in that: The receiving a storage instruction from a user, and storing the document fragment data in a preset vector database according to the storage instruction comprises: Receive a storage instruction from a user, wherein the storage instruction includes: a page ID; The display data of the paging interface corresponding to the page ID is stored in a preset vector database.

8. The document classification and splitting method according to claim 1, characterized in that: The receiving a storage instruction from a user, and storing the document fragment data in a preset vector database according to the storage instruction further comprises: Cache the document shard data in a preset MongoDB database; Receive a storage instruction from a user, and according to the storage instruction, synchronously store the document fragment data corresponding to the storage instruction from the MongoDB database to the preset vector database.

9. A document classification and splitting device, characterized in that: The document classification and splitting device comprises: a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instructions in the memory to enable the document classification and splitting device to perform the document classification and splitting method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the document classification and splitting method according to any one of claims 1 to 8 is implemented.

Images