Document processing method and related equipment

Abstract

The invention discloses a document processing method and related equipment, and belongs to the technical field of data processing.The method comprises the steps that in response to a document input instruction, a to-be-processed target document is obtained; performing structure analysis on a to-be-processed target document to generate a directory tree of the target document; calling a dynamic recursive slicing algorithm, traversing and analyzing the directory tree of the target document, and generating a plurality of text slices with dynamic lengths; and vectorizing the plurality of text slices with the dynamic lengths, and uploading and storing the vectorized text slices to a database of a retrieval enhancement generation system. By constructing the directory tree and traversing the directory tree by adopting the dynamic recursive slicing algorithm, the document structure can be more accurately understood, the semantic integrity of the document slices is improved, and the retrieval accuracy and the answer generation quality of the retrieval enhancement generation system are further effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a document processing method and related equipment. BACKGROUND

[0002] Currently, as a mainstream architecture integrating private knowledge bases and general models, the performance of a retrieval-augmented generation (RAG) system is highly dependent on the quality of the knowledge base, and document slicing is a key link affecting the quality of the knowledge base.

[0003] In related technologies, a fixed-length sliding window, simple semantic segmentation, or rule-based segmentation method is usually used for document slicing processing. However, in actual applications, it is found that the traditional document slicing processing method generally has problems such as destroying the original logical structure of the document, which can easily lead to separation of the title and the content, and thus the retrieval results of the RAG system are inaccurate.

[0004] To sum up, the technical problems existing in related technologies need to be improved. SUMMARY

[0005] Embodiments of the present application provide a document processing method and related equipment, which can improve the semantic integrity of document slicing, and thus effectively improve the retrieval accuracy and answer generation quality of a retrieval-augmented generation system.

[0006] In one aspect, the present application provides a document processing method, which includes the following steps: In response to a document input instruction, a target document to be processed is obtained; The target document to be processed is structurally parsed to generate a directory tree of the target document; A dynamic recursive slicing algorithm is called to traverse and parse the directory tree of the target document to generate a plurality of text slices of dynamic lengths; The plurality of text slices of dynamic lengths are vectorized and uploaded and stored in a database of a retrieval-augmented generation system.

[0007] Optionally, the dynamic recursive slicing algorithm is called to traverse and parse the directory tree of the target document to generate a plurality of text slices of dynamic lengths, including: The directory tree of the target document is input to obtain node information of the directory tree; Based on a depth-first principle, each node and all child nodes thereof are recursively parsed, and a dynamic decision function is called to determine a slicing strategy; The recursive parsing results of all nodes are integrated to generate a plurality of text slices of dynamic lengths.

[0008] Optionally, the dynamic recursive slicing algorithm comprises: determining and inputting a current node to be recursively parsed; adding content of the current node to a current text slice and calling a dynamic decision function to determine a slicing strategy; recursively traversing each child node of the current node and executing the calling of the dynamic decision function to determine the slicing strategy; after completing the recursive traversal of all child nodes, executing the calling of the dynamic decision function to determine the slicing strategy again, completing the recursive parsing of the current node, and jumping to a next node.

[0009] Optionally, the dynamic decision function comprises: determining a size relationship between a length of the current text slice and a preset threshold; determining a change of a node type attribute; determining a change of a node semantic label; determining whether a preset forced slicing point is met.

[0010] Optionally, the structure parsing of the target document to be processed comprises: inputting the target document to be processed and performing document parsing; extracting document elements according to a document parsing result; wherein the document elements comprise a title, a paragraph text, and a table; extracting a title level for a title type document element and creating a title node; extracting text content for a paragraph text type document element and creating a paragraph text node; extracting table content and a title for a table type document element and creating a table node; constructing a pointer according to a hierarchical relationship of the target document, thereby outputting a directory tree of the target document.

[0011] Optionally, before the calling of the dynamic recursive slicing algorithm and the recursive traversal of the directory tree of the target document, the method further comprises: determining a document type of the target document; configuring a text slice granularity based on the document type of the target document.

[0012] In another aspect, an embodiment of the present application provides a document processing device, which comprises: a document acquisition module configured to acquire a target document to be processed in response to a document input instruction; a directory tree generation module, configured to perform structural analysis on the target document to be processed to generate a directory tree of the target document; a text slice module, configured to call a dynamic recursive slicing algorithm to traverse and analyze the directory tree of the target document to generate a plurality of text slices with dynamic lengths; a slice uploading module, configured to vectorize the plurality of text slices with dynamic lengths and upload and store the plurality of text slices with dynamic lengths to a database of a retrieval enhancement generation system.

[0013] In another aspect, an electronic device is provided, which includes a memory and a processor. The memory stores a computer program. The processor implements the above method when executing the computer program.

[0014] In another aspect, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the above method.

[0015] In another aspect, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the above method.

[0016] The embodiments of the present application can more accurately understand the document structure by constructing a directory tree and traversing the directory tree using a dynamic recursive slicing algorithm, improve the semantic integrity of the document slices, and thus effectively improve the retrieval accuracy and answer generation quality of the retrieval enhancement generation system. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is an implementation environment schematic diagram of a document processing method provided by the embodiments of the present application; Figure 2 is a flow schematic diagram of a document processing method provided by the embodiments of the present application; Figure 3 is a flow schematic diagram of constructing a directory tree provided by the embodiments of the present application Figure 4 is a flow schematic diagram of calling a dynamic recursive slicing algorithm provided by the embodiments of the present application; Figure 5 is a flow schematic diagram of calling a dynamic decision function provided by the embodiments of the present application; Figure 6 is a structural schematic diagram of a document processing apparatus provided by the embodiments of the present application; Figure 7 is a hardware structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0019] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0020] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0022] Currently, Retrieval-augmented Generation (RAG) systems, as the mainstream architecture for integrating private knowledge bases and general models, have performance that is highly dependent on the quality of the knowledge base, and document slicing is a key factor affecting the quality of the knowledge base.

[0023] In related technologies, document slicing is typically performed using fixed-length sliding windows, simple semantic segmentation, or rule-based segmentation methods. However, in practical applications, it has been found that traditional document slicing methods generally suffer from problems such as disrupting the original logical structure of the document, easily leading to the separation of titles and content, and consequently resulting in inaccurate search results from the RAG system.

[0024] In view of this, this application provides a document processing method and related device. By constructing a directory tree and traversing the directory tree using a dynamic recursive slicing algorithm, the document structure can be understood more accurately, the semantic integrity of document slices can be improved, and the retrieval accuracy and answer generation quality of the retrieval enhancement generation system can be effectively improved.

[0025] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0026] The specific implementation methods of the embodiments of this application will be described in detail below with reference to the accompanying drawings. First, a document processing method provided in the embodiments of this application will be described with reference to the accompanying drawings.

[0027] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating the implementation environment of a document processing method provided in this application embodiment. In this implementation environment, the main hardware and software components involved include a terminal processor 110 and a server 120.

[0028] Specifically, the terminal processor 110 may have a control program for a related document processing method installed, and the server 120 is the backend server for this control program. The terminal processor 110 and the backend server 120 are connected via a communication link. The document processing method provided in this embodiment can be executed on the terminal processor 110 side.

[0029] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0030] In addition, server 120 can also be a node server in a blockchain network.

[0031] The terminal processor 110 and the server 120 can establish a communication connection via a wireless network. This wireless network uses standard communication technologies and / or protocols. The network can be the Internet or any other network, including but not limited to a Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), mobile, or any combination of wireless networks, private networks, or virtual private networks. Furthermore, these hardware and software components can use the same or different communication connection methods; this application does not impose specific limitations in this regard.

[0032] Of course, this is understandable. Figure 1 The implementation environment described in this application is only one of the optional application scenarios for the document processing method provided in this embodiment. The actual application is not fixed. Figure 1 The software and hardware environment shown is not specifically limited in this application.

[0033] like Figure 2 As shown, Figure 2 This is a flowchart illustrating a document processing method provided in an embodiment of this application, specifically including but not limited to steps 100 to 400.

[0034] Step 100: In response to the document input instruction, obtain the target document to be processed.

[0035] In this embodiment, the document input instruction can be generated by the user by triggering a relevant button, or it can be issued by the background system based on a task order. The document input instruction may include information such as the target document ID to be processed, upload location, and source type. The document format to be processed may include various document formats such as PDF, Word, HTML, plain text, and scanned images.

[0036] Furthermore, after receiving and responding to the document input instruction, the system can initiate verification instructions and permissions based on the target document information carried in the document input instruction, confirm whether it has the right to access the target document to be processed, obtain the target document, and create the original document object for subsequent document parsing.

[0037] Step 200: Perform structural parsing on the target document to be processed to generate a directory tree of the target document.

[0038] In this embodiment, the target document to be processed is structurally parsed, and the linear or layout-based document content is converted into a structured directory tree, so as to provide skeleton and positioning information for subsequent structure-aware text slicing. The directory tree is a tree-shaped data model used to represent the hierarchical structure of documents. Its node types include root nodes, heading nodes, text nodes, table nodes, etc., and the topological relationship between nodes is maintained through parent-child pointers and sibling pointers.

[0039] Specifically, as an optional implementation, the step of performing structural parsing on the target document to be processed to generate a directory tree of the target document includes: Input the target document to be processed and perform document parsing; Document elements are extracted based on the document parsing results; wherein, the document elements include headings, paragraph text, and tables; Extract heading levels from document elements of heading type and create heading nodes; Extract the text content from document elements of paragraph text type and create paragraph text nodes; Extract table content and titles from table-type document elements and create table nodes; Based on the hierarchical relationship of the target documents, pointers are constructed to output the directory tree of the target documents.

[0040] For example, please refer to Figure 3 , Figure 3 This is a flowchart illustrating a process for constructing a directory tree, provided in an embodiment of this application. The target document to be processed is input into the directory tree generation module, and the target document is parsed, such as by parsing a PDF using OCR, or by decompressing a docx file.

[0041] Furthermore, document elements are extracted based on the document parsing results. These document elements may include different types of semantic units such as headings, paragraph text, and tables.

[0042] Furthermore, different extraction operations are performed based on different types of document elements. Specifically, for document elements identified as headings, the heading level is extracted and heading nodes are created; for document elements identified as paragraph text, the text content is extracted and paragraph text nodes are created; and for document elements identified as tables, the table content and headings are extracted and table nodes are created.

[0043] In practical applications, semantic tags can also be created for each heading node, paragraph text node, and table node to serve as a basis for dynamic decision-making during subsequent recursive parsing and slicing processes.

[0044] Finally, based on the hierarchical relationship of the target documents, parent-child or sibling pointers are constructed, and the final output is the directory tree of the target documents, for example, as follows: Figure 3As shown, the root node's child node is the heading node, which corresponds to heading level one. The heading node has two child nodes: a paragraph text node and another heading node, which corresponds to heading level two. Each of these has two additional child nodes.

[0045] Therefore, this application parses the input target document, identifies different document elements (headings at all levels, body paragraphs, tables, lists, etc.), and constructs a directory tree containing multiple node types and carrying semantic tags, which can deeply and clearly perceive the structural information of the target document.

[0046] Step 300: Call the dynamic recursive slicing algorithm to traverse and parse the directory tree of the target document, and generate multiple text slices of dynamic length.

[0047] In this embodiment, a dynamic recursive slicing algorithm is invoked to traverse and parse the directory tree of the target document, thereby generating multiple text slices of dynamic length. Specifically, starting from the root node of the directory tree, each branch is recursively traversed, and the slice boundaries are dynamically determined based on semantic integrity and context window length, so that the generated text blocks maintain the logical structure of the original document.

[0048] Therefore, compared to traditional document processing methods that use fixed-length sliding windows and simple semantic segmentation, which can damage the original logical structure of the document, this application first constructs a richer directory tree of node types and data fields to support deep structure awareness. Then, it adopts a recursive slicing algorithm with dynamic decision-making to dynamically construct and segment text blocks during the recursive traversal process. This allows for a more scientific understanding of the document structure, avoids errors that may be caused by mechanical matching, and makes the text slicing more semantically intact, thereby improving the retrieval accuracy and answer generation quality of the RAG system.

[0049] Specifically, as an optional implementation, the step of invoking a dynamic recursive slicing algorithm to traverse and parse the directory tree of the target document and generate multiple text slices of dynamic length includes: Input the directory tree of the target document and obtain the node information of the directory tree; Based on the depth-first principle, each node and all its child nodes are recursively parsed, and a dynamic decision function is called to determine the slicing strategy. By combining the recursive parsing results of all nodes, multiple text slices with dynamic lengths are generated.

[0050] In this embodiment of the application, when slicing a document, the target document's directory tree is first input to obtain the node information. Based on the depth-first principle, each branch is traversed first, each node and all its child nodes are recursively parsed before jumping to the next node, and the slicing strategy is determined through a dynamic decision function.

[0051] Furthermore, after traversing and parsing all the node information in the directory tree, the text slices retained after each slice determination in the current directory tree can be obtained. Based on the parsing results of all nodes, all text slices can be obtained. Since each text slice is obtained by determining the slice through a dynamic decision function, its text length is dynamically adjusted, thereby generating multiple text slices with dynamic lengths.

[0052] In practical applications, please refer to Figure 4 , Figure 4 This is a flowchart illustrating a dynamic recursive slicing algorithm provided in this application embodiment. First, the parsed and constructed directory tree and initial configuration are passed in. The current node N to be recursively parsed is determined and passed in. Reading and judgment logic is executed on this node to read its text / table / title content from the reference or cache of node N and add it to the current text slice. For example, if node N is a title node, the title text can be used as the beginning of the text slice for contextual hints. If the node is a paragraph text node, the paragraph text content can be added to the text slice while retaining the paragraph division, thereby updating the text slice and changing its length and semantic context.

[0053] Furthermore, after appending the content of the current node, the decision function is called to determine whether to immediately cut out the currently updated text slice, i.e., end a text block and save it, or continue to accumulate the child node content of the current node N.

[0054] Optionally, if it is determined that a slice is needed, the current text slice is saved to the parsing result list. It can be determined whether context information needs to be carried when saving the text slice, and a new empty text slice is created as the subsequent accumulation container. Then the recursion continues. If it is determined that a slice is not needed, the current text slice is kept, and the processing of the child nodes of node N continues.

[0055] For example, the depth-first approach executes the same recursive parsing process as described above for each child node of the current node N: "input node → append content → dynamic decision function to determine slicing strategy → recursively traverse each child node" (i.e., treat the child node as the new current node, append its text and make slicing judgments) until it is determined that all child nodes have been recursively parsed, and the current branch traversal is considered to have ended.

[0056] Therefore, during the recursive parsing of child nodes, it is possible to trigger the addition of the child node content to the text slice of the parent node, or to trigger the text slice in the child node, thereby detaching from the parent node and independently parsing to obtain a new text slice.

[0057] Furthermore, after processing the current child node (and all its descendant nodes), before leaving the current node N, the dynamic decision function can be called again to determine whether slicing is needed. If so, the current text slice is saved and a new text slice is created, and the process jumps to the sibling node S of the current node N, making it the next current node and repeating the above recursive parsing process. If not, the process jumps directly to the sibling node S of the current node N, making it the next current node and repeating the above recursive parsing process.

[0058] Therefore, once all descendants and sibling branches of the root node have been processed and all text slicing operations have been completed, the recursive parsing process can be stopped, resulting in a list of text slices with semantic integrity, which can then be vectorized and stored in the RAG system.

[0059] In practical applications, please refer to Figure 5 , Figure 5 This is a flowchart illustrating the calling of a dynamic decision function provided in an embodiment of this application. When calling the dynamic decision function to determine the slicing strategy, the judgment criteria can be based on determining the relationship between the current text slice length and the preset threshold, determining the changes in node type attributes, determining the changes in node semantic tags, and determining whether the preset mandatory slicing point is met.

[0060] Specifically, when any of the following conditions are met: the current text slice length is greater than a preset threshold, the node type attribute changes, the node semantic label changes, or a preset forced slicing point is met, the slicing strategy is determined to be slicing required. When none of the above conditions are met, the slicing strategy is determined to be slicing not required, the current text slice is retained, and content can continue to be added.

[0061] For example, whether a preset mandatory slicing point is met can be determined based on a user-configured strategy.

[0062] Specifically, before invoking the dynamic recursive slicing algorithm to traverse and parse the directory tree of the target document and generate multiple text slices of dynamic length, the method further includes: Determine the document type of the target document; Configure the text slicing granularity based on the document type of the target document.

[0063] In this application embodiment, different slicing strategy parameters can also be configured for different types of documents (such as papers, product manuals, technical reports, API documents) or different application scenarios.

[0064] For example, an enterprise knowledge base typically contains a large number of product manuals, API documents, and technical reports. You can set finer granularity for API documents (based on a single interface) and coarser granularity for technical reports (based on chapters).

[0065] Furthermore, during the recursive parsing and slicing process, for API documentation, each interface name can be used as a title node, with its description, parameter table, and code examples as its child nodes. Based on a pre-set finer slicing granularity, each interface and all its contents are kept within a single text slice as much as possible. This allows operators to return a complete and accurate text block containing all parameter descriptions for the interface when using the RAG system to search for questions, greatly improving the accuracy of question answering.

[0066] Step 400: Vectorize the multiple dynamically lengthed text slices, upload and store them in the database of the retrieval enhancement generation system.

[0067] In this embodiment of the application, the text blocks after text slicing are vectorized and stored in the database of the RAG system, and finally integrated into the complete RAG question answering process, so that relevant fragments can be recalled based on vector similarity during subsequent retrieval and the context input of the generation model can be supported.

[0068] Therefore, by constructing a directory tree and traversing it using a dynamic recursive slicing algorithm, this application can more accurately understand the document structure, improve the semantic integrity of document slices, and thus effectively improve the retrieval accuracy and answer generation quality of the retrieval enhancement generation system. It can better adapt to documents of various types and structures and has stronger versatility.

[0069] Please see Figure 6 , Figure 6 This is a schematic diagram of a document processing apparatus provided in an embodiment of this application. This application also provides a document processing apparatus that can implement the above-described document processing method. The apparatus includes: The document acquisition module 610 is used to acquire the target document to be processed in response to a document input command; The directory tree generation module 620 is used to perform structural parsing on the target document to be processed and generate a directory tree of the target document; The text slicing module 630 is used to call the dynamic recursive slicing algorithm to traverse and parse the directory tree of the target document and generate multiple text slices of dynamic length. The slice upload module 640 is used to vectorize the multiple dynamically lengthed text slices, upload and store them in the database of the retrieval enhancement generation system.

[0070] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0071] Please see Figure 7 , Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device includes: The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 702 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701 using the methods described in the embodiments of this application. The input / output interface 703 is used to implement information input and output; The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704); The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0072] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0073] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0074] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0075] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0076] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0077] This application provides a document processing method and related device that, by constructing a directory tree and traversing it using a dynamic recursive slicing algorithm, can more accurately understand the document structure, improve the semantic integrity of document slices, and thus effectively improve the retrieval accuracy and answer generation quality of the retrieval enhancement generation system.

[0078] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0079] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0080] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0082] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0083] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0084] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0085] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0086] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0087] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A document processing method, characterized in that, The method includes the following steps: In response to a document input command, retrieve the target document to be processed; The target document to be processed is subjected to structural parsing to generate a directory tree of the target document; The dynamic recursive slicing algorithm is invoked to traverse and parse the directory tree of the target document, generating multiple text slices of dynamic length. The multiple dynamically lengthed text slices are vectorized, uploaded, and stored in the database of the retrieval enhancement generation system.

2. The method according to claim 1, characterized in that, The dynamic recursive slicing algorithm is invoked to traverse and parse the directory tree of the target document, generating multiple text slices of dynamic length, including: Input the directory tree of the target document and obtain the node information of the directory tree; Based on the depth-first principle, each node and all its child nodes are recursively parsed, and a dynamic decision function is called to determine the slicing strategy. By combining the recursive parsing results of all nodes, multiple text slices with dynamic lengths are generated.

3. The method according to claim 2, characterized in that, The process of recursively parsing each node and all its child nodes based on the depth-first principle, and calling a dynamic decision function to determine the slicing strategy, includes: Determine and pass in the current node to be recursively parsed; Add the content of the current node to the current text slice, and call the dynamic decision function to determine the slicing strategy; Recursively traverse each child node of the current node and execute the call to the dynamic decision function to determine the slicing strategy; After completing the recursive traversal of all child nodes, the dynamic decision function is called again to determine the slicing strategy, complete the recursive parsing of the current node, and jump to the next node.

4. The method according to claim 2, characterized in that, The dynamic decision function includes: Determine the relationship between the current text slice length and the preset threshold; Determine the changes in node type attributes; Determine the changes in the semantic labels of nodes; Determine whether the preset forced slicing points are met.

5. The method according to claim 1, characterized in that, The step of performing structural parsing on the target document to be processed to generate a directory tree of the target document includes: Input the target document to be processed and perform document parsing; Document elements are extracted based on the document parsing results; wherein, the document elements include headings, paragraph text, and tables; Extract heading levels from document elements of heading type and create heading nodes; Extract the text content from document elements of paragraph text type and create paragraph text nodes; Extract table content and titles from table-type document elements and create table nodes; Based on the hierarchical relationship of the target documents, pointers are constructed to output the directory tree of the target documents.

6. The method according to claim 1, characterized in that, Before invoking the dynamic recursive slicing algorithm to traverse and parse the directory tree of the target document and generate multiple text slices of dynamic length, the method further includes: Determine the document type of the target document; Configure the text slicing granularity based on the document type of the target document.

7. A document processing apparatus, characterized in that, The device includes: The document acquisition module is used to acquire the target document to be processed in response to document input instructions; The directory tree generation module is used to perform structural parsing on the target document to be processed and generate a directory tree of the target document; The text slicing module is used to call the dynamic recursive slicing algorithm to traverse and parse the directory tree of the target document and generate multiple text slices of dynamic length. The slice upload module is used to vectorize the multiple dynamically lengthed text slices, upload and store them in the database of the retrieval enhancement generation system.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images