Text processing method and device, retrieval method and device, electronic equipment and storage medium

Abstract

The invention provides a text processing method and device, a retrieval method and device, electronic equipment and a storage medium. The processing method comprises the steps of obtaining a to-be-processed original document; determining the position relationship between each title or pseudo title in the original document and the corresponding text block; performing title hierarchical relationship analysis on the position relationship to obtain a title hierarchical structure between the text blocks; constructing a text hierarchical tree based on a mapping relationship between the title or the pseudo title of each layer in the title hierarchical structure and the corresponding index text block; and flattening and storing node information of each node on the text hierarchy tree into a vector database so as to retrieve a complete text through the mapping relationship between the text blocks during retrieval. According to the method, the text processing process before the vector storage is optimized, the existing structure information of the original document is introduced, the retrieval based on the incidence relation between the text blocks during retrieval is facilitated, the retrieved text is more complete, and the accuracy of the text recall rate is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a text processing method, a text vector retrieval method, a device, an electronic device and a storage medium. Background Art

[0002] In the related technology, with the explosive growth of Internet information, people's demand for text content generation has become more and more frequent. The retrieval enhancement technology of large language models (LLM) combined with vector databases has been widely used in knowledge questions and answers in vertical fields. For document knowledge in vertical fields, the text-based documents are encoded into vector form for storage through the text encoding model. However, due to the input length limit of the text encoding model, it is usually necessary to pre-segment the text when building a vector database, that is, to divide the text into fixed-length text blocks according to the preset text block length. In order to maintain some contextual semantics between text blocks, the contents of two adjacent text blocks will be overlapped during segmentation to ensure that there is an intersection between the previous and next text blocks.

[0003] However, the existing text block segmentation method is rather violent. Although some content overlap is ensured when the previous and subsequent text blocks are segmented, some discrete text blocks will still be recalled during recall. Due to the violent overlap between text blocks, the number of overlapping words is difficult to control and usually needs to be manually set based on past experience. In particular, when context information is introduced, two unrelated texts will forcibly introduce context information, thereby increasing the interference information of the retrieval and reducing the accuracy of text block recall and the retrieval capability of large language models. Summary of the invention

[0004] The present invention provides a text processing method, a text vector retrieval method, a device, an electronic device and a storage medium, so as to at least solve the problem in the related art that due to the overlapping segmentation method between text blocks, two irrelevant text blocks forcibly introduce context information, increase the interference information of the retrieval, and reduce the accuracy of the text block recall. The technical solution of the present invention is as follows:

[0005] According to a first aspect of an embodiment of the present invention, there is provided a text processing method, comprising:

[0006] Obtaining original documents to be processed;

[0007] Determining the positional relationship between each title or pseudo-title in the original document and the corresponding text block;

[0008] Performing title hierarchical relationship analysis on the positional relationship between each title or pseudo-title and the corresponding text block to obtain a title hierarchical structure between the text blocks;

[0009] Constructing a text hierarchy tree based on the mapping relationship between the title or pseudo-title of each layer in the title hierarchy structure and the corresponding index text block;

[0010] The node information of each node on the text hierarchy tree is flattened and stored in a vector database, so that all text blocks can be retrieved through the hierarchical relationship between text blocks during retrieval.

[0011] Optionally, determining the positional relationship between the title or pseudo-title and the corresponding text block in the original document includes:

[0012] Detecting the layout of the original document;

[0013] When detecting that there is a title in the original document, determining a positional relationship between the title and a corresponding text block in the original document;

[0014] When it is detected that no title exists in the original document, a positional relationship between the pseudo title and the corresponding text block in the original document is determined.

[0015] Optionally, before detecting the layout of the original document, determining the positional relationship between the title or pseudo-title and the corresponding text block in the original document further includes:

[0016] Convert the original document into a PDF document;

[0017] Extract each page of the PDF document in the PDF document into a picture format to obtain multiple pictures;

[0018] The detecting the layout of the original document includes: detecting the layout of multiple pictures of the original document.

[0019] Optionally, determining the positional relationship between the title and the corresponding text block in the original document includes:

[0020] Analyze all titles of the original document to determine each title and a corresponding text block in the original document; and cluster and group the determined titles and corresponding text blocks to obtain a positional relationship between the titles and the corresponding text blocks;

[0021] The determining the positional relationship between the pseudo title and the corresponding text block in the original document includes:

[0022] The original document is segmented according to the set window size to obtain a plurality of text blocks; the pseudo title corresponding to each text block after segmentation and the positional relationship between each pseudo title and the corresponding text block are determined.

[0023] Optionally, performing title level relationship analysis on the positional relationship between each title or pseudo-title and the corresponding text block to obtain a title level structure between the text blocks includes:

[0024] Performing a title hierarchical relationship analysis on the positional relationship between each title or pseudo title and the corresponding text block to determine the hierarchical relationship between each title or pseudo title and the corresponding text block;

[0025] Based on the hierarchical relationship, similar titles or pseudo titles are merged using a large language model, and the merged title name or pseudo title name is renamed to obtain a mapping relationship between the renamed title or pseudo title and the corresponding index text block;

[0026] The mapping relationship between the title or pseudo-title and the corresponding index text block is extracted in a recursive manner, and the title hierarchical structure between the text blocks is generated based on the mapping relationship.

[0027] Optionally, performing title level relationship analysis on the positional relationship between each title or pseudo-title and the corresponding text block to obtain a title level structure between the text blocks includes:

[0028] Based on the positional relationship between each title or pseudo-title and the corresponding text block, determining the hierarchical relationship between each title or pseudo-title and the corresponding text block;

[0029] Based on the hierarchical relationship, encoding each text block;

[0030] Determine the cosine similarity of the encoded pairwise text blocks;

[0031] The text blocks whose cosine similarity values ​​are greater than a threshold are hierarchically clustered and grouped to obtain a hierarchical structure of titles between the text blocks.

[0032] Optionally, constructing a text hierarchy tree based on a mapping relationship between a title or pseudo-title of each layer in the title hierarchy structure and a corresponding index text block includes:

[0033] Traversing each level of the title hierarchy structure, extracting a mapping relationship between the title or pseudo-title of each level and a corresponding index text block;

[0034] According to the mapping relationship between the title or pseudo-title of each level and the corresponding index text block, a text hierarchy tree is constructed, wherein the text hierarchy tree includes: a root node, leaf nodes, layer information, all node information, and a mapping relationship between layer indexes and corresponding nodes.

[0035] According to a second aspect of an embodiment of the present invention, a text vector retrieval method is provided, comprising:

[0036] Get the text vector of the content to be detected;

[0037] The text vector is matched with the vector of the vector database to obtain a matching result; wherein the vector database stores the node information of each node on the text hierarchy tree, and the text hierarchy tree is based on the title hierarchy relationship analysis of the positional relationship between each title or pseudo-title in the original document and the corresponding text block to obtain the title hierarchy structure between the text blocks; and is constructed based on the mapping relationship between the title or pseudo-title of each layer in the title hierarchy structure and the corresponding index text block;

[0038] When the matching result is a parent node, based on the hierarchical relationship in the node information of the parent node, the text blocks of all child nodes under the parent node are searched from the vector database.

[0039] According to a third aspect of an embodiment of the present invention, there is provided a text processing device, comprising:

[0040] An acquisition module, used for acquiring the original document to be processed;

[0041] A determination module, used to determine the positional relationship between each title or pseudo-title in the original document and the corresponding text block;

[0042] An analysis module, used to perform title level relationship analysis on the positional relationship between each title or pseudo-title and the corresponding text block, to obtain a title level structure between the text blocks;

[0043] A construction module, used for constructing a text hierarchy tree based on the mapping relationship between the title or pseudo-title of each layer in the title hierarchy structure and the corresponding index text block;

[0044] The storage module is used to store the node information of each node on the text hierarchy tree in a flattened manner in a vector database, so that all text blocks can be retrieved through the hierarchical relationship between the text blocks during retrieval.

[0045] Optionally, the determining module includes:

[0046] A detection module, used for detecting the layout of the original document;

[0047] A first determination module, configured to determine a positional relationship between a title and a corresponding text block in the original document when the detection module detects that there is a title in the original document;

[0048] The second determination module is used to determine the positional relationship between the pseudo title and the corresponding text block in the original document when the detection module detects that there is no title in the original document.

[0049] Optionally, the determining module further includes:

[0050] A conversion module, used for converting the original document into a PDF document before the detection module detects the layout of the original document;

[0051] A first extraction module, used to extract each page of the PDF document in the PDF document into a picture format to obtain multiple pictures;

[0052] The detection module is specifically used to detect the layout of multiple pictures of the original document.

[0053] Optionally, the first determining module includes:

[0054] A title analysis module, used for analyzing all titles of the original document;

[0055] A first title determination module, used to determine each title and corresponding text block in the original document;

[0056] A first clustering module, used for clustering and grouping the titles and corresponding text blocks determined by the first title determination module to obtain positional relationships between the titles and corresponding text blocks;

[0057] The second determining module comprises:

[0058] A segmentation module, used for segmenting the original document into text blocks according to a set window size to obtain a plurality of text blocks;

[0059] The second title determination module is used to determine the pseudo title corresponding to each text block after segmentation, and the positional relationship between each pseudo title and the corresponding text block.

[0060] Optionally, the analysis module includes:

[0061] A first relationship determination module is used to analyze the positional relationship between each title or pseudo title and the corresponding text block, and determine the hierarchical relationship between each title or pseudo title and the corresponding text block;

[0062] A similarity merging module is used to merge similar titles or pseudo titles based on the hierarchical relationship using a large language model, and rename the merged title name or pseudo title name to obtain a mapping relationship between the renamed title or pseudo title and the corresponding index text block;

[0063] The generation module is used to extract the mapping relationship between the title or pseudo-title and the corresponding index text block in a recursive manner, and generate a title hierarchy structure between the text blocks based on the mapping relationship.

[0064] Optionally, the analysis module includes:

[0065] A first relationship determination module is used to analyze the positional relationship between each title or pseudo title and the corresponding text block, and determine the hierarchical relationship between each title or pseudo title and the corresponding text block;

[0066] An encoding module, used for encoding each text block based on the hierarchical relationship;

[0067] A similarity determination module, used to determine the cosine similarity of two encoded text blocks;

[0068] The clustering module is used to perform hierarchical clustering and grouping on the text blocks whose cosine similarity values ​​are greater than a threshold value, so as to obtain a hierarchical structure of titles between the text blocks.

[0069] Optionally, the building blocks include:

[0070] A traversal module, used for traversing each level of the title hierarchy structure;

[0071] An extraction module, used for extracting the mapping relationship between the title or pseudo-title of each level and the corresponding index text block;

[0072] According to the mapping relationship between the title or pseudo-title of each level and the corresponding index text block, a text hierarchy tree is constructed, wherein the text hierarchy tree includes: a root node, leaf nodes, layer information, all node information, and a mapping relationship between layer indexes and corresponding nodes.

[0073] According to a fourth aspect of an embodiment of the present invention, a text vector retrieval device is provided, comprising:

[0074] An acquisition module is used to obtain the text vector of the content to be detected;

[0075] A matching module, used for matching the text vector with the vector of the vector database to obtain a matching result; wherein the vector database stores the node information of each node on the text hierarchy tree, and the text hierarchy tree is based on the title hierarchy relationship analysis of the positional relationship between each title or pseudo-title in the original document and the corresponding text block to obtain the title hierarchy structure between the text blocks; and is constructed based on the mapping relationship between the title or pseudo-title of each layer in the title hierarchy structure and the corresponding index text block;

[0076] A search module is used to search for text blocks of all child nodes under the parent node from the vector database based on the hierarchical relationship in the node information of the parent node when the matching result is a parent node.

[0077] According to a fifth aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0078] processor;

[0079] a memory for storing instructions executable by the processor;

[0080] The processor is configured to execute the instructions to implement the text processing method or the text vector retrieval method as described above.

[0081] According to a sixth aspect of an embodiment of the present invention, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the text processing method as described above or the text vector retrieval method as described above.

[0082] According to a seventh aspect of an embodiment of the present invention, there is provided a computer program product, comprising a computer program or instructions, which, when executed by a processor of an electronic device, implements the text processing method as described above or the text vector retrieval method as described above.

[0083] The technical solution provided by the embodiments of the present invention brings at least the following beneficial effects:

[0084] In an embodiment of the present invention, an original document to be processed is obtained; the positional relationship between each title or pseudo-title in the original document and the corresponding text block is determined; the positional relationship between each title or pseudo-title and the corresponding text block is analyzed to obtain the title hierarchy structure between the text blocks; a text hierarchy tree is constructed based on the mapping relationship between the title or pseudo-title of each layer in the title hierarchy structure and the corresponding index text block; the node information of each node on the text hierarchy tree is flattened and stored in a vector database, so that the complete text can be retrieved through the mapping relationship between the text blocks during retrieval. That is to say, the embodiment of the present invention optimizes the text processing process before the vector is stored in the library, first determines the positional relationship between each title or pseudo-title in the original document and the corresponding text block, introduces the existing structural information of the original document in advance, and then analyzes the title hierarchy relationship based on the positional relationship to obtain the title hierarchy structure between the text blocks; constructs a text hierarchy tree based on the title hierarchy structure, and flattens the node information of each node in the text hierarchy tree and stores it in the vector database, so that it is convenient to find the connection between the text blocks during indexing, so that the retrieved text is more complete, and the accuracy of text recall is improved.

[0085] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention, and do not constitute an improper limitation of the present invention. In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0087] Figure 1 It is a flowchart of a text processing method provided by an embodiment of the present invention.

[0088] Figure 2 It is a schematic diagram of determining a hierarchical relationship based on text block similarity provided by an embodiment of the present invention.

[0089] Figure 3 It is a flowchart of a text vector retrieval method provided by an embodiment of the present invention.

[0090] Figure 4 It is a block diagram of a text processing device provided by an embodiment of the present invention.

[0091] Figure 5 It is a block diagram of a determination module provided by an embodiment of the present invention.

[0092] Figure 6 It is another block diagram of a determination module provided by an embodiment of the present invention.

[0093] Figure 7 It is a block diagram of a text processing device provided by an embodiment of the present invention.

[0094] Figure 8 It is a block diagram of an electronic device provided by an embodiment of the present invention.

[0095] Fig. 9 The invention is a block diagram of a device for text processing provided by an embodiment of the invention. DETAILED DESCRIPTION

[0096] In order to enable ordinary persons in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0097] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0098] Figure 1 is a flowchart of a text processing method provided by an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0099] Step 101: Obtain an original document to be processed.

[0100] Step 102: Determine the positional relationship between each title or pseudo-title in the original document and the corresponding text block.

[0101] Step 103: Perform title level relationship analysis on the positional relationship between each title or pseudo-title and the corresponding text block to obtain the title level structure between the text blocks.

[0102] Step 104: construct a text hierarchy tree based on the mapping relationship between the title or pseudo-title of each layer in the title hierarchy structure and the corresponding index text block.

[0103] Step 105: Flatten the node information of each node on the text hierarchy tree and store it in a vector database, so that all text blocks can be retrieved through the hierarchical relationship between text blocks during retrieval.

[0104] The text processing method described in the present invention can be applied to terminals, servers, etc., without limitation herein. The terminal implementation device can be electronic devices such as smart phones, laptops, tablet computers, desktop computers, personal digital assistants (PDAs) and wearable devices. The server can be an independent server or a server cluster, or a server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, intermediate services, domain name services, security services, content distribution networks, or a big data and artificial intelligence platform, etc., without limitation herein.

[0105] Combine the following Figure 1 , the specific implementation steps of a text processing method provided by an embodiment of the present invention are described in detail.

[0106] In step 101, an original document to be processed is obtained.

[0107] The original document to be processed in this step can be a document in any format, such as doc, docx, txt, etc. The acquisition method can be obtained locally or remotely, and the specific acquisition method is not limited in this embodiment.

[0108] In step 102, the positional relationship between each title or pseudo-title in the original document and the corresponding text block is determined.

[0109] In this step, the layout of the original document is detected. When it is detected that there is a title (which can also be understood as a title box) in the original document, it is indicated that the format of the original document is a standard format, and the positional relationship between the title and the corresponding text block in the original document is determined; or when it is detected that there is no title in the original document, it is indicated that the format of the original document is a non-standard format, and the positional relationship between the pseudo title and the corresponding text block in the original document is determined.

[0110] The layout of the original document can be detected by a layout detection model or a document layout detection algorithm, that is, the coordinate frames of key text blocks such as text titles, paragraph blocks, etc. of the original document can be detected. Of course, in this embodiment, the layout of the original document can also be detected by other detection methods, which are not limited in this embodiment.

[0111] In this embodiment, the layout detection model is usually a deep learning model, which is a common open source model. The open source model may include but is not limited to: document intelligent multimodal pre-training model (layoutlmv3), doclayout_yolo (a document layout analysis tool based on yolo-v10), etc. These open source models can detect text titles, paragraph blocks, and line blocks in documents, and then they can be selected according to specific task scenarios.

[0112] In this embodiment, the original document is detected through the layout detection model, and the positional relationship between the title in the document and the corresponding text block can be obtained, that is, the subordinate relationship can be roughly obtained. However, through the detection of the open source model, the hierarchical relationship between specific titles cannot be detected.

[0113] Based on this, the embodiment of the present invention sorts the detection boxes of each title according to their height, and then groups them using a clustering algorithm (such as a density-based clustering algorithm DBSCAN, etc.), so that the hierarchical relationship between the multi-level titles can be divided in order from high to low.

[0114] After identifying the text position, the embodiment of the present invention can also perform optical character recognition (OCR) detection on the text block and the document title area to convert the text block in the original document into text format data. The document content is basically in regular fonts, which is not a complex task for the OCR algorithm and is relatively easy to operate. Among them, the OCR algorithm in this embodiment includes but is not limited to the text recognition model kit (PaddleOCR), convolutional recurrent neural network (CRNN) algorithm and other open source algorithms.

[0115] Among them, when the presence of titles in the original document is detected by the layout detection model, it is indicated that the format of the original document is a standard format, and the positional relationship between the titles and the corresponding text blocks in the original document is determined; specifically, it includes: analyzing all the titles of the original document to determine each title and the corresponding text block in the original document; and clustering and grouping the determined titles and corresponding text blocks to obtain the positional relationship between the titles and the corresponding text blocks.

[0116] That is to say, for an original document in a standard format (such as doc, docx, etc.), the positional relationship between the title and the text block in the document can be obtained by combining the layout detection algorithm in the layout detection model with the clustering algorithm.

[0117] Among them, when it is detected through the layout detection model that there is no title in the original document, it means that the format of the original document is a non-standard format, and the positional relationship between the pseudo-title and the corresponding text block in the original document is determined, specifically including: dividing the text of the original document according to the set window size to obtain multiple text blocks; determining the pseudo-title corresponding to each text block after segmentation, and the positional relationship between each pseudo-title and the corresponding text block.

[0118] That is to say, for original documents in non-standard formats (such as notes, etc.), they are divided into blocks according to a fixed window size, and the segmented text blocks are sent to a large language model for title prediction. After the prediction, the large language model will return a pseudo-title for each text block. This embodiment uses these pseudo-titles as the titles of the corresponding text blocks.

[0119] Optionally, in another embodiment, before detecting the layout of the original document, determining the positional relationship between the title or pseudo-title and the corresponding text block in the original document further includes: converting the original document into a portable document format (PDF, Portable Document Format)) document; then, extracting each page of the PDF document in the PDF document into a picture format to obtain a plurality of pictures;

[0120] The detecting of the layout of the original document includes: detecting the layout of multiple pictures of the original document.

[0121] In this embodiment, for an input document of any format, the document is first converted into PDF format, each page of the PDF document is extracted into an image format, and then the text content of each image is subjected to layout detection, that is, the coordinate frames of key text blocks such as titles and paragraphs in the text content are detected, and the purpose of the detection is to obtain the positional relationship between the titles and text blocks in the document. The detection process is the same as above and will not be repeated here.

[0122] In step 103, a title level relationship analysis is performed on the positional relationship between each title or pseudo title and the corresponding text block to obtain a title level structure between the text blocks.

[0123] In this step, first, the positional relationship between each title or pseudo-title and the corresponding text block is analyzed to determine the hierarchical relationship between each title or pseudo-title and the corresponding text block; secondly, based on the hierarchical relationship, similar titles or pseudo-titles are merged using a large language model, and the merged title name or pseudo-title name is renamed to obtain a mapping relationship between the renamed title or pseudo-title and the corresponding index text block; finally, the mapping relationship between the title or pseudo-title and the corresponding index text block is extracted recursively, and a title hierarchy structure is generated based on the mapping relationship.

[0124] In this embodiment, based on the positional relationship analysis between each title or pseudo-title and the corresponding text block, the hierarchical relationship between each title or pseudo-title and the corresponding text block can be determined. Then, the hierarchical relationship is input into a large language model to analyze the title hierarchical relationship. For example, a basic prompt word scheme is provided as shown below:

[0125]

[0126]

[0127] That is to say, the input is a text title and a title index (or a pseudo title and a pseudo title index). The large language model will merge and combine similar titles (or pseudo titles), redefine the combined title name (or pseudo title name), and finally output a mapping relationship between a title (or pseudo title) and the corresponding index text block. For the returned structured data, the mapping relationship between the title or pseudo title and the corresponding index text block can be extracted recursively, thereby obtaining title index information similar to a directory structure.

[0128] It should be noted that, in addition to the above-mentioned use of a large language model to analyze the positional relationship between each title or pseudo-title and the corresponding text block to obtain a title hierarchical structure, this embodiment can also perform hierarchical clustering of text blocks according to a text block threshold to determine the hierarchical relationship and then determine the title hierarchical structure, that is, perform hierarchical clustering of text blocks according to the text block threshold, that is, classify text blocks greater than the threshold into one group, and classify text blocks not greater than the threshold into another group. This method saves the cost of calling a large language model and replaces it with a cheaper vector encoding scheme. By encoding each text block and then calculating the cosine similarity between each two to perform grouping, it specifically includes:

[0129] Based on the positional relationship between each title or pseudo-title and the corresponding text block, determine the hierarchical relationship between each title or pseudo-title and the corresponding text block; based on the hierarchical relationship, encode each text block; determine the cosine similarity between the encoded text blocks; perform hierarchical clustering and grouping on the text blocks whose cosine similarity values ​​are greater than a threshold, and obtain a title hierarchical structure. Figure 2 As shown, Figure 2 A schematic diagram of determining a hierarchical relationship based on text block similarity provided by an embodiment of the present invention. Figure 2 In the example, the encoded text blocks include chunk1, chunk2, chunk3, ... chunkn; then, the cosine similarity of each pair of text blocks is calculated to determine whether the cosine similarity value is greater than a threshold, and the text blocks greater than the threshold are divided into a group, such as group1, and the text blocks not greater than the threshold are divided into another group, such as group2. It can also be understood as determining the proportion of the current text block and the grouped text with a cosine similarity greater than the threshold to group them. For example, after calculating and comparing the cosine similarity, chunk1, chunk2, and chunk3 are divided into group1, etc.

[0130] In this embodiment, each text block is encoded, and each text block can be encoded using a text encoder, wherein the text encoder includes but is not limited to chatgpt, BGE, BCE, etc. The grouping logic needs to be manually designed, such as determining the proportion of the current text block and the grouped text with a similarity greater than a threshold, etc., and is specifically debugged according to the actual effect.

[0131] In step 104, a text hierarchy tree is constructed based on the mapping relationship between the title or pseudo-title of each layer in the title hierarchy structure and the corresponding index text block.

[0132] In this step, each level of the title hierarchy structure is first traversed to extract the mapping relationship between the title or pseudo-title of each level and the corresponding index text block; then, according to the mapping relationship between the title or pseudo-title identifier of each level and the corresponding index text block, a text hierarchy tree is constructed, wherein the text hierarchy tree includes: a root node, leaf nodes, layer information, all node information, and a mapping relationship between layer indexes and corresponding nodes. In this embodiment, the purpose of constructing the text hierarchy tree is to obtain the association relationship between text blocks.

[0133] That is to say, in this embodiment, after obtaining the title hierarchy structure, the levels of the text hierarchy tree can be recursively traversed to extract the mapping relationship between the title and the text block index, and a text hierarchy tree can be rebuilt based on the extracted mapping relationship. The text hierarchy tree includes the following attribute values: (1) root node, (2) leaf node, (3) layer information, (4) all node information, and (5) layer index and corresponding node mapping relationship.

[0134] The node information of each node includes the following attributes: (1) node unique ID, (2) text, (3) text encoding vector, (4) parent node ID, and (5) related question attributes. A hierarchical tree can be reconstructed using the node information of each node, and the hierarchical relationship between text blocks can be obtained. The text encoding vector in this embodiment is the vectorization result obtained by inputting the text into the text encoder, which can be implemented by a large language model. The details are as mentioned above and will not be repeated here.

[0135] It should be noted that the related question attributes in the node information use the HyDE method to generate pseudo queries to enhance the retrieval accuracy. Specifically, some questions or document sections related to the current text block can be collected or generated by a large language model to narrow the distance between the query and the queried text (matching from query to text block, converted to query to query, or query to summary, etc.).

[0136] In step 105, the node information of each node on the text hierarchy tree is flattened and stored in a vector database, so that all text blocks can be retrieved through the hierarchical relationship between text blocks during retrieval.

[0137] In this step, after the text hierarchy tree is rebuilt, the node information of each node can be extracted and written into the vector database in a flat manner. It should be noted that there is no special requirement for the type of vector database in this embodiment, and it only needs to support vector retrieval. Among them, the vector database of this embodiment includes but is not limited to ElasticSearch (Lucene-based search server), Millvus (open source high-performance vector database), ChromaDB (vector database), etc. Among them, the flat storage in this embodiment is to reduce the management level, shorten the information transmission chain, etc., such as storing the association relationship between the id-text block-vector of each node in the vector database, etc. When searching by vector, all text blocks are retrieved through the association relationship between the vector and the text block.

[0138] In an embodiment of the present invention, the original document to be processed is obtained; the positional relationship between each title or pseudo-title in the original document and the corresponding text block is determined; the positional relationship between each title or pseudo-title and the corresponding text block is analyzed to obtain a title hierarchy structure; a text hierarchy tree is constructed based on the mapping relationship between the title or pseudo-title of each layer in the title hierarchy structure and the corresponding index text block; the node information of each node on the text hierarchy tree is flattened and stored in a vector database. That is to say, in an embodiment of the present invention, the text processing process before the vector is stored in the library is optimized, the positional relationship between each title or pseudo-title in the original document and the corresponding text block is first determined, the existing structural information of the original document is introduced in advance, and the title hierarchy relationship analysis is performed based on the positional relationship to obtain the title hierarchy structure between the text blocks; the text hierarchy tree is constructed based on the title hierarchy structure, and the node information of each node in the text hierarchy tree is flattened and stored in the vector database, so as to facilitate the search for the connection between the text blocks during indexing, make the retrieved text blocks more complete, and improve the text recall rate.

[0139] In the embodiments of the present invention, any format of document content can be targeted, including documents in standard format or files in non-standard format (i.e., documents without titles and document hierarchy). The present invention introduces document layout analysis before text segmentation, and introduces the hierarchical relationship of the original text on the basis of brute force segmentation, making full use of the prior knowledge of the original data and improving the recall rate of subsequent text blocks.

[0140] See also Figure 3 , is a flowchart of a text vector retrieval method provided by an embodiment of the present invention, the method comprising:

[0141] Step 301: Obtaining a text vector of the content to be detected;

[0142] Step 302: matching the text vector with the vector in the vector database to obtain a matching result; wherein the vector database stores node information of each node on the text hierarchy tree, and the text hierarchy tree is based on analyzing the positional relationship between each title or pseudo-title in the original document and the corresponding text block to obtain a title hierarchy structure between text blocks; and is constructed based on the mapping relationship between the title or pseudo-title of each layer in the title hierarchy structure and the corresponding index text block;

[0143] Step 303: When the matching result is a parent node, based on the hierarchical relationship in the node information of the parent node, the text blocks of all child nodes under the parent node are searched from the vector database.

[0144] In this embodiment, the design of the retrieval strategy is crucial to the recall results. The purpose of the process of constructing the text hierarchy tree is to obtain the association relationship between text blocks. Therefore, in the retrieval stage, this embodiment makes full use of these association relationships for retrieval, especially the parent-child relationship in the text hierarchy tree, to recall all the child text blocks involved.

[0145] In an embodiment of the present invention, the content to be detected is first obtained, and then the content to be detected is converted into a text vector, and the document vector is matched with the vector of the vector database to obtain a matching result; when the matching result is a parent node, based on the node information of the parent node (such as the hierarchical relationship of the parent node, that is, one or more child node IDs related to the parent node, etc.), it is matched with the keywords of the vector database to match all child nodes under the parent node; or when the matching result is a child node, the search is ended directly.

[0146] That is to say, in this embodiment, when searching, you can choose to use text vectors for searching first. When the search result is a parent node, you can further search for text blocks of all child nodes under the parent node, that is, use a search method that mixes vector search and keyword search. This can avoid missing some important but isolated text blocks that are distributed far away.

[0147] Therefore, in the embodiment of the present invention, when using text vector retrieval, if the retrieval result itself is a parent node, all child nodes under the parent node can be found through the hierarchical relationship, and all sub-text blocks under the text block can be obtained. These sub-text blocks can be summarized and refined, or directly sent to a large language model, which can ensure a more comprehensive recall. Furthermore, during retrieval, this embodiment can also adopt a hybrid retrieval method, that is, mixing the keywords of the text block itself and the keywords of related questions to improve the accuracy of text block recall.

[0148] The embodiment of the present invention proposes combining a document layout analysis algorithm with a large language model (LLM) to analyze the content of a text block, find the hierarchical relationship between text blocks through methods such as hierarchical clustering, use the connection between text blocks as a priori for recall, and improve the accuracy of text block recall, thereby improving the retrieval enhancement capability of a large language model.

[0149] The embodiment of the present invention should use a large language model to infer the hierarchical relationship of document blocks, obtain the positional relationship between ordered or unordered text blocks, analyze the title hierarchical relationship based on the positional relationship, and establish a text hierarchical tree (i.e., relationship tree). In the retrieval stage, all text blocks involved are recalled using the parent-child relationship of the tree structure, solving the problems of isolated text blocks and missed recalls, and then superimposing a hybrid retrieval method to greatly improve the accuracy of text block recall.

[0150] Based on the existing document question-answering retrieval method, the large language model may not give accurate responses in some scenarios, which may lead to omissions, duplications, errors, and other problems in the recalled text blocks. This paper proposes a document layout analysis algorithm combined with the LLM large model to analyze the content of the text block, find the hierarchical relationship between the text blocks through hierarchical clustering and other methods, and use the relationship between the text blocks as the prior for recall, thereby improving the accuracy of the text block recall, thereby improving the retrieval enhancement capability of the large model.

[0151] It should be noted that, for the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that this implementation disclosure is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present invention.

[0152] Figure 4 4 is a block diagram of a text processing device provided by an embodiment of the present invention. The device comprises: an acquisition module 401, a determination module 402, an analysis module 403, a construction module 404 and a storage module 405, wherein:

[0153] The acquisition module 401 is used to acquire the original document to be processed;

[0154] A determination module 402 is used to determine the positional relationship between each title or pseudo title in the original document and the corresponding text block;

[0155] An analysis module 403 is used to perform a title level relationship analysis on the positional relationship between each title or pseudo title and the corresponding text block to obtain a title level structure between the text blocks;

[0156] A construction module 404 is used to construct a text hierarchy tree based on the mapping relationship between the title or pseudo-title of each layer in the title hierarchy structure and the corresponding index text block;

[0157] The storage module 405 is used to store the node information of each node on the text hierarchy tree in a flattened manner into a vector database, so as to retrieve all text blocks through the hierarchical relationship between text blocks during retrieval.

[0158] Optionally, in another embodiment, based on the above embodiment, the determination module 402 includes: a detection module 501, a first determination module 502 and a second determination module 503, and its structural block diagram is as follows: Figure 5 As shown,

[0159] A detection module 501 is used to detect the layout of the original document;

[0160] A first determination module 502, configured to determine a positional relationship between a title and a corresponding text block in the original document when the detection module 501 detects that there is a title in the original document;

[0161] The second determination module 503 is used to determine the positional relationship between the pseudo title and the corresponding text block in the original document when the detection module 501 detects that there is no title in the original document.

[0162] Optionally, in another embodiment, based on the above embodiment, the determination module 402 further includes: a conversion module 601 and a first extraction module 602, wherein the structural block diagram thereof is as follows: Figure 6 As shown,

[0163] The conversion module 601 is used to convert the original document into a PDF document before the detection module 501 detects the layout of the original document;

[0164] A first extraction module 602 is used to extract each page of the PDF document in the PDF document into a picture format to obtain multiple pictures;

[0165] The detection module 501 is specifically used to detect the layout of multiple pictures of the original document.

[0166] Optionally, in another embodiment, based on the above embodiment, the first determination module includes: a title analysis module, a first title determination module and a first clustering module, wherein:

[0167] A title analysis module, used for analyzing all titles of the original document;

[0168] A first title determination module, used to determine each title and corresponding text block in the original document;

[0169] The first clustering module is used to cluster and group the titles and corresponding text blocks determined by the first title determination module to obtain the positional relationship between the titles and corresponding text blocks.

[0170] Optionally, in another embodiment, based on the above embodiment, the second determination module includes: a segmentation module and a second title determination module, wherein:

[0171] A segmentation module, used for segmenting the original document into text blocks according to a set window size to obtain a plurality of text blocks;

[0172] The second title determination module is used to determine the pseudo title corresponding to each text block after segmentation, and the positional relationship between each pseudo title and the corresponding text block.

[0173] Optionally, in another embodiment, based on the above embodiment, the analysis module includes:

[0174] A first relationship determination module is used to analyze the positional relationship between each title or pseudo title and the corresponding text block, and determine the hierarchical relationship between each title or pseudo title and the corresponding text block;

[0175] A similarity merging module is used to merge similar titles or pseudo titles based on the hierarchical relationship using a large language model, and rename the merged title name or pseudo title name to obtain a mapping relationship between the renamed title or pseudo title and the corresponding index text block;

[0176] The generation module is used to extract the mapping relationship between the title or pseudo-title and the corresponding index text block in a recursive manner, and generate a title hierarchy structure between the text blocks based on the mapping relationship.

[0177] Optionally, in another embodiment, based on the above embodiment, the analysis module includes:

[0178] A first relationship determination module is used to analyze the positional relationship between each title or pseudo title and the corresponding text block, and determine the hierarchical relationship between each title or pseudo title and the corresponding text block;

[0179] An encoding module, used for encoding each text block based on the hierarchical relationship;

[0180] A similarity determination module, used to determine the cosine similarity of two encoded text blocks;

[0181] The clustering module is used to perform hierarchical clustering and grouping on the text blocks whose cosine similarity values ​​are greater than a threshold value, so as to obtain a hierarchical structure of titles between the text blocks.

[0182] Optionally, in another embodiment, based on the above embodiment, the building blocks include:

[0183] A traversal module, used for traversing each level of the title hierarchy structure;

[0184] An extraction module, used for extracting the mapping relationship between the title or pseudo-title of each level and the corresponding index text block;

[0185] According to the mapping relationship between the title or pseudo-title of each level and the corresponding index text block, a text hierarchy tree is constructed, wherein the text hierarchy tree includes: a root node, leaf nodes, layer information, all node information, and a mapping relationship between layer indexes and corresponding nodes.

[0186] See also Figure 7 , is a block diagram of a text vector retrieval device provided by an embodiment of the present invention, the device comprises: an acquisition module 701, a matching module 702 and a search module 703, wherein,

[0187] An acquisition module 701 is used to acquire a text vector of the content to be detected;

[0188] A matching module 702 is used to match the text vector with the vector of the vector database to obtain a matching result; wherein the vector database stores the node information of each node on the text hierarchy tree, and the text hierarchy tree is based on the title hierarchy relationship analysis of the positional relationship between each title or pseudo-title in the original document and the corresponding text block to obtain the title hierarchy structure between the text blocks; and is constructed based on the mapping relationship between the title or pseudo-title of each layer in the title hierarchy structure and the corresponding index text block;

[0189] The search module 703 is used to search for text blocks of all child nodes under the parent node from the vector database based on the hierarchical relationship in the node information of the parent node when the matching result is a parent node.

[0190] Optionally, an embodiment of the present invention further provides an electronic device, including:

[0191] processor;

[0192] a memory for storing instructions executable by the processor;

[0193] The processor is configured to execute the instructions to implement the text processing method or the text vector retrieval method as described above.

[0194] Optionally, an embodiment of the present invention further provides a computer-readable storage medium, which, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the text processing method described above or the text vector retrieval method described above.

[0195] Optionally, an embodiment of the present invention further provides a computer program product, including a computer program or instructions, which, when executed by a processor of an electronic device, implements the text processing method as described above or the text vector retrieval method as described above.

[0196] In an embodiment of the present invention, by combining document layout analysis, the existing structural information of the original document is introduced in advance, and then combined with a large language model, the hierarchical information of the text blocks is extracted, and a hierarchical structure between the text blocks is constructed, which facilitates finding the connection between the text blocks during indexing, making the retrieved text more complete and improving the accuracy of text recall.

[0197] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0198] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0199] Figure 8 800 is a block diagram of an electronic device 800 provided in an embodiment of the present invention. For example, the electronic device 800 may be a mobile terminal or a server, and the electronic device is a mobile terminal as an example for explanation in the embodiment of the present invention. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0200] Reference Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0201] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0202] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0203] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0204] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0205] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0206] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0207] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0208] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0209] In an embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the text processing method or text vector retrieval method shown above.

[0210] In an embodiment, a computer-readable storage medium is also provided, and when the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device 800 can execute the text processing method or text vector retrieval method shown above. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0211] In an embodiment, a computer program product is also provided, including a computer program or instructions. When the computer program or instructions are executed by the processor 820 of the electronic device 800, the electronic device 800 executes the above-mentioned text processing method or text vector retrieval method.

[0212] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0213] Fig. 9900 is a block diagram of a text processing method or text vector retrieval device provided by an embodiment of the present invention. For example, the device 900 can be provided as a server. Fig. 9 , the apparatus 900 includes a processing component 922, which further includes one or more processors, and a memory resource represented by a memory 932 for storing instructions, such as an application, that can be executed by the processing component 922. The application stored in the memory 932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute the instructions to perform the above method.

[0214] The device 900 may also include a power supply component 926 configured to perform power management of the device 900, a wired or wireless network interface 950 configured to connect the device 900 to a network, and an input / output (I / O) interface 958. The device 900 may operate based on an operating system stored in the memory 932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0215] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0216] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A text processing method, characterized in that: include: Obtaining original documents to be processed; Determining the positional relationship between each title or pseudo-title in the original document and the corresponding text block; Performing title hierarchical relationship analysis on the positional relationship between each title or pseudo-title and the corresponding text block to obtain a title hierarchical structure between the text blocks; Constructing a text hierarchy tree based on the mapping relationship between the title or pseudo-title of each layer in the title hierarchy structure and the corresponding index text block; The node information of each node on the text hierarchy tree is flattened and stored in a vector database, so that all text blocks can be retrieved through the hierarchical relationship between text blocks during retrieval.

2. The text processing method according to claim 1, characterized in that: The determining of the positional relationship between the title or pseudo-title and the corresponding text block in the original document includes: Detecting the layout of the original document; When detecting that there is a title in the original document, determining a positional relationship between the title and a corresponding text block in the original document; When it is detected that no title exists in the original document, a positional relationship between the pseudo title and the corresponding text block in the original document is determined.

3. The text processing method according to claim 2, characterized in that: Before detecting the layout of the original document, determining the positional relationship between the title or pseudo-title and the corresponding text block in the original document further includes: Convert the original document into a PDF document; Extract each page of the PDF document in the PDF document into a picture format to obtain multiple pictures; The detecting the layout of the original document includes: detecting the layout of multiple pictures of the original document.

4. The text processing method according to claim 2, characterized in that: The determining the positional relationship between the title and the corresponding text block in the original document includes: Analyze all titles of the original document to determine each title and a corresponding text block in the original document; and cluster and group the determined titles and corresponding text blocks to obtain a positional relationship between the titles and the corresponding text blocks; The determining the positional relationship between the pseudo title and the corresponding text block in the original document includes: The original document is segmented according to the set window size to obtain a plurality of text blocks; the pseudo title corresponding to each text block after segmentation and the positional relationship between each pseudo title and the corresponding text block are determined.

5. The text processing method according to claim 1, characterized in that: The step of performing title hierarchical relationship analysis on the positional relationship between each title or pseudo title and the corresponding text block to obtain a title hierarchical structure between the text blocks includes: Performing a title hierarchical relationship analysis on the positional relationship between each title or pseudo title and the corresponding text block to determine the hierarchical relationship between each title or pseudo title and the corresponding text block; Based on the hierarchical relationship, similar titles or pseudo titles are merged using a large language model, and the merged title name or pseudo title name is renamed to obtain a mapping relationship between the renamed title or pseudo title and the corresponding index text block; The mapping relationship between the title or pseudo-title and the corresponding index text block is extracted in a recursive manner, and the title hierarchical structure between the text blocks is generated based on the mapping relationship.

6. The text processing method according to claim 1, characterized in that: The step of performing title hierarchical relationship analysis on the positional relationship between each title or pseudo title and the corresponding text block to obtain a title hierarchical structure between the text blocks includes: Based on the positional relationship between each title or pseudo-title and the corresponding text block, determining the hierarchical relationship between each title or pseudo-title and the corresponding text block; Based on the hierarchical relationship, encoding each text block; Determine the cosine similarity of the encoded pairwise text blocks; The text blocks whose cosine similarity values ​​are greater than a threshold are hierarchically clustered and grouped to obtain a hierarchical structure of titles between the text blocks.

7. The text processing method according to claim 1, characterized in that: The step of constructing a text hierarchy tree based on a mapping relationship between a title or pseudo-title of each layer in the title hierarchy structure and a corresponding index text block includes: Traversing each level of the title hierarchy structure, extracting a mapping relationship between the title or pseudo-title of each level and a corresponding index text block; According to the mapping relationship between the title or pseudo-title of each level and the corresponding index text block, a text hierarchy tree is constructed, wherein the text hierarchy tree includes: a root node, leaf nodes, layer information, all node information, and a mapping relationship between layer indexes and corresponding nodes.

8. A text vector retrieval method, characterized in that: include: Get the text vector of the content to be detected; The text vector is matched with the vector of the vector database to obtain a matching result; wherein the vector database stores the node information of each node on the text hierarchy tree, and the text hierarchy tree is based on the title hierarchy relationship analysis of the positional relationship between each title or pseudo-title in the original document and the corresponding text block to obtain the title hierarchy structure between the text blocks; and is constructed based on the mapping relationship between the title or pseudo-title of each layer in the title hierarchy structure and the corresponding index text block; When the matching result is a parent node, based on the hierarchical relationship in the node information of the parent node, the text blocks of all child nodes under the parent node are searched from the vector database.

9. A text processing device, characterized in that: include: An acquisition module, used for acquiring the original document to be processed; A determination module, used to determine the positional relationship between each title or pseudo-title in the original document and the corresponding text block; An analysis module, used to perform title level relationship analysis on the positional relationship between each title or pseudo-title and the corresponding text block, to obtain a title level structure between the text blocks; A construction module, used for constructing a text hierarchy tree based on the mapping relationship between the title or pseudo-title of each layer in the title hierarchy structure and the corresponding index text block; The storage module is used to store the node information of each node on the text hierarchy tree in a flattened manner in a vector database, so that all text blocks can be retrieved through the hierarchical relationship between the text blocks during retrieval.

10. A text vector retrieval device, characterized in that: include: An acquisition module is used to obtain the text vector of the content to be detected; A matching module, used for matching the text vector with the vector of the vector database to obtain a matching result; wherein the vector database stores the node information of each node on the text hierarchy tree, and the text hierarchy tree is based on the title hierarchy relationship analysis of the positional relationship between each title or pseudo-title in the original document and the corresponding text block to obtain the title hierarchy structure between the text blocks; and is constructed based on the mapping relationship between the title or pseudo-title of each layer in the title hierarchy structure and the corresponding index text block; A search module is used to search for text blocks of all child nodes under the parent node from the vector database based on the hierarchical relationship in the node information of the parent node when the matching result is a parent node.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the text processing method according to any one of claims 1 to 7 or the text vector retrieval method according to claim 8.

12. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the text processing method according to any one of claims 1 to 7 or the text vector retrieval method according to claim 8.

Images