Document processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the invention provide a document processing method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining a first text of a current dialogue; extracting N first features of the first text; splicing the N first features of the first text into a second feature; acquiring a first hash code of the second feature; in a current document extension tree, a second hash code is retrieved, the similarity between the second hash code and the first hash code is larger than or equal to a first threshold value, the document extension tree comprises a plurality of nodes, each node is associated with at least one document, and the hash codes of the documents associated with the same node are the same; under the condition that the second hash code is retrieved, a document associated with the first target node is determined as a document matched with the current dialogue, and the first target node is a node to which the second hash code belongs; and adjusting the first target node as a root node of the document extension tree. Therefore, according to the embodiment of the invention, the document retrieval speed can be increased.

Description

Technical Field

[0001] The present invention relates to the field of mobile communication technology, and in particular to a document processing method, device, electronic equipment and storage medium. Background Art

[0002] Retrieval-Augmented Generation (RAG) is a technology that combines the power of a pre-trained large language model (LLM) with external data sources. It combines retrieval and generation to retrieve content related to user questions from external data sources and use it as input to the large language model together with the questions, helping the model better understand the context and thus generate more relevant and accurate responses.

[0003] Among them, LLM is a natural language processing model based on deep learning technology, which can process and understand various tasks of human language, such as text generation, text classification, question-answering system, language translation, etc. LLM is trained on large-scale text data to learn the statistical laws and patterns of language, so that it can generate natural and fluent text and deeply understand and analyze the input text.

[0004] The work of Retrieval is to retrieve the most relevant information from a large amount of text data. It usually uses a retrieval system to find text segments related to the input in a large-scale document collection.

[0005] The Generation part uses a language model similar to Generative Pre-Trained Transformer (GPT) to generate responses or answers based on the retrieved information. This process involves understanding the retrieved content and generating coherent, relevant, and informative text based on it.

[0006] Among them, the retrieval part of the current traditional RAG technology is usually optimized based on the vector database, and the current similarity retrieval algorithm of the vector database only provides optimization for a single similar document search, and the retrieval speed is still slow. Summary of the invention

[0007] Embodiments of the present application provide a document processing method, device, electronic device, and storage medium to solve the problem of slow document retrieval speed in the prior art.

[0008] In a first aspect, an embodiment of the present application provides a document processing method, the method comprising:

[0009] Get the first text of the current conversation;

[0010] Extracting N first features of the first text, where N is an integer greater than or equal to 1;

[0011] splicing N first features of the first text into a second feature;

[0012] Obtain a first hash code of the second feature;

[0013] Retrieving a second hash code whose similarity to the first hash code is greater than or equal to a first threshold in the current document spreading tree, wherein the document spreading tree includes a plurality of nodes, each node is associated with at least one document, and the hash codes of the documents associated with the same node are the same;

[0014] In case the second hash code is retrieved, determining a document associated with a first target node as a document matching the current conversation, wherein the first target node is the node to which the second hash code belongs;

[0015] The first target node is adjusted to be the root node of the document spanning tree.

[0016] In a second aspect, an embodiment of the present application provides a document processing device, the device comprising:

[0017] A first acquisition module, used to acquire the first text of the current conversation;

[0018] A first processing module, used for extracting N first features of the first text, where N is an integer greater than or equal to 1;

[0019] A second processing module, configured to combine the N first features of the first text into a second feature;

[0020] A second acquisition module, used to acquire a first hash code of the second feature;

[0021] A first retrieval module is used to retrieve a second hash code whose similarity with the first hash code is greater than or equal to a first threshold in a current document spreading tree, wherein the document spreading tree includes a plurality of nodes, each node is associated with at least one document, and the hash codes of the documents associated with the same node are the same;

[0022] A first determination module, configured to determine, when the second hash code is retrieved, a document associated with a first target node as a document matching the current conversation, wherein the first target node is a node to which the second hash code belongs;

[0023] The first adjustment module is used to adjust the first target node to be the root node of the document extension tree.

[0024] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a transceiver, and a processor:

[0025] A memory for storing a computer program; a transceiver for sending and receiving data under the control of the processor; and a processor for reading the computer program in the memory and executing the document processing method described in the first aspect above.

[0026] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the document processing method described in the first aspect is implemented.

[0027] In an embodiment of the present application, the first text of the current conversation can be obtained, N first features of the first text can be extracted, and the N first features of the first text can be spliced ​​into a second feature, thereby obtaining a first hash code of the second feature, and in the current document extension tree, a second hash code whose similarity with the first hash code is greater than or equal to a first threshold is retrieved, and when the second hash code is retrieved, the document associated with the first target node to which the second hash code belongs is determined as a document matching the current conversation, and the first target node is adjusted to the root node of the document extension tree; wherein N is an integer greater than or equal to 1, the document extension tree includes multiple nodes, each node is associated with at least one document, and the hash codes of the documents associated with the same node are the same.

[0028] Among them, when searching in the document extension tree, traversal often starts from the root node. In the embodiment of the present application, after retrieving a second hash code whose similarity with the first hash code corresponding to the current conversation is greater than or equal to a first threshold, the first target node to which the second hash code belongs is adjusted to the root node of the document extension tree. In this way, the first target node can be traversed preferentially the next time a search is performed in the document extension tree.

[0029] Moreover, in the same round of conversation, the content discussed in the next conversation of the current conversation is often related, that is, there is an implicit association between the contexts of several questions and answers in the same round of conversation. In an embodiment of the present application, after the above-mentioned first target node is adjusted to the root node of the document extension tree, when searching for a hash code similar to the hash code of the next conversation in the document extension tree, the first target node can be traversed first, that is, the document matching the next conversation can be found first, thereby improving the retrieval speed.

[0030] It can be seen that the embodiment of the present application introduces a splay tree and, when performing document retrieval, takes into account the implicit association between the contexts of several questions and answers in the same round of dialogue, thereby being able to retrieve matching documents more quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 A flowchart of a document processing method provided in an embodiment of the present application;

[0033] Figure 2 A schematic diagram of constructing a feature encoder in an embodiment of the present application;

[0034] Figure 3 This is a schematic diagram of the processing flow of the document encoder in the embodiment of the present application;

[0035] Figure 4 A schematic diagram of a document extension tree in an embodiment of the present application;

[0036] Figure 5 A schematic diagram of a topic index expansion tree in an embodiment of the present application;

[0037] Figure 6 A schematic diagram of performing a zig operation in an embodiment of the present application;

[0038] Figure 7 A schematic diagram of performing a left-hand (zag) operation in an embodiment of the present application;

[0039] Figure 8 This is a schematic diagram before the zig-zag operation is performed in an embodiment of the present application;

[0040] Fig. 9 A structural block diagram of a document processing device provided in an embodiment of the present application;

[0041] Fig.10 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In the embodiments of the present application, the term "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0043] In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] The embodiments of the present application provide a document processing method and device to solve the problem of slow document retrieval speed in the prior art.

[0046] Among them, the method and the device are based on the same application concept. Since the method and the device solve the problem in a similar principle, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated.

[0047] Figure 1 A schematic diagram of a document processing method provided by an embodiment of the present application is shown, and the method may include the following steps:

[0048] Step 101: Get the first text of the current conversation.

[0049] The current conversation may be a conversation in a voice clip or a conversation in a video clip. In the embodiment of the present application, when the current conversation is acquired, it is converted into text content.

[0050] Step 102: extracting N first features of the first text.

[0051] Wherein, N is an integer greater than or equal to 1. Optionally, the N first features include: topic features, emotion features, and content summary features.

[0052] Optionally, N first features of the first text are extracted by a feature encoder, wherein the feature encoder may be a large model, such as a decoder-only large model, such as Llama3.1.

[0053] Step 103: concatenate the N first features of the first text into a second feature.

[0054] It can be understood that both the first feature and the second feature can be vectors. After extracting the vectors of N first features of the first text in step 102, the N vectors can be concatenated into one vector.

[0055] Step 104: Obtain a first hash code of the second feature.

[0056] Optionally, a first hash code of the second feature may be obtained through a hash encoder, that is, the second feature may be input into the hash encoder to obtain the first hash code.

[0057] Step 105: Retrieve a second hash code in the current document extension tree, the second hash code having a similarity with the first hash code greater than or equal to a first threshold.

[0058] The document spreading tree includes a plurality of nodes, each node is associated with at least one document, and the documents associated with the same node have the same hash code.

[0059] In addition, a splay tree is a binary sorted tree that can maintain balance in most empty situations through splay operations, complete insertion, search, and deletion operations within O(log n), and ensure that frequently accessed nodes have a higher access speed.

[0060] In the embodiment of the present application, the document expansion tree may be, for example, Figure 4 As shown, each splay tree node is identified by a document hash code (doc_hash), and each node is associated with a document list (docs), and the hash codes of the documents in the document list are all hash codes that identify the node. Figure 4 In the code, left_doc_tree_node indicates the left child node pointer of the document spreading tree node, and right_doc_tree_node indicates the right child node pointer of the document spreading tree node.

[0061] Step 106: When the second hash code is retrieved, the document associated with the first target node is determined as the document matching the current conversation.

[0062] The first target node is the node to which the second hash code belongs.

[0063] In addition, if the similarity between the first Hash value and the second Hash value is greater than or equal to the first threshold, it means that the similarity between the two is relatively high, that is, the feature of the first text is relatively similar to the feature of the document of the second Hash value.

[0064] It can be understood that the hash code of a document is obtained based on the characteristics of the document, so the hash code of a document can represent the characteristics of the document; and the first hash code mentioned above is obtained based on the characteristics of the first text of the current conversation, so the first hash code can represent the characteristics of the current conversation. Therefore, if there is a second hash code in the document spreading tree whose similarity with the first hash code is greater than or equal to the first threshold, it means that there is a document matching the characteristics of the current conversation in the documents associated with the nodes of the document spreading tree.

[0065] In addition, it should be noted that the hash coding in the embodiment of the present application is special, and the similarity difference can be obtained by directly comparing the size of integers.

[0066] Step 107: Adjust the first target node to be the root node of the document extension tree.

[0067] It can be seen from the above steps 101 to 107 that the embodiment of the present application can obtain the first text of the current conversation, extract N first features of the first text, and splice the N first features of the first text into a second feature, thereby obtaining a first hash code of the second feature, and in the current document expansion tree, retrieve a second hash code whose similarity with the first hash code is greater than or equal to a first threshold value, and when the second hash code is retrieved, the document associated with the first target node to which the second hash code belongs is determined as a document matching the current conversation, and the first target node is adjusted to the root node of the document expansion tree; wherein N is an integer greater than or equal to 1, the document expansion tree includes multiple nodes, each node is associated with at least one document, and the hash codes of the documents associated with the same node are the same.

[0068] Among them, when searching in the document extension tree, traversal often starts from the root node. In the embodiment of the present application, after retrieving a second hash code whose similarity with the first hash code corresponding to the current conversation is greater than or equal to a first threshold, the first target node to which the second hash code belongs is adjusted to the root node of the document extension tree. In this way, the first target node can be traversed preferentially the next time a search is performed in the document extension tree.

[0069] Moreover, in the same round of conversation, the content discussed in the next conversation of the current conversation is often related, that is, there is an implicit association between the contexts of several questions and answers in the same round of conversation. In an embodiment of the present application, after the above-mentioned first target node is adjusted to the root node of the document extension tree, when searching for a hash code similar to the hash code of the next conversation in the document extension tree, the first target node can be traversed first, that is, the document matching the next conversation can be found first, thereby improving the retrieval speed.

[0070] It can be seen that the embodiment of the present application introduces a splay tree and, when performing document retrieval, takes into account the implicit association between the contexts of several questions and answers in the same round of dialogue, thereby being able to retrieve matching documents more quickly.

[0071] That is, the splay tree has a memory property: the most recently retrieved documents are closer to the root of the tree, and in the same round of dialogue, each question and answer is in the same context, and the knowledge documents associated with each question are likely to be similar. Therefore, in a continuous dialogue, the embodiment of the present application will be more efficient than the ordinary similarity search algorithm.

[0072] Optionally, in step 102, extracting N first features of the first text includes:

[0073] Extracting N first features of the first text through a feature encoder;

[0074] The process of determining the feature encoder includes the following steps B-1 to B-3:

[0075] Step B-1: When j is an integer from 1 to K, perform the following process to obtain the degree of change corresponding to each specific transformer block corresponding to the 1st to Kth sample documents:

[0076] Inputting the j-th sample document into a language model, obtaining first outputs of M specific transformer blocks of the language model, and second outputs of the language model, wherein transformer block represents a codec block;

[0077] Inputting the second output into the language model to obtain a third output of the M specific transformer blocks;

[0078] Determine the degree of change from the first output to the third output of the same specific transformer block, and obtain the degree of change corresponding to each specific transformer block corresponding to the j-th sample document;

[0079] Step B-2: When v is an integer from 1 to M, perform the following process to obtain the first parameter corresponding to each specific transformer block:

[0080] Calculate the average value of the degree of change corresponding to the v-th specific transformer block corresponding to the 1st to K-th sample documents, and use it as the first parameter corresponding to the v-th specific transformer block;

[0081] Step B-3: Among the M specific transformer blocks, select the adjacent transformer blocks whose first parameter is less than or equal to the third threshold and serve as components of the feature encoder.

[0082] The above-mentioned language model includes multiple transformer blocks, and the multiple transformer blocks are connected in series, that is, among the multiple transformer blocks, for every two adjacent transformer blocks, the output of the former serves as the input of the latter. The above-mentioned specific transformer block includes several transformer blocks among the multiple transformer blocks specified in advance. For example, the above-mentioned specific transformer block includes the 1st to 15th last transformer blocks of the language model. In this case, the ability from the head to this part of the transformer block is sufficient to extract high-level semantics. At the same time, due to the certain distance from the model output, sufficient abstraction ability is retained. The selection of this range balances the accuracy and abstractness of semantic extraction.

[0083] In order to facilitate the understanding of the above steps B-1 to B-3, the following examples are given:

[0084] If the above training samples include 10, the feature encoder determination process is as follows:

[0085] The training sample -1 is used as i1, input into the large model, and output is i2; then i2 is input into the large model again, and output is i3; for these two processes, the degree of change in the output of the same specific transformer block can be compared;

[0086] Similarly, by taking training sample -2 as i1 and repeating the above process, we can compare the degree of change in the output of the same specific transformer block;

[0087] Repeat the above process for training sample -10, and you can get the degree of change in the output of each specific transformer block corresponding to each training sample;

[0088] For example, if the degree of change of a specific transformer block-1 is a1 (corresponding to sample-1), a2 (corresponding to sample-2), ... a10 (corresponding to sample-10), then the average value of a1 to a10 can be calculated, which can be understood as the comprehensive degree of change of the output of the specific transformer block-1 (i.e., the first parameter mentioned above);

[0089] Similarly, the comprehensive change degree of the output of each specific transformer block can be obtained. In this way, by comparing the comprehensive change degree of the output of each specific transformer block with the third threshold, a transformer block whose output is basically unchanged can be found in the specific transformer.

[0090] For example, if the specific transformer block includes the 1st to 15th transformer blocks of the large model, and the comprehensive output change degrees of the 1st to the eth last transformer blocks are all less than or equal to the third threshold, it means that the 1st to the eth last transformer blocks can be selected as the semantic feature extractors (i.e., the above-mentioned feature extractors) for the semantic task.

[0091] Among them, the output of each of the above transformer blocks is a vector, and the degree of change from the first output to the third output can be determined by the distance between the vector corresponding to the above first output and the vector corresponding to the above third output (for example, Euclidean distance or Manhattan distance). The greater the distance between the two, the greater the degree of change from the first output to the second output; or, the degree of change from the first output to the third output can also be determined by the cosine similarity between the vector corresponding to the above first output and the vector corresponding to the above third output, that is, the greater the cosine similarity between the two, the closer they are, that is, the smaller the degree of change from the first output to the third output.

[0092] Among them, cosine similarity is a mathematical measurement method for measuring the similarity between two vectors. It is often used in information retrieval, natural language processing, image processing and other fields to determine whether the directions of two vectors in multidimensional space are close or similar.

[0093] In addition, you can use the t-distributed Stochastic Neighbor Embedding (t-SNE) high-dimensional data visualization tool or directly compare the degree of change of the output before and after each transformer block. Among them, t-SNE is a machine learning algorithm for data visualization, especially suitable for dimensionality reduction of high-dimensional data. The main goal of t-SNE is to represent high-dimensional data in a low-dimensional space (usually 2D or 3D) while maintaining the relative relationship between data points as much as possible. It is particularly good at preserving the local structure of the data, which makes it very suitable for revealing clusters in the data.

[0094] In addition, in some embodiments of the present application, the above-mentioned large model can select a Decoder-Only large model that can better perform semantic extraction tasks, such as Llama3.1, and use pre-designed prompt words to allow the large model to perform semantic extraction tasks (i.e., extract the N first features of the above-mentioned first text).

[0095] Among them, the semantic task prompt words need to clarify the task elements of the semantic task (that is, from what angles and elements to determine the semantics), task steps, task restrictions, and format requirements. Tasks need to be layered and progressive, and broken down and refined layer by layer. The input and output need to be given in a certain format, and the output should include a formatted judgment of the semantics but not other redundant content. In the embodiment of the present application, taking the above-mentioned N kinds of first features including theme features, sentiment features, and content summary features as an example, the specific prompt word design for each semantic extraction task is introduced.

[0096] 1. Extraction of topic features:

[0097] Use the following prompt words to perform the task of extracting topic features:

[0098] Task definition: As a professional text analysis assistant, it is good at extracting and organizing topics from various documents. The specific task is to read the given text carefully and identify the topics in it in a hierarchical manner.

[0099] To extract topic features, you need to follow these steps:

[0100] 1. Read the entire document carefully to make sure you understand the overall content and context of the text.

[0101] 2. Extract topics using a hierarchical approach, divided into three levels:

[0102] First level: 1 overall theme;

[0103] Level 2: 5 medium-level themes;

[0104] The third layer: 10 more subdivided topics.

[0105] 3. For the first layer (i.e. the overall theme):

[0106] Identify a topic that captures the core content of the entire document.

[0107] This topic should be the broadest and most comprehensive, covering the overall purpose and main content of the document.

[0108] 4. For the second level (i.e. medium general topics):

[0109] Identify 5 more specific topics that should be subdivisions of the first-level broad topics.

[0110] Each mid-level topic should cover a major aspect or section of the document.

[0111] Make sure the 5 topics do not overlap and together form a comprehensive overview of the document's content.

[0112] 5. For the third level (i.e. more subdivided topics):

[0113] Identify 10 more specific and detailed topics.

[0114] These topics should be further subdivisions of the second-level topics, or important specific points in the document.

[0115] Make sure these subtopics cover key details and specific examples in your document.

[0116] 6. When identifying topics, consider the following:

[0117] recurrence of key words and phrases;

[0118] Document structure, such as titles, subheadings, and paragraph topic sentences;

[0119] The point or argument emphasized by the author;

[0120] The overall purpose and intent of the document;

[0121] Document type and target audience.

[0122] 7. Ensure that there is a logical connection between topics at each level. Lower-level topics should be refinements or concretizations of higher-level topics.

[0123] 8. Provide a concise description or explanation of each topic, explaining its importance and relationship to the overall content. Provide the analysis results in the following format:

[0124] I. Overall theme (1):

[0125] [Provide the overall theme and a brief explanation].

[0126] II. Medium-level topics (5):

[0127] 1. [Medium topic 1 and its brief explanation];

[0128] 2. [Medium topic 2 and its brief explanation];

[0129] 3. [Medium topic 3 and its brief explanation];

[0130] 4. [Medium topic 4 and its brief explanation];

[0131] 5. [Medium topic 5 and its brief explanation].

[0132] III. More detailed topics (10):

[0133] 1. [Sub-topic 1 and brief explanation];

[0134] 2. [Sub-topic 2 and brief explanation]; ...

[0136] 10. [Sub-topic 10 and brief explanation].

[0137] Now, analyze the following document and extract its topics as per the above format:

[0138] [The document content to be analyzed].

[0139] 2. Extraction of emotional features:

[0140] Use the following prompt words to perform the task of extracting emotional features:

[0141] Task definition: As a professional text sentiment analysis expert, I am good at extracting and analyzing sentiment from various documents. The specific task is to read the given text carefully and analyze the sentiment in it in a hierarchical manner. To extract sentiment features, you need to follow the following steps:

[0142] 1. Read the entire document carefully to make sure you understand the overall content, context, and tone of the text.

[0143] 2. Use a hierarchical approach to analyze sentiment, divided into three levels:

[0144] First layer: 1 overall emotional tone;

[0145] Second level: 5 main emotional themes;

[0146] The third level: 10 specific emotional expressions.

[0147] 3. For the first layer (i.e. overall emotional tone):

[0148] Identify a tone that summarizes the main sentiment of the entire document.

[0149] This tone should reflect the overall emotional atmosphere of the document, which can be positive, negative, neutral, or a complex mixture.

[0150] 4. For the second layer (i.e. the main emotional theme):

[0151] Identify 5 more specific emotional themes that should be subdivisions of the first level of overall emotional tone.

[0152] Each sentiment topic should cover a major sentiment aspect or variation in the document.

[0153] Make sure these five sentiment themes can fully reflect the emotional changes and complexity of the document.

[0154] 5. For the third level (i.e. specific emotional expression):

[0155] Identify 10 more specific and detailed expressions of emotion.

[0156] These representations should be the embodiment of the second-level emotional themes, or important emotional details in the document.

[0157] Make sure these specific representations cover the key sentiment descriptions, language usage, and emotional turning points in the document.

[0158] 6. When analyzing emotions, consider the following:

[0159] the use and frequency of emotion words;

[0160] changes in tone and intonation;

[0161] The use of rhetoric (such as metaphor, hyperbole, etc.);

[0162] Choice of narrative point of view and person;

[0163] emotional changes during the development of the plot or exposition;

[0164] the author’s attitude and stance;

[0165] The influence of document type and target audience on emotional expression.

[0166] 7. Ensure that there is a logical connection between the various levels of sentiment analysis, and that lower-level sentiment expressions should support and refine higher-level sentiment themes.

[0167] 8. Provide a concise description or explanation for each sentiment analysis result, including relevant textual evidence or examples. The analysis results should be provided in the following format:

[0168] I. Overall emotional tone (1):

[0169] [Provide the overall emotional tone and a brief explanation of it, including an assessment of the overall emotional intensity].

[0170] II. Main emotional themes (5):

[0171] 1. [Emotional theme 1 and a brief explanation of it, including its manifestation in the text];

[0172] 2. [Emotional theme 2 and its brief explanation, including its manifestation in the text;

[0173] 3. [Emotional theme 3 and its brief explanation, including its manifestation in the text];

[0174] 4. [emotional theme 4 and its brief explanation, including its manifestation in the text];

[0175] 5. [Emotional theme 5 and a brief explanation, including its manifestation in the text].

[0176] III. Specific emotional expressions (10):

[0177] 1. [Specific manifestation 1 and brief explanation thereof, including relevant textual citations];

[0178] 2. [Specific manifestation 2 and brief explanation thereof, including relevant textual citations]; ...

[0180] 10. [Specific manifestation 10 and a brief explanation thereof, including relevant textual citations].

[0181] IV. Emotional change analysis:

[0182] [Briefly analyze the overall emotional trends in the document, including any significant emotional turning points or climaxes]

[0183] V. Conclusion:

[0184] [Summarize the overall sentiment of the document and discuss how these sentiments affect the reader or convey the author’s intent]

[0185] Now, analyze the following documents and extract their sentiment as per the above format:

[0186] [The document content to be analyzed].

[0187] 3. Extraction of content summary features:

[0188] Use the following prompt words to perform the task of extracting content summary features:

[0189] Task definition: As a professional text analysis and summarization expert, you are good at extracting key information from various documents and summarizing them concisely. The specific task is to read the given text carefully and summarize its content in a hierarchical manner, strictly adhering to the specified word limit. To extract content summary features, you need to follow the following steps:

[0190] 1. Read the entire document carefully to make sure you fully understand the content, structure, and main ideas of the text.

[0191] 2. Use a hierarchical approach to summarize content, divided into three levels:

[0192] First level: 200-word comprehensive summary;

[0193] Second level: 100 words of core points;

[0194] The third layer: 5 keywords.

[0195] 3. For the first level (i.e. 200-word comprehensive abstract):

[0196] Provide a comprehensive but concise summary of the document's contents.

[0197] Cover the document's main themes, key arguments, and important findings.

[0198] Be objective and accurately reflect the main content and tone of the original text.

[0199] Make sure your summary is coherent and flows logically.

[0200] Strictly controlled within 200 words.

[0201] 4. For the second level (i.e. the core points of 100 words):

[0202] Distill the document's core message and most important points.

[0203] Present these points in a clear and concise manner.

[0204] Make sure these 100 words stand alone in the 200-word summary and provide the essence of the document.

[0205] Strictly control within 100 words.

[0206] 5. For the third layer (i.e. 5 keywords):

[0207] Select 5 keywords or phrases that best represent the core content of the document.

[0208] These keywords should summarize the theme, focus, or uniqueness of the document.

[0209] Keywords can be single words or short phrases, but should not exceed 3 words each.

[0210] 6. When making generalizations, consider the following factors:

[0211] The overall structure and logic of the document;

[0212] The main points emphasized by the author;

[0213] Important data, facts or examples;

[0214] The purpose and target audience of the document;

[0215] Any innovative ideas or unique insights.

[0216] 7. Ensure that there is a logical connection between the summaries at each level, and the content at the lower level should support and be reflected in the summaries at the higher level.

[0217] 8. Strictly abide by the word limit for each level and do not exceed the prescribed number of words.

[0218] Among them, the analysis results are provided in the following format:

[0219] I.200-word comprehensive summary:

[0220] [Provide a 200-word summary here to ensure that it fully captures the content of the document]

[0221] II.100-word key points:

[0222] [Provide a 100-word core summary here, highlighting the most important information].

[0223] III.5 Keywords:

[0224] 1.[Keyword 1];

[0225] 2.[Keyword 2];

[0226] 3.[Keyword 3];

[0227] 4.[Keyword 4];

[0228] 5.[Keyword 5];

[0229] Now, analyze the following document and summarize its content according to the above format:

[0230] [The document content to be analyzed].

[0231] As can be seen from the above, in some embodiments of the present application, the feature encoder used to extract the N first features of the first text can be determined by the following process:

[0232] like Figure 2 As shown, each document and prompt word as a training sample are input into the large model together to obtain the semantic features of the large model output (i.e., N first features). Then, the semantic features and prompt words of each document are re-input into the large model to obtain a new output of the large model, i.e., similar semantics. Based on the above semantic features and similar semantic features, the degree of change of the output of the same specific transformer block is determined, and then the last transformer block whose output is basically unchanged is found as the dividing point to divide the large model, thereby obtaining the feature encoder.

[0233] Optionally, in the above step 103, the step of concatenating the N first features of the first text into a second feature includes:

[0234] Through the document encoder, the encoding lengths of the N first features of the first text are all adjusted to a predetermined length, and the N first features of the first text of the predetermined length are spliced ​​into a second feature according to a predetermined order of feature types.

[0235] Optionally, the predetermined length is the maximum length of N first features.

[0236] It can be seen that the document encoder unifies the encoding length of each first feature to a specific length, and fills the feature encoding with 0 if the length is insufficient. Taking Llama3 as an example, the encoding length of each feature is unified to 4096. Then the document encoder concatenates each feature in the order of category as output.

[0237] For example Figure 3 As shown, when the encoding of N first features includes topic feature encoding, sentiment feature encoding and content summary feature encoding, the document encoder unifies the length of each feature through a zero-padding operation and splices them into a complete feature, for example, called a document feature.

[0238] Optionally, in the above step 104, the obtaining the first hash code of the second feature includes: obtaining the first hash code of the second feature through a hash encoder.

[0239] In some embodiments of the present application, a hash function may be designed that takes a vector of size 1*L as input (L is the maximum length that the hash encoder can process) and outputs a positive integer hash value of a specified number of bits. The closer the hash values ​​of two vectors are, the more similar the two vectors are.

[0240] Among them, such a hash function can be obtained by a neural network, that is, select any two vectors and reshape them into a vector of size 1*L and record them as When the vector length exceeds the maximum length, it is truncated, and if it is less than the maximum length, it is padded with zeros. Then the loss function of the hash encoder is:

[0241]

[0242] in, represents the mth vector in a vector set V, where the vector set includes multiple vectors input to the hash encoder;

[0243] q represents the hash code length, express The cosine similarity of the two vectors, MSE represents the mean square error, and hash represents the output of the hash encoder.

[0244] Under the supervision of this loss function, the neural network can obtain the expected value of each vector hash code, so that the vectors in the hash code value range are and The total expected error is minimized.

[0245] Alternatively, select 2 q-1 The normalization factor is used to limit the value range of the generated hash code. This ensures that the hash value is within a controllable range, which is beneficial for subsequent tree structure construction and query operations.

[0246] Optionally, the process of constructing the document spreading tree includes the following steps A-1 to A-4:

[0247] Step A-1: ​​Obtain N first features of each of a plurality of documents in a database;

[0248] Step A-2: concatenate N first features of the same document into a third feature;

[0249] Step A-3: Obtain a fifth hash code of the third feature as a hash code of the corresponding document;

[0250] Step A-4: construct the document extension tree according to the hash codes of the documents.

[0251] It can be understood that the method of extracting N first features of each document in step A-1 is the same as the method of extracting N first features of the first text in step 102, and will not be repeated here.

[0252] Similarly, in step A-2, the method of splicing N first features of the same document into a third feature is the same as the method of splicing N first features of the first text into a second feature in step 103, and will not be repeated here.

[0253] Similarly, in step A-3, the method for obtaining the fifth hash code of the third feature is the same as the method for obtaining the first hash code of the second feature in step 104, and will not be repeated here.

[0254] It can be seen from the above steps A-1 to A-4 that in some embodiments of the present application, the hash code of each document in the database can be obtained, so as to construct a document extension tree based on the hash code of each document, so that document retrieval can be performed in the document extension tree.

[0255] It can be seen that the document splay tree is used to quickly retrieve the entire document, specifically when it is necessary to retrieve based on the overall semantics of the document. That is, the document splay tree uses the hash code of the second feature of N first features as the index key.

[0256] Among them, a balanced binary tree insertion method can be used to construct the document splay tree according to the hash code of each document. For example, the hash code of each document is used as the key of the node, and the document splay tree is constructed according to the following method:

[0257] (1) When the key of the node to be inserted is smaller than the key of the current node, the node to be inserted is inserted into the left subtree of the current node;

[0258] (2) When the key of the node to be inserted is equal to the key of the current node, the document of the node to be inserted is added to the document list of the current node;

[0259] (3) When the key of the node to be inserted is greater than the key of the current node, the document of the node to be inserted is added to the right subtree of the current node.

[0260] Optionally, the method further comprises the following steps D-1 to D-4:

[0261] Step D-1: Obtain a third hash code of the i-th first feature of the first text, where the value of i includes at least one integer from 1 to N;

[0262] Step D-2: in the current index extension tree of the i-th first feature, searching for a fourth hash code whose similarity to the third hash code is greater than or equal to a second threshold, wherein the index extension tree of the i-th first feature includes a plurality of nodes, each node is associated with at least one document, and the hash codes of the i-th feature of the documents associated with the same node are the same;

[0263] Step D-3: when the fourth hash code is retrieved, determining the document associated with the second target node as the document matching the i-th first feature of the current conversation, wherein the second target node is the node to which the fourth hash code belongs;

[0264] Step D-4: Adjust the second target node to be the root node of the index extension tree of the i-th first feature.

[0265] It can be seen that in some embodiments of the present application, a hash code of a first feature of each document in a plurality of documents can be used to construct an index extension tree for the first feature, so that document retrieval can be performed based on the first feature. In this way, more types of document retrieval are provided.

[0266] Among them, for steps D-1 to D-4, when searching in the above-mentioned index extension tree, traversal is often started from the root node. In the embodiment of the present application, after retrieving a fourth hash code whose similarity with the third hash code corresponding to the current conversation is greater than or equal to the second threshold, the second target node to which the fourth hash code belongs is adjusted to the root node of the index extension tree. In this way, the second target node can be traversed preferentially when searching in the index extension tree next time.

[0267] Moreover, in the same round of conversation, the content discussed in the next conversation of the current conversation is often related, that is, there is an implicit association between the contexts of several questions and answers in the same round of conversation. In an embodiment of the present application, after the above-mentioned second target node is adjusted to the root node of the index extension tree, when retrieving a hash code similar to the hash code of the i-th first feature of the next conversation in the index extension tree, the second target node can be traversed first, that is, the document matching the i-th first feature of the next conversation can be found first, thereby improving the retrieval speed.

[0268] In addition, taking the i-th first feature as the theme feature as an example, the index tree can be Figure 5 As shown, each Splay tree node is identified by the hash code (theme_hash) of the document's theme feature, and each node is associated with a document list (docs), and the hash codes of the theme features of the documents in the document list are all hash codes that identify the node. Figure 5 In the, left_theme_tree_node represents the left child node pointer of the theme index extension tree node, and right_theme_tree_node represents the right child node pointer of the theme feature extension tree node.

[0269] Optionally, in step D-1, obtaining the third hash code of the i-th first feature of the first text includes:

[0270] According to the order in which the N first features of the first text are spliced ​​into the second feature, the third hash code of the i-th first feature of the first text is segmented from the first hash code;

[0271] or,

[0272] The i-th first feature of the first text is input into a hash encoder to obtain the third hash code of the i-th first feature of the first text.

[0273] It can be seen that in some embodiments of the present application, after obtaining the first hash code of the second feature, the third hash code of the i-th first feature can be segmented from the first hash code in the order in which the N first features are spliced ​​into the second feature. Alternatively, the i-th first feature of the first text can be directly input into the hash encoder to output the third hash code.

[0274] Optionally, the process of constructing the index splay tree of the i-th first feature includes:

[0275] Obtaining the i-th first feature of each document in a plurality of documents in a database;

[0276] Obtain the hash code of the i-th first feature of each document;

[0277] According to the hash code of the i-th first feature of each document, an index extension tree of the i-th first feature is constructed.

[0278] It can be understood that the method for obtaining the i-th first feature of each document here is the same as the method for extracting N first features of the first text in the above step 101, and will not be repeated here; the method for obtaining the hash code of the i-th first feature is the same as the method for obtaining the third hash code of the i-th first feature of the first text in the above step D-1, and will not be repeated here.

[0279] In addition, the specific method of constructing the index splay tree of the i-th first feature according to the hash code of the i-th first feature of each document is the same as the method of constructing the document splay tree according to the hash code of each document in the above step A-4, and will not be repeated here.

[0280] Optionally, the adjusting the first target node to be the root node of the document extension tree includes:

[0281] Through the stretching operation, the first target node is adjusted to be the root node of the document stretching tree.

[0282] The stretching operation means that after each visit to a node, the node is rotated and moved from the current position to the root node position. By continuously rotating and moving, the node is adaptively adjusted so that the frequently visited nodes are closer to the root node.

[0283] Optionally, the stretching operation includes at least one of a left-hand rotation operation and a right-hand rotation operation.

[0284] The left-hand operation and the right-hand operation are introduced as follows:

[0285] 1. Right rotation (zig) operation, such as Figure 6As shown in the figure, when p is the root node, x is the left child node of p, and the depth of the left subtree of p is greater than the depth of the right subtree, the zig operation is performed on the x node, thereby promoting the x node to the root node position. That is, the zig operation is to change the right child node of the current node (i.e., the x node) to the left child node of the father node, and at the same time, change the father node to the right child node of the current node.

[0286] 2. Left rotation (zag) operation, such as Figure 7 As shown in the figure, when p is the root node, x is the right child node of p, and the right subtree depth of p is greater than the left subtree depth, the zag operation is performed on the x node, thereby promoting the x node to the root node position. That is, the zag operation is to change the left child node of the current node (i.e., the x node) to the right child node of the father node, and at the same time, change the father node to the left child node of the current node.

[0287] 3. Zig-zag: When y is the root node of x, x is the left child node of y, the depth of y's left subtree is greater than the depth of its right subtree, z is the root node of y, and the depth of z's right subtree is greater than the depth of its left subtree (i.e. Figure 8 As shown in the figure, first perform a zig operation on x, then perform a zag operation on x, and the x node can be promoted to the root node position.

[0288] 4. Zag-zig operation: When y is the root node of x, x is the right child node of y, the right subtree depth of y is greater than the left subtree depth, z is the root node of y, and the left subtree depth of z is greater than the right subtree depth, first perform a zag operation on x, and then perform a zig operation, then the x node can be promoted to the root node position.

[0289] 5. Zig-zig operation: When y is the root node of x, x is the left child node of y, the depth of the left subtree of y is greater than the depth of the right subtree, z is the root node of y, and the depth of the left subtree of z is greater than the depth of the right subtree, first perform the zig operation on x, and then perform the zig operation.

[0290] 6. Zag-zag operation: When y is the root node of x, x is the right child node of y, the right subtree depth of y is greater than the left subtree depth, z is the root node of y, and the right subtree depth of z is greater than the left subtree depth, first perform the zag operation on x, and then perform the zag operation.

[0291] The advantages of the technical solution provided in the embodiment of the present application are explained below by comparison with related technologies.

[0292] The retrieval part of the current traditional RAG technology is usually optimized based on the vector database, and the current similarity retrieval algorithm of the vector database only provides optimization for a single similar document search, and does not take into account that in the same round of dialogue, each question and answer is in the same context, and the associated documents may have certain feature associations. The embodiment of the present application proposes a solution based on splay tree and deep hashing based on this content, which can effectively improve the speed of RAG technology to retrieve knowledge documents related to the question.

[0293] Specifically, in the embodiments of the present application, a special hash encoder can be constructed, and the hash of the hash encoder can ensure that the closer the hash values ​​of two vectors are, the more similar the two vectors are; and the features of the document can be extracted, and the hash code of the features can be obtained through the above hash encoder, so as to construct a spreading tree with the hash code as the key. Among them, similar document retrieval can be performed based on the spreading tree, which reduces the algorithm complexity of each similar document matching to O (log n). And considering that the spreading tree has a memory property: the most recently retrieved document is closer to the root of the tree, and in the same round of dialogue, each question and answer is in the same context, and the knowledge documents associated with each question are likely to be similar, so in continuous dialogue, the embodiments of the present application will be more efficient than ordinary similarity search algorithms.

[0294] The document processing method provided by the embodiment of the present application is introduced above. The document processing device provided by the embodiment of the present application will be introduced below with reference to the accompanying drawings.

[0295] See also Fig. 9 The present application also provides a document processing device, the device comprising:

[0296] A first acquisition module 901, used to acquire a first text of a current conversation;

[0297] A first processing module 902 is used to extract N first features of the first text, where N is an integer greater than or equal to 1;

[0298] A second processing module 903, configured to combine the N first features of the first text into a second feature;

[0299] A second acquisition module 904 is used to acquire a first hash code of the second feature;

[0300] A first retrieval module 905 is configured to retrieve, in a current document splay tree, a second hash code whose similarity to the first hash code is greater than or equal to a first threshold, wherein the document splay tree includes a plurality of nodes, each node is associated with at least one document, and the hash codes of documents associated with the same node are the same;

[0301] A first determination module 906 is configured to determine, when the second hash code is retrieved, a document associated with a first target node as a document matching the current conversation, wherein the first target node is a node to which the second hash code belongs;

[0302] The first adjustment module 907 is used to adjust the first target node to the root node of the document extension tree.

[0303] Optionally, the device further comprises a document spreading tree construction module, which is used to:

[0304] Obtaining N first features of each of a plurality of documents in a database;

[0305] Concatenate N first features of the same document into a third feature;

[0306] Obtaining a fifth hash code of the third feature as a hash code of the corresponding document;

[0307] The document splay tree is constructed according to the hash codes of the documents.

[0308] Optionally, the device further comprises:

[0309] A third acquisition module, used to obtain a third hash code of the i-th first feature of the first text, where the value of i includes at least one integer from 1 to N;

[0310] a second retrieval module, configured to retrieve, in the current index extension tree of the i-th first feature, a fourth hash code whose similarity to the third hash code is greater than or equal to a second threshold, wherein the index extension tree of the i-th first feature includes a plurality of nodes, each node is associated with at least one document, and the hash codes of the i-th feature of the documents associated with the same node are the same;

[0311] A second determination module is configured to, when the fourth hash code is retrieved, determine a document associated with a second target node as a document matching the i-th first feature of the current conversation, wherein the second target node is a node to which the fourth hash code belongs;

[0312] The second adjustment module is used to adjust the second target node to be the root node of the index extension tree of the i-th first feature.

[0313] Optionally, the third acquisition module is specifically used to:

[0314] According to the order in which the N first features of the first text are spliced ​​into the second feature, the third hash code of the i-th first feature of the first text is segmented from the first hash code;

[0315] or,

[0316] The i-th first feature of the first text is input into a hash encoder to obtain the third hash code of the i-th first feature of the first text.

[0317] Optionally, the device further comprises an index splay tree construction module, configured to:

[0318] Obtaining the i-th first feature of each document in a plurality of documents in a database;

[0319] Obtain the hash code of the i-th first feature of each document;

[0320] According to the hash code of the i-th first feature of each document, an index extension tree of the i-th first feature is constructed.

[0321] Optionally, the first processing module 902 is specifically configured to:

[0322] Extracting N first features of the first text through a feature encoder;

[0323] The device also includes a feature encoder determination module, which is used to:

[0324] When j is an integer from 1 to K, the following process is performed to obtain the similarity corresponding to each specific transformer block corresponding to the 1st to Kth sample documents:

[0325] Inputting the j-th sample document into the language model, obtaining the first outputs of the M specific transformer blocks of the language model, and the second output of the language model;

[0326] Inputting the second output into the language model to obtain a third output of the M specific transformer blocks;

[0327] Determine the degree of change from the first output to the third output of the same specific transformer block, and obtain the degree of change corresponding to each specific transformer block corresponding to the j-th sample document;

[0328] When v is an integer from 1 to M, perform the following process to obtain the first parameter corresponding to each specific transformer block:

[0329] Calculate the average value of the degree of change corresponding to the v-th specific transformer block corresponding to the 1st to K-th sample documents, and use it as the first parameter corresponding to the v-th specific transformer block;

[0330] Among the M specific transformer blocks, adjacent transformer blocks having a first parameter less than or equal to a third threshold are selected as components of the feature encoder.

[0331] Optionally, the second processing module 903 is specifically configured to:

[0332] Through the document encoder, the encoding lengths of the N first features of the first text are all adjusted to a predetermined length, and the N first features of the first text of the predetermined length are spliced ​​into a second feature according to a predetermined order of feature types.

[0333] Optionally, the first adjustment module 907 is specifically configured to:

[0334] Through the stretching operation, the first target node is adjusted to be the root node of the document stretching tree.

[0335] Optionally, the N first features include: topic features, emotion features, and content summary features.

[0336] It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0337] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0338] It should be noted here that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0339] The embodiment of the present application also provides an electronic device, such as Fig.10 As shown, the electronic device includes a memory 1020, a transceiver 1010, and a processor 1000;

[0340] Memory 1020, used for storing computer programs;

[0341] The transceiver 1010 is used to receive and send data under the control of the processor 1000;

[0342] The processor 1000 is used to read the computer program in the memory 1020 and execute the document processing method described in the first aspect.

[0343] Among them, Fig.10 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 1000 and memory represented by memory 1020. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 1010 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium, which transmission medium may include a wireless channel, a wired channel, an optical cable, and the like. The processor 1000 is responsible for managing the bus architecture and general processing, and the memory 1020 may store data used by the processor 1000 when performing operations.

[0344] The processor 1000 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD). The processor 1000 may also adopt a multi-core architecture.

[0345] It should be noted here that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0346] An embodiment of the present application further provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the document processing method described in the first aspect above is implemented.

[0347] The computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (Magnet-Optica, MO)), etc.), optical storage (such as Compact Disk (CD), Digital Versatile Disc (DVD), Blu-ray Disc (BD), Holographic Versatile Disc (HVD), etc.), and semiconductor memory (such as Read-Only Memory (ROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), non-volatile memory (NAND FLASH), solid-state drive (Solid State Disk or Solid State Drive, SSD)), etc.

[0348] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0349] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer executable instructions. These computer executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0350] These processor executable instructions may also be stored in a processor readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the processor readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0351] These processor-executable instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0352] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A document processing method, characterized in that: The method comprises: Get the first text of the current conversation; Extracting N first features of the first text, where N is an integer greater than or equal to 1; splicing N first features of the first text into a second feature; Obtain a first hash code of the second feature; Retrieving a second hash code whose similarity to the first hash code is greater than or equal to a first threshold in the current document spreading tree, wherein the document spreading tree includes a plurality of nodes, each node is associated with at least one document, and the hash codes of the documents associated with the same node are the same; In case the second hash code is retrieved, determining a document associated with a first target node as a document matching the current conversation, wherein the first target node is the node to which the second hash code belongs; The first target node is adjusted to be the root node of the document spanning tree.

2. The method according to claim 1, characterized in that: The construction process of the document extension tree includes: Obtaining N first features of each of a plurality of documents in a database; Concatenate N first features of the same document into a third feature; Obtaining a fifth hash code of the third feature as a hash code of the corresponding document; The document splay tree is constructed according to the hash codes of the documents.

3. The method according to claim 1, characterized in that The method further comprises: Obtaining a third hash code of the i-th first feature of the first text, where the value of i includes at least one integer from 1 to N; In the current index extension tree of the i-th first feature, a fourth hash code whose similarity to the third hash code is greater than or equal to a second threshold is retrieved, wherein the index extension tree of the i-th first feature includes a plurality of nodes, each node is associated with at least one document, and the hash codes of the i-th feature of the documents associated with the same node are the same; In the case where the fourth hash code is retrieved, determining the document associated with the second target node as the document matching the i-th first feature of the current conversation, wherein the second target node is the node to which the fourth hash code belongs; The second target node is adjusted to be the root node of the index extension tree of the i-th first feature.

4. The method according to claim 3, characterized in that The obtaining of the third hash code of the i-th first feature of the first text includes: According to the order in which the N first features of the first text are spliced ​​into the second feature, the third hash code of the i-th first feature of the first text is segmented from the first hash code; or, The i-th first feature of the first text is input into a hash encoder to obtain the third hash code of the i-th first feature of the first text.

5. The method according to claim 3 or 4, characterized in that: The construction process of the index splay tree of the i-th first feature includes: Obtaining the i-th first feature of each document in a plurality of documents in a database; Obtain the hash code of the i-th first feature of each document; According to the hash code of the i-th first feature of each document, an index extension tree of the i-th first feature is constructed.

6. The method according to any one of claims 1 to 4, characterized in that: The extracting N first features of the first text includes: Extracting N first features of the first text through a feature encoder; The process of determining the feature encoder includes: When j is an integer from 1 to K, the following process is performed to obtain the similarity corresponding to each specific transformer block corresponding to the 1st to Kth sample documents: Inputting the j-th sample document into a language model, obtaining first outputs of M specific transformer blocks of the language model, and a second output of the language model, wherein transformer block represents a codec block; Inputting the second output into the language model to obtain a third output of the M specific transformer blocks; Determine the degree of change from the first output to the third output of the same specific transformer block, and obtain the degree of change corresponding to each specific transformer block corresponding to the j-th sample document; When v is an integer from 1 to M, perform the following process to obtain the first parameter corresponding to each specific transformer block: Calculate the average value of the degree of change corresponding to the v-th specific transformer block corresponding to the 1st to K-th sample documents, and use it as the first parameter corresponding to the v-th specific transformer block; Among the M specific transformer blocks, adjacent transformer blocks having a first parameter less than or equal to a third threshold are selected as components of the feature encoder.

7. The method according to any one of claims 1 to 4, characterized in that: The step of combining the N first features of the first text into a second feature includes: Through the document encoder, the encoding lengths of the N first features of the first text are all adjusted to a predetermined length, and the N first features of the first text of the predetermined length are spliced ​​into a second feature according to a predetermined order of feature types.

8. A document processing device, characterized in that: The device comprises: A first acquisition module, used to acquire the first text of the current conversation; A first processing module, used for extracting N first features of the first text, where N is an integer greater than or equal to 1; A second processing module, configured to combine the N first features of the first text into a second feature; A second acquisition module, used to acquire a first hash code of the second feature; A first retrieval module is used to retrieve a second hash code whose similarity with the first hash code is greater than or equal to a first threshold in a current document spreading tree, wherein the document spreading tree includes a plurality of nodes, each node is associated with at least one document, and the hash codes of the documents associated with the same node are the same; A first determination module, configured to determine, when the second hash code is retrieved, a document associated with a first target node as a document matching the current conversation, wherein the first target node is a node to which the second hash code belongs; The first adjustment module is used to adjust the first target node to be the root node of the document extension tree.

9. An electronic device, characterized in that: Including memory, transceiver, processor: Memory for storing computer programs; A transceiver, used for sending and receiving data under the control of the processor; a processor, used for reading the computer program in the memory and executing the document processing method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the document processing method according to any one of claims 1 to 7 is implemented.

Images