Document query method and device, equipment, storage medium and computer program product

Abstract

The application discloses a document query method and device, equipment, storage medium and computer program product, relates to the technical field of natural language processing, and the method comprises the following steps: generating a target user query based on a current user query input by a user and a historical question and answer pair sequence of the user; determining a target compressed text paragraph in a preset document semantic tree according to a target paragraph index matched by the target user query, the preset document semantic tree storing a mapping relationship between compressed text paragraphs in a target document and paragraph indexes; and generating a document query result corresponding to the current user query based on the target compressed text paragraph. According to the application, the compressed text paragraph can be matched in the document semantic tree according to the user query, and the document query result is generated based on the compressed text paragraph, so that the technical problem that the accuracy of the query result is not high due to the fact that the retrieved document paragraph exceeds the context processing capability of the model in the existing document query method is solved.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to document retrieval methods, apparatus, devices, storage media, and computer program products. Background Technology

[0002] With the rapid development of information technology and the deepening of digital transformation, the storage, management, and efficient retrieval of massive amounts of document data have become core challenges in the field of information processing. Especially in complex application scenarios such as engineering code understanding, long document question answering, and multi-document analysis, users often need to go through multiple rounds of interaction to obtain accurate answers, thus placing higher demands on the semantic understanding capabilities, context preservation capabilities, and retrieval efficiency of document query systems.

[0003] Currently, traditional document query methods mainly rely on keyword matching or vector similarity calculation to retrieve document fragments similar to the query and return results. However, in complex scenarios such as multi-turn interactions, especially when dealing with the understanding of multiple or long documents, the set of relevant document segments retrieved by the document query system may exceed the model's context processing capabilities. This can lead to the loss of key semantic information or the disruption of contextual coherence, thereby affecting the accuracy and completeness of the query results. Summary of the Invention

[0004] The main objective of this application is to provide a document query method, apparatus, device, storage medium, and computer program product, which aims to solve the technical problem that existing document query methods may result in low accuracy of query results because the retrieved document paragraphs exceed the context processing capabilities of the model.

[0005] To achieve the above objectives, this application proposes a document retrieval method, the method comprising: A target user query is generated based on the current user query input and the user's historical question-and-answer pair sequence. The target compressed text paragraph is determined in the preset document semantic tree based on the target paragraph index matched by the target user query. The preset document semantic tree stores the mapping relationship between the compressed text paragraphs in the target document and the paragraph index. The target document is the document that matches the query scenario to which the current user query belongs. Based on the target compressed text paragraph, the document query results corresponding to the current user's query are generated.

[0006] In one embodiment, before the step of generating a target user query based on the current user query input and the user's historical question-and-answer pair sequence, the method further includes: Based on the query scenario of the current user query, obtain the user query set and the documents to be processed; Clustering is performed on each user query in the user query set to obtain query cluster centers, and a query semantic tree is constructed based on the query cluster centers. The query semantic tree is used to represent the semantic relationship between each user query. The document to be processed is compressed based on the query semantic tree to obtain a compressed document; The compressed documents are hierarchically clustered to construct a preset document semantic tree.

[0007] In one embodiment, the step of compressing the document to be processed based on the query semantic tree to obtain a compressed document includes: A preset vector compression objective function is constructed based on the query vector in the query semantic tree and the text paragraph vector in the document to be processed. The document to be processed is compressed based on the preset vector compression objective function to obtain the compressed document.

[0008] In one embodiment, the step of constructing a preset vector compression objective function based on the query vector in the query semantic tree and the text paragraph vectors in the document to be processed includes: The second-order momentum matrix of the query latent space is determined based on the query vector in the query semantic tree; A preset vector compression objective function is constructed based on the second-order momentum matrix and the text paragraph vectors in the document to be processed.

[0009] In one embodiment, the step of performing hierarchical clustering on the compressed document to construct a preset document semantic tree includes: The compressed document is segmented into paragraphs to obtain several text paragraphs; Hierarchical clustering is performed on each of the text paragraphs to determine the paragraph index corresponding to each text paragraph; A preset document semantic tree is constructed based on each of the text paragraphs and the paragraph index.

[0010] In one embodiment, the paragraph index includes a first-level paragraph index and a second-level paragraph index; the step of performing hierarchical clustering on each of the text paragraphs to determine the paragraph index corresponding to each text paragraph includes: Perform a first-level clustering on each of the text paragraphs to obtain a first-level paragraph index, which is used to characterize at least one text paragraph in the compressed document. Cross-document text paragraphs are identified from each of the text paragraphs, and a second-level clustering is performed on the cross-document text paragraphs to obtain a second-level paragraph index, which is used to characterize text paragraphs in at least two different documents; The step of constructing a preset document semantic tree based on each of the text paragraphs and the paragraph index includes: A hierarchical document semantic tree corresponding to the compressed document is constructed based on each of the text paragraphs, the first-level paragraph index, and the second-level paragraph index.

[0011] In one embodiment, the step of performing a first-level clustering on each of the text paragraphs to obtain a first-level paragraph index includes: The first-level clustering of each text segment is performed using a residual clustering algorithm to determine the summary embedding vector corresponding to each text segment. The range of text paragraph identifiers is determined based on the text paragraph identifiers corresponding to each of the aforementioned text paragraphs; The first-level paragraph index is constructed based on the summary embedding vector, the tree height identifier, and the text paragraph identifier range.

[0012] In one embodiment, after the step of performing hierarchical clustering on the compressed document to construct a preset document semantic tree, the method further includes: Using the query cluster center as the key, construct an inverted index linked list pointing to the second-level paragraph index in the preset document semantic tree; The step of determining the target compressed text paragraph in the preset document semantic tree based on the target paragraph index matched by the target user query includes: The target paragraph index matching the target user query is determined based on the inverted index linked list, and the target compressed text paragraph is determined in the preset document semantic tree based on the target paragraph index.

[0013] In one embodiment, the step of determining the target compressed text paragraph in a preset document semantic tree based on the target paragraph index matched by the target user query includes: The target matching document is determined in the preset document semantic tree based on the target paragraph index matched by the target user query. Based on the target matching document, the system queries the preset document semantic tree to find the target compressed text paragraph that matches the current user's query.

[0014] In one embodiment, the step of generating a target user query based on the current user query input and the user's historical question-and-answer pair sequence includes: A pre-filled context is constructed based on the current user query input and the user's historical question-and-answer pair sequence; The document query results corresponding to the current user's query are generated based on the pre-populated context.

[0015] Furthermore, to achieve the above objectives, this application also proposes a document retrieval device, the device comprising: The query generation module is used to generate a target user query based on the current user query input by the user and the user's historical question-and-answer pair sequence; The document query module is used to determine the target compressed text paragraph in a preset document semantic tree based on the target paragraph index matched by the target user's query. The preset document semantic tree stores the mapping relationship between the compressed text paragraphs in the document and the paragraph index. The result generation module is used to generate the document query result corresponding to the current user query based on the target compressed text paragraph.

[0016] In one embodiment, the document query module is further configured to obtain a user query set and documents to be processed based on the query scenario to which the current user query belongs, as input by the user. The document query module is further configured to cluster each user query in the user query set to obtain query cluster centers, and construct a query semantic tree based on the query cluster centers. The query semantic tree is used to represent the semantic relationship between each user query. The document query module is also used to compress the document to be processed based on the query semantic tree to obtain the compressed document; The document query module is also used to perform hierarchical clustering on the compressed documents to construct a preset document semantic tree.

[0017] In one embodiment, the document query module is further configured to construct a preset vector compression target function based on the query vector in the query semantic tree and the text paragraph vector in the document to be processed; The document query module is also used to compress the document to be processed based on the preset vector compression objective function to obtain the compressed document.

[0018] In one embodiment, the document query module is further configured to determine the second-order momentum matrix of the query latent space based on the query vector in the query semantic tree; The document query module is also used to construct a preset vector compression objective function based on the second-order momentum matrix and the text paragraph vectors in the document to be processed.

[0019] In one embodiment, the document query module is further configured to perform paragraph segmentation on the compressed document to obtain several text paragraphs; The document query module is also used to perform hierarchical clustering on each of the text paragraphs to determine the paragraph index corresponding to each of the text paragraphs; The document query module is also used to construct a preset document semantic tree based on each of the text paragraphs and the paragraph index.

[0020] In one embodiment, the paragraph index includes a first-level paragraph index and a second-level paragraph index; the document query module is further configured to perform a first-level clustering on each of the text paragraphs to obtain a first-level paragraph index, wherein the first-level paragraph index is used to characterize at least one text paragraph in the compressed document; The document query module is further configured to determine cross-document text paragraphs from each of the text paragraphs, and perform a second-level clustering on the cross-document text paragraphs to obtain a second-level paragraph index, wherein the second-level paragraph index is used to characterize text paragraphs in at least two different documents; The document query module is also used to construct a hierarchical document semantic tree corresponding to the compressed document based on each of the text paragraphs, the first-level paragraph index, and the second-level paragraph index.

[0021] In one embodiment, the document query module is further configured to use the query cluster center as a key to construct an inverted index linked list pointing to the second-level paragraph index in the preset document semantic tree; The document query module is further configured to determine the target paragraph index matching the target user query based on the inverted index list, and determine the target compressed text paragraph in the preset document semantic tree based on the target paragraph index.

[0022] In addition, to achieve the above objectives, this application also proposes a document query device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the document query method as described above.

[0023] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the document query method described above.

[0024] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the document query method described above.

[0025] This application provides a document query method. It discloses a method for generating a target user query based on the current user query input and the user's historical question-and-answer sequence. The method involves determining a target compressed text segment in a preset document semantic tree based on the target segment index matched by the target user query. The preset document semantic tree stores the mapping relationship between compressed text segments and segment indices in the target document. The target document is a document that matches the query scenario of the current user query. The method generates document query results corresponding to the current user query based on the target compressed text segment. Compared to existing document query systems where the retrieved document segment set may exceed the model's context processing capabilities, easily leading to the loss of key semantic information or disruption of contextual coherence, this application solves the technical problem that existing document query methods may result in low accuracy of query results due to the retrieved document segments exceeding the model's context processing capabilities. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating the document retrieval method of this application in Embodiment 1; Figure 2 This is a flowchart illustrating Embodiment 2 of the document retrieval method for this application. Figure 3 This is a flowchart illustrating the document retrieval method of this application in Embodiment 3; Figure 4 This is a schematic diagram of the document semantic tree structure in the document query method of this application; Figure 5 This is a schematic diagram of the module structure of the document query device according to an embodiment of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the document query method in this application embodiment.

[0029] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0030] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0031] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0032] The main solution of this application embodiment is as follows: a target user query is generated based on the current user query input by the user and the user's historical question-and-answer pair sequence; a target compressed text segment is determined in a preset document semantic tree according to the target segment index matched by the target user query. The preset document semantic tree stores the mapping relationship between the compressed text segments in the target document and the segment index. The target document is a document that matches the query scenario to which the current user query belongs; and a document query result corresponding to the current user query is generated based on the target compressed text segment.

[0033] Existing document query methods primarily rely on keyword matching or vector similarity calculations to retrieve results by comparing document fragments with the query. However, in complex scenarios such as multi-turn interactions, especially when dealing with multiple or long documents, the set of relevant document segments retrieved by the document query system may exceed the model's context processing capabilities. This can lead to the loss of key semantic information or disruption of contextual coherence, ultimately affecting the accuracy and completeness of the final query results.

[0034] This application provides a solution that can match compressed text paragraphs in the document semantic tree based on the paragraph index matched by the user query, and generate document query results based on the compressed text paragraphs. This solves the technical problem that existing document query methods may result in low accuracy of query results because the retrieved document paragraphs exceed the context processing capabilities of the model.

[0035] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device, document query device, or document query system containing the above functions. The following description uses a document query system (hereinafter referred to as the system) as an example to illustrate this embodiment and the following embodiments.

[0036] Based on this, this application provides a document query method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the document retrieval method of this application, provided in Embodiment 1.

[0037] In this embodiment, the document query method includes steps S10 to S30: Step S10: Generate a target user query based on the current user query input by the user and the user's historical question-and-answer pair sequence.

[0038] It should be understood that the above-mentioned current user query can be the original query text entered by the user in the current round. For example, in a multi-round dialogue, if the user enters "What is the specific implementation logic of this function" in the nth round, the content of this text is the current user query.

[0039] It should also be understood that the aforementioned historical question-and-answer pair sequence can be a record of question-and-answer interactions completed between the user and the system before the current round. In this embodiment, the historical question-and-answer pair sequence can be represented as follows: ,in, Indicates the first User queries in the round Indicates the first The response returned by the round system, each element This forms a question-and-answer pair.

[0040] Understandably, the aforementioned target user query can be a query generated based on the fusion of the current user query and historical question-and-answer pairs, used to fully express the user's current information needs. In practical applications, historical question-and-answer pairs can provide dialogue context, enabling the system to understand the implicit intent of the current user query, thereby eliminating ambiguity in the current user query and recording the dialogue state to avoid duplicate retrieval and information loss. In this embodiment, the system can perform semantic fusion and intent completion on the current user query based on the contextual information in the target user query to obtain the target user query. At this time, the target user query can eliminate the ambiguity and omitted components of the current user query input by the user, thereby improving the accuracy of subsequent document queries.

[0041] Furthermore, the step of generating a target user query based on the current user query input by the user and the user's historical question-and-answer pair sequence includes: constructing a pre-filled context based on the current user query input by the user and the user's historical question-and-answer pair sequence; and generating document query results corresponding to the current user query based on the pre-filled context.

[0042] Understandably, the aforementioned pre-filled context can be the pre-input text used to guide the language model in generating an answer to a user query. It can be used to allow the model to fully understand the historical context of the dialogue and the complete intent of the current query before generating an answer, so as to ensure that the generated answer is coherent and context-aware.

[0043] In practical applications, the system can leverage a large language model to perform semantic summarization on historical question-answer pair sequences, generating a new historical question-answer pair sequence. This sequence is then concatenated with the current user query to form a pre-filled context. This pre-filled context serves as input to the large language model during the pre-filling stage, enabling the model to clarify intent and semantically complete the current user query based on a comprehensive understanding of the dialogue context. Finally, the large language model can generate the target user query based on the pre-filled context. The pre-filling stage can be considered the input preparation stage in the large model's inference process. During this stage, the large model can process the input prompt in parallel and generate the first output token.

[0044] Step S20: Determine the target compressed text paragraph in the preset document semantic tree according to the target paragraph index matched by the target user query. The preset document semantic tree stores the mapping relationship between the compressed text paragraphs and paragraph indexes in the target document. The target document is a document that matches the query scenario to which the current user query belongs.

[0045] It should be noted that the above-mentioned target paragraph index can be the identification information of the paragraph node in the preset document semantic tree that has the highest semantic matching degree with the target user's query, which may include paragraph ID, node ID and corresponding semantic vector representation, etc. This embodiment does not limit this.

[0046] It should be noted that the aforementioned preset document semantic tree can be a pre-constructed tree data structure obtained by performing multi-level clustering on compressed text paragraphs. In this embodiment, the compressed text paragraphs stored in the preset document semantic tree can retain the semantic information most relevant to potential queries.

[0047] It should also be noted that the preset document semantic tree in this embodiment can be a three-layer structure, including: leaf nodes, first-layer index nodes, and second-layer index nodes. The leaf nodes can correspond to paragraphs in the original document; the first-layer index nodes can correspond to paragraph clustering centers within a single document; and the second-layer index nodes can correspond to topic clustering centers across documents.

[0048] In practical applications, the system first determines the query scenario of the current user's query, obtains all user queries and target documents matching that scenario, and guides the compression of the target document based on these user queries to obtain the compressed document. Then, the system can divide the compressed document into several paragraphs and perform hierarchical clustering on these paragraphs from bottom to top to construct a predefined document semantic tree. Specifically, it can first perform a first-level clustering of paragraphs within a single document, grouping semantically similar paragraphs together to obtain first-level index nodes. Then, it can continue to perform a second-level clustering of paragraphs across different documents to obtain second-level index nodes. Finally, it constructs the predefined document semantic tree based on the paragraphs within the document, the first-level index nodes, and the second-level index nodes.

[0049] It should be understood that the aforementioned target compressed text paragraph can be a text paragraph in a preset document semantic tree that matches the target user's query. This text paragraph has been compressed using a query-aware compression method when constructing the document semantic tree, but retains the semantic information most relevant to the potential query, thereby ensuring the accuracy of document queries while reducing storage space.

[0050] It is understood that the target document mentioned above can be a document that matches the query scenario to which the current user query belongs. The query scenario can be the application field or task type to which the current user query belongs, such as engineering code understanding scenario, long document analysis scenario, multi-document analysis scenario, etc. This embodiment does not limit this.

[0051] In this embodiment, the system can determine the query scenario of the current user's query through keyword matching, classification model prediction (such as BERT-based classifier), etc. After determining the query scenario, it matches the corresponding document from the database containing multiple documents according to the scenario identifier of the query scenario, and then determines the matched document as the target document.

[0052] Step S30: Generate the document query result corresponding to the current user query based on the target compressed text paragraph.

[0053] It should be understood that the above document query results can be the final output information used to respond to the current user's query, which may include, but is not limited to, response content, source identifier, context summary, etc. The response content can be text information used to directly answer the user's query; the source identifier can be the document source, paragraph position, and confidence level of the cited content; the context summary can be a general description of the background information related to the user's query.

[0054] In practical applications, after obtaining the target compressed text segment, the system can sort and filter the segment according to its relevance to the query, remove redundant and low-relevance content, and extract historical dialogue summaries. Current user query The system integrates the filtered target compressed text paragraphs to form a complete input context. Then, the system inputs this complete context into a large language model. After processing in the pre-filling stage, the model can generate the final answer. Finally, the system can combine the question-and-answer pairs from the current interaction round. Added to the historical question-and-answer pair sequence and updated to This is used for dialogue in subsequent rounds.

[0055] This embodiment provides a document query method that discloses generating a target user query based on the current user query input and the user's historical question-and-answer pair sequence; determining a target compressed text segment in a preset document semantic tree according to the target segment index matched by the target user query; the preset document semantic tree stores the mapping relationship between compressed text segments and segment indices in the target document; the target document is a document that matches the query scenario to which the current user query belongs; and generating document query results corresponding to the current user query based on the target compressed text segment. Compared with existing document query systems where the set of relevant document segments retrieved may exceed the model's context processing capabilities, easily leading to the loss of key semantic information or disruption of contextual coherence, this embodiment can match compressed text segments in the document semantic tree based on the segment index matched by the user query and generate document query results based on the compressed text segments. This solves the technical problem that existing document query methods may result in low accuracy of query results due to the retrieved document segments exceeding the model's context processing capabilities.

[0056] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the document retrieval method for this application.

[0057] In this embodiment, before step S10, the method further includes steps S01 to S04: Step S01: Obtain the user query set and the documents to be processed based on the query scenario of the current user query input.

[0058] It should be understood that the aforementioned user query set can be a collection of historical or expected queries associated with the query scenario to which the current user query belongs. In this embodiment, the user query set can be derived from user query logs in the query scenario to which the current user query belongs, a typical Frequently Asked Questions (FAQ) library, or query samples generated by simulating user behavior. This embodiment does not impose any limitations on this.

[0059] It should also be understood that the aforementioned documents to be processed can be a collection of original documents that match the query scenario of the current user's query and require compression and indexing, such as technical manuals, product documents, contract texts, code libraries, etc., and this embodiment does not impose any limitations on this. In this embodiment, the selection of documents to be processed can be based on tag matching of the query scenario to ensure that the document content is consistent with the user's potential information needs in terms of domain.

[0060] Step S02: Cluster each user query in the user query set to obtain query cluster centers, and construct a query semantic tree based on the query cluster centers. The query semantic tree is used to represent the semantic relationship between each user query.

[0061] It should be noted that the aforementioned query cluster center can be the centroid vector of each cluster obtained after clustering the user query set. In this embodiment, the query cluster center can be obtained by averaging or weighted averaging the vector representations of all user queries within a cluster, which can characterize the common features of a class of semantically similar queries.

[0062] It should also be noted that the aforementioned query semantic tree can be a hierarchical tree data structure used to represent the semantic relationships between user queries. In this embodiment, the query semantic tree can be constructed using a hierarchical clustering algorithm, with query cluster centers as internal nodes and original user queries as leaf nodes. In practical applications, the query semantic tree can be used to represent semantic inclusion, similarity, or progression relationships between user queries, so as to provide semantic guidance information for the query space in subsequent document compression.

[0063] Step S03: Compress the document to be processed based on the query semantic tree to obtain the compressed document.

[0064] It should be understood that the compressed document mentioned above can be a document obtained by compressing the document to be processed, and the text paragraphs therein can be stored in the form of compressed vectors. In this embodiment, the compression process of the document to be processed can be guided by the query semantic tree, so that the compression process can reduce the storage space of the document and the computational overhead of subsequent processing while preserving the semantic information related to the query as much as possible, and at the same time make the document content more adaptable to the distribution characteristics of the query space, that is, the structure of the compressed document can correspond to the hierarchical structure of the query semantic tree.

[0065] Further, the step of compressing the document to be processed based on the query semantic tree to obtain the compressed document includes: Step S031: Construct a preset vector compression objective function based on the query vector in the query semantic tree and the text paragraph vector in the document to be processed.

[0066] It should be noted that the aforementioned query vector can be the vector representation of the user query; correspondingly, the aforementioned text paragraph vector can be the vector representation of each text paragraph in the document to be processed. In this embodiment, the system can encode the user query text using a pre-trained language model (such as BERT, RoBERTa, or Sentence-BERT) to obtain the query vector, and encode the text paragraphs using the language model to obtain the text paragraph vectors.

[0067] It should also be noted that the aforementioned preset vector compression objective function can be an optimization objective function used to guide the document compression process. Its optimization objective can be to maintain the semantic similarity relationship between the query and the text paragraph before and after compression. In this embodiment, the preset vector compression objective function differs from the traditional compression objective that only considers the document's own reconstruction error. It can introduce the guidance of query semantics, so that the document compression process can prioritize the preservation of semantic information that is highly related to the distribution of the query space.

[0068] Furthermore, the step of constructing a preset vector compression objective function based on the query vector in the query semantic tree and the text paragraph vector in the document to be processed includes: determining the second-order momentum matrix of the query latent space based on the query vector in the query semantic tree; and constructing the preset vector compression objective function based on the second-order momentum matrix and the text paragraph vector in the document to be processed.

[0069] It should be noted that the latent space of a query can be a semantic space composed of vector representations of user queries. In practical applications, the system can map user queries into vectors using a semantic embedding model. The mathematical space formed by these query vectors is the latent space of the query. The coordinate axes of the latent space can correspond to the latent dimensions of the semantics. Each query vector can be a point in the latent space, and the geometric relationships between vectors (such as distance and angle) are used to reflect the semantic similarity between queries.

[0070] It should also be noted that the second-order momentum matrix of the query latent space mentioned above can be obtained by taking the second moment of the distribution of the query vector in the query latent space, which can be used to characterize the statistical properties of the query semantic space.

[0071] It should be noted that in vector quantization compression, codebooks are typically used. The original vector space is represented by a set of cluster centers. In this embodiment, the document compression process can reduce each original vector space to a single value. Mapped to the most recent codeword The compressed vector is obtained. Specifically, it can be expressed as And soon Mapped to the most recent codeword The objective of traditional compression is to minimize the reconstruction error between the original vector and the compressed vector. In this case, the objective function of traditional vector compression is... It can be represented as:

[0072] In the formula, Represents the original text paragraph vector (from the distribution of the document to be processed). ), This represents the compressed reconstructed vector (obtained through codebook quantization). Indicates document distribution The expectation of all sentences in the codebook is calculated to minimize the reconstruction error between the original vector and the compressed vector. In other words, the original vector is mapped to the nearest cluster center in the codebook, so as to minimize the information loss.

[0073] Therefore, the optimization objective of traditional vector compression is to make the compressed vector as close as possible to the original vector in the absolute vector space. In other words, it only considers the document's own reconstruction error during compression, and is unrelated to the query. However, in information retrieval scenarios, it is usually necessary to consider the relative relationship between the query and the document, i.e., the query vector... With document vectors inner product Because traditional vector compression objective functions do not consider this retrieval relevance, the compressed vector, while having a small error in absolute space, undergoes a significant change in its inner product with the query vector, thus affecting query accuracy.

[0074] Therefore, in order to improve the accuracy of document retrieval, this embodiment can guide document compression based on query awareness. In this case, the preset vector compression objective function based on query guidance in this embodiment can be expressed as:

[0075]

[0076]

[0077]

[0078]

[0079] In the formula, Let be the joint expectation in the predefined vector compression objective function, representing the expectation of both the document space and the query space simultaneously. The text paragraph vectors represent the distribution of the documents to be processed. , This indicates that the query vector comes from the distribution of documents to be processed. .also, This represents the original query vector.

[0080] Therefore, it can be seen that the optimization objective of the preset vector compression objective function based on query guidance in this embodiment is to ensure that the compressed vector is optimized. and The inner product should be as close as possible to the original. and The inner product. Meanwhile, this embodiment, through considering all possible... The expectation is to ensure that the compressed representation maintains the same relevance properties as the original representation across the entire query space.

[0081] In practical applications, ,in, To guide the query, the second-order momentum matrix of the latent space is used, which can capture the distribution characteristics of the query space. At this point, the objective function for pre-defined vector compression based on the query guidance can be simplified to:

[0082] As can be seen from the above formula, the compression error at this time is... Can be matrix Weighting is used to enable query-aware adaptive document compression.

[0083] Step S032: Compress the document to be processed based on the preset vector compression objective function to obtain the compressed document.

[0084] In practical applications, the system can extract all query vectors from the query semantic tree and calculate the second-order momentum matrix of the query latent space based on the set of query vectors. Then, a preset vector compression objective function is constructed based on the second-order momentum matrix. Subsequently, when document compression is needed, the system can first initialize the codebook based on the text paragraph vectors. The system iteratively optimizes the codebook based on a preset vector compression objective function to assign each text segment vector to the codebook center with the closest weighted distance and updates each codebook center, thereby minimizing the weighted reconstruction error. Then, the system can use the optimized codebook to quantize and compress the text segment vectors and organize the compressed text segments into a compressed document.

[0085] Step S04: Perform hierarchical clustering on the compressed document to construct a preset document semantic tree.

[0086] In practical applications, after receiving the current user query, the system can identify the query scenario to which the query belongs through keyword matching or classification models. Based on the identified query scenario, it extracts a set of related user queries from the query database and simultaneously filters a set of documents matching the query scenario from the document library. Then, the system encodes each user query in the query set into a dense vector representation, obtaining a set of user query vectors. It then performs cluster analysis on this set to obtain query cluster centers, and recursively clusters these centers to construct a deeper query semantic tree. Next, the system divides each document in the document set into several paragraphs and calculates query guidance compression parameters based on the query semantic tree, such as the second-order momentum matrix of the query latent space. Based on this query guidance matrix, it compresses the text paragraph vectors in the documents, organizes the compressed text paragraphs into compressed documents, and finally performs hierarchical clustering on the compressed documents to construct a predefined document semantic tree.

[0087] This embodiment discloses a method for obtaining a user query set and a document to be processed based on the query scenario of the current user query input by the user; clustering each user query in the user query set to obtain query cluster centers, and constructing a query semantic tree based on the query cluster centers, the query semantic tree being used to represent the semantic relationships between each user query; compressing the document to be processed based on the query semantic tree to obtain a compressed document; and performing hierarchical clustering on the compressed document to construct a preset document semantic tree. This embodiment, by clustering the user query set and constructing a query semantic tree, and compressing the document to be processed based on the query semantic tree, enables query-aware adaptive document compression, thereby maintaining a high compression rate while prioritizing the retention of semantic information highly relevant to the user's query needs, thus improving the accuracy of subsequent document queries.

[0088] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating the document retrieval method of this application in Embodiment 3.

[0089] In this embodiment, step S04 further includes steps S041 to S043: Step S041: Segment the compressed document into paragraphs to obtain several text paragraphs.

[0090] Understandably, the aforementioned text paragraphs can be basic text units with independent semantic integrity within a compressed document. In practical applications, each text paragraph can be a continuous text segment with relatively independent semantics within a document, such as a chapter, a paragraph, a code function block, or a table description.

[0091] Step S042: Perform hierarchical clustering on each of the text paragraphs to determine the paragraph index corresponding to each text paragraph.

[0092] It should be understood that hierarchical clustering can be a progressive clustering method. In this embodiment, by performing multiple rounds of clustering on text paragraphs, a hierarchical document semantic tree from fine-grained to coarse-grained can be constructed. In this embodiment, hierarchical clustering may include single-document clustering at the first layer and cross-document clustering at the second layer.

[0093] It should also be understood that the above-mentioned paragraph index can be identification information used to uniquely identify and locate text paragraphs in a preset document semantic tree. In this embodiment, the paragraph index may include, but is not limited to, paragraph ID, node ID and corresponding semantic vector representation. For example, for leaf nodes, the paragraph index can directly point to the original text paragraph; for internal nodes (such as text paragraph centroids, document-level centroids), the paragraph index can point to the corresponding cluster center node and contain a list of pointers to its child nodes.

[0094] Step S043: Construct a preset document semantic tree based on each of the text paragraphs and the paragraph index.

[0095] In this embodiment, the system can use a residual clustering algorithm to process text paragraphs, generating text paragraph centroids as first-level index nodes. Then, cross-document clustering is performed on text paragraphs from different documents to generate document-level centroids as second-level index nodes. Finally, a tree structure with semantic hierarchical relationships is constructed based on the text paragraphs and these indexes (including first-level and second-level index nodes) to obtain a preset document semantic tree. Each text paragraph node in the preset document semantic tree can contain information such as: text content, its offset position in the original document, and a unique identifier of its document.

[0096] Further, the paragraph index includes a first-level paragraph index and a second-level paragraph index; step S042 includes: performing a first-level clustering on each of the text paragraphs to obtain a first-level paragraph index, the first-level paragraph index being used to characterize at least one text paragraph in the compressed document; determining cross-document text paragraphs from each of the text paragraphs, and performing a second-level clustering on the cross-document text paragraphs to obtain a second-level paragraph index, the second-level paragraph index being used to characterize text paragraphs in at least two different documents.

[0097] It should be noted that the aforementioned first-level paragraph index can be an index node generated by clustering text paragraphs within a single document. It can be used to represent the aggregation relationship of at least one text paragraph within the same document. In practical applications, the first-level paragraph index can correspond to the first-level internal node in the document semantic tree, i.e., the segment-level centroid. It can organize multiple semantically similar paragraphs within the same document together to form a semantic block. In this embodiment, the first-level paragraph index can include a summary representation of semantically similar semantic blocks, the covered paragraph range, and pointers to child nodes (original paragraphs). Specifically, it can be represented as follows: ,in, This represents the first-level paragraph index node in the document semantic tree. Indicates to The summary vector representation obtained by summarizing the paragraphs corresponding to the included paragraph IDs. Indicates the height of the tree. This represents the paragraph ID value corresponding to the tree node.

[0098] It should also be noted that the aforementioned second-level paragraph index can be an index node generated after clustering text paragraphs across documents. It can be used to represent the aggregation relationship of text paragraphs in at least two different documents. In practical applications, the second-level paragraph index can correspond to the second-level internal node in the document semantic tree, that is, the document-level centroid. It can organize semantically similar paragraphs (which can be original paragraphs or first-level paragraph index nodes) in different documents together to form a cross-document topic cluster. In this embodiment, the second-level paragraph index node can contain a summary representation of the cross-document topic cluster, the document range it covers, and pointers to child nodes (such as first-level paragraph index nodes or original paragraphs). This embodiment does not impose any limitations on this.

[0099] Step S043 includes: constructing a hierarchical document semantic tree corresponding to the compressed document based on each of the text paragraphs, the first-level paragraph index, and the second-level paragraph index.

[0100] It should be understood that the aforementioned hierarchical document semantic tree can be a multi-layered tree index structure built based on the document's semantic structure. It can capture the semantic hierarchical relationships and associations within and between documents, thus providing a structured index foundation for accurate semantic retrieval. In this embodiment, the hierarchical document semantic tree may contain three core layers: a leaf node layer (corresponding to the original text paragraphs), a first-level index node layer (corresponding to the paragraph-level centroid), and a second-level index node layer (corresponding to the document-level centroid).

[0101] In practical applications, after obtaining the compressed document, the system can segment the document into paragraphs to obtain a set of document paragraphs. Then, a residual clustering algorithm can be used to perform a first-level clustering of the text paragraphs in a single document, and a first-level paragraph index can be built for each cluster obtained from the clustering. Subsequently, the system can filter paragraphs or sets of paragraphs with cross-document relationships from the text paragraph set, and continue to use the residual clustering algorithm to perform a second-level clustering of the identified cross-document text paragraphs, and then build a second-level paragraph index for each cluster obtained from the clustering. Finally, the system can organize the text paragraphs, the first-level paragraph index, and the second-level paragraph index into a tree structure to obtain a hierarchical document semantic tree.

[0102] Further, the step of performing a first-level clustering on each of the text segments to obtain a first-level segment index includes: performing a first-level clustering on each of the text segments using a residual clustering algorithm to determine the summary embedding vector corresponding to each of the text segments; determining the text segment identifier range based on the text segment identifier corresponding to each of the text segments; and constructing a first-level segment index based on the summary embedding vector, the tree height identifier, and the text segment identifier range.

[0103] Understandably, the above residual clustering algorithm can be a hierarchical clustering algorithm based on residual iteration. This algorithm can calculate the residual after clustering the data of the current layer in each iteration, and continue to cluster the residual in the next layer until the stopping condition is met.

[0104] It is also understood that the above-mentioned summary embedding vector can be a vector representation obtained by summarizing the text content of all text paragraphs within a cluster, which can be used to characterize the overall semantic features of all text paragraphs within a cluster. In this embodiment, the summary embedding vector can be the mean of all paragraph vectors within a cluster (i.e., the cluster center), or it can be a semantic vector obtained by generating a text summary through a large language model and then encoding it. This embodiment does not impose any restrictions on this.

[0105] It should be noted that the tree height identifier mentioned above can be a numerical marker used to identify the level of a node in a hierarchical document semantic tree, such as h=1; the text paragraph identifier range mentioned above can be the identifier set of the original text paragraphs covered by each cluster obtained from the first-level clustering, which can be used to clearly define the coverage of each cluster.

[0106] In practical applications, the system can group text paragraph sets according to their respective documents and perform an iterative process of residual clustering algorithm on the text paragraph subsets of a single document to obtain multiple clusters and their corresponding cluster centers. Then, for each cluster, the system can extract the original text content of all text paragraphs contained within it and use a pre-trained language model to generate semantic summaries of the text within the cluster. The generated summary text is then encoded into a dense vector of fixed dimensions to obtain a summary embedding vector. Simultaneously, for each cluster, the system can collect unique identifiers for all text paragraphs contained within it and determine the text paragraph identification range based on these identifiers. Finally, a first-level paragraph index is constructed based on the summary embedding vector, tree height identifier (e.g., h=1), and text paragraph identification range.

[0107] In the specific implementation, refer to Figure 4 , Figure 4 This is a schematic diagram of the document semantic tree structure in the document query method of this application. For example... Figure 4As shown, the preset document semantic tree in this embodiment can be a three-layer tree hierarchy, including a leaf node layer, a first-layer index node layer, and a second-layer index node layer. Each node in the leaf node layer corresponds to a text paragraph in the document, such as paragraph S1, paragraph S2, paragraph S3, etc. in document 1. The first-layer index node can correspond to a paragraph-level centroid, such as text paragraph centroid n1, text paragraph centroid n2, etc., and the corresponding tree height identifier can be h=1. The second-layer index node can correspond to a document-level centroid.

[0108] Furthermore, after the step of performing hierarchical clustering on the compressed document to construct a preset document semantic tree, the method further includes: using the query cluster center as the key to construct an inverted index linked list pointing to the second-level paragraph index in the preset document semantic tree; the step of determining the target compressed text paragraph in the preset document semantic tree based on the target paragraph index matched by the target user query includes: determining the target paragraph index matched by the target user query based on the inverted index linked list, and determining the target compressed text paragraph in the preset document semantic tree based on the target paragraph index.

[0109] It should be noted that the aforementioned inverted index linked list can be a mapping data structure with the query cluster center as the key and the second-level paragraph index list as the value. In this embodiment, the inverted index linked list can achieve a fast mapping from the query semantic space to the document semantic space, thereby improving document query efficiency.

[0110] In practical applications, the system can extract query cluster centers from the query semantic tree as the key set for the inverted index, and extract second-level paragraph indices from the preset document semantic tree as the value candidate set for the inverted index. Then, it calculates the semantic similarity between each query cluster center and each second-level paragraph index to determine the target second-level paragraph index with the highest similarity to each query cluster center. Finally, it establishes a mapping relationship between each query cluster center and the target second-level paragraph index to construct the inverted index linked list. Subsequently, when performing document queries, the system can directly query the inverted index linked list for the target paragraph index that matches the user's query, and determine the target compressed text paragraph in the preset document semantic tree based on the matching target paragraph index.

[0111] Further, the step of determining the target compressed text paragraph in the preset document semantic tree based on the target paragraph index matched by the target user query includes: determining the target matching document in the preset document semantic tree based on the target paragraph index matched by the target user query; and querying the preset document semantic tree for the target compressed text paragraph that matches the current user query based on the target matching document.

[0112] It should be understood that the aforementioned target matching documents can be one or more documents most relevant to the query, retrieved from a pre-defined document semantic tree. In practical applications, after determining the target paragraph index matching the target user query, the system can directly obtain the set of document identifiers associated with the target paragraph index from the paragraph index list, and determine the corresponding target matching document based on the document identifier set. Then, the system can obtain several candidate first-level index nodes corresponding to the target matching document in the pre-defined document semantic tree, and calculate the similarity between the current user query and the summary vector of each candidate first-level index node to determine the target first-level index node with the highest matching degree to the current user query. Finally, the system can calculate the similarity between the current user query and the compressed vector (i.e., text paragraph vector) of the leaf node corresponding to the target first-level index node, thereby determining the target leaf node with the highest matching degree to the current user query, and determining the target compressed text paragraph matching the current user query based on the identifier of the target leaf node.

[0113] In this embodiment, a method is disclosed to segment the compressed document into paragraphs to obtain several text paragraphs; perform hierarchical clustering on each text paragraph to determine the paragraph index corresponding to each text paragraph; and construct a preset document semantic tree based on each text paragraph and the paragraph index. Since this embodiment can construct a preset document semantic tree based on the text paragraphs in the compressed document and their corresponding paragraph indexes, subsequent document queries can directly determine the matching text paragraphs based on the index corresponding to the user's query, thereby improving document query efficiency.

[0114] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the document search method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0115] This application also provides a document retrieval device; please refer to [reference needed]. Figure 5 The document query device includes: The query generation module 10 is used to generate a target user query based on the current user query input by the user and the user's historical question-and-answer pair sequence; Document query module 20 is used to determine the target compressed text paragraph in a preset document semantic tree based on the target paragraph index matched by the target user query. The preset document semantic tree stores the mapping relationship between compressed text paragraphs and paragraph indexes in the document. The result generation module 30 is used to generate the document query result corresponding to the current user query based on the target compressed text paragraph.

[0116] The document query device provided in this application, employing the document query method in the above embodiments, can solve the technical problem that existing document query methods may result in low accuracy of query results due to the retrieved document paragraphs exceeding the model's context processing capabilities. Compared with the prior art, the beneficial effects of the document query device provided in this application are the same as those of the document query method provided in the above embodiments, and other technical features in the document query device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0117] This application provides a document query device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the document query method in Embodiment 1 above.

[0118] The following is for reference. Figure 6 The diagram illustrates a structural schematic of a document retrieval device suitable for implementing embodiments of this application. The document retrieval device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The document query device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0119] like Figure 6As shown, the document retrieval device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the document retrieval device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the document retrieval device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows document retrieval devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0120] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0121] The document query device provided in this application, employing the document query method in the above embodiments, can solve the technical problem of document query. Compared with the prior art, the beneficial effects of the document query device provided in this application are the same as those of the document query method provided in the above embodiments, and other technical features of the document query device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0122] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0123] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0124] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the document query method in the above embodiments.

[0125] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0126] The aforementioned computer-readable storage medium may be included in the document retrieval device; or it may exist independently and not be assembled into the document retrieval device.

[0127] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a document query device, cause the document query device to: generate a target user query based on the current user query input by the user and the user's historical question-and-answer pair sequence; determine a target compressed text segment in a preset document semantic tree according to the target segment index matched by the target user query, wherein the preset document semantic tree stores the mapping relationship between compressed text segments and segment indexes in the target document, and the target document is a document that matches the query scenario to which the current user query belongs; and generate a document query result corresponding to the current user query based on the target compressed text segment.

[0128] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0130] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0131] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described document query method. This solves the technical problem that existing document query methods may result in low accuracy of query results due to retrieved document paragraphs exceeding the model's context processing capabilities. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the document query method provided in the above embodiments, and will not be repeated here.

[0132] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the document query method described above.

[0133] The computer program product provided in this application can solve the technical problem that existing document query methods may result in low accuracy of query results because the retrieved document paragraphs exceed the context processing capabilities of the model. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the document query method provided in the above embodiments, and will not be repeated here.

[0134] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

[0135] This invention discloses A1, a document query method, the method comprising: A target user query is generated based on the current user query input and the user's historical question-and-answer pair sequence. The target compressed text paragraph is determined in the preset document semantic tree based on the target paragraph index matched by the target user query. The preset document semantic tree stores the mapping relationship between the compressed text paragraphs in the target document and the paragraph index. The target document is the document that matches the query scenario to which the current user query belongs. Based on the target compressed text paragraph, the document query results corresponding to the current user's query are generated.

[0136] A2. As described in A1, before the step of generating a target user query based on the current user query input and the user's historical question-and-answer pair sequence, the method further includes: Based on the query scenario of the current user query, obtain the user query set and the documents to be processed; Clustering is performed on each user query in the user query set to obtain query cluster centers, and a query semantic tree is constructed based on the query cluster centers. The query semantic tree is used to represent the semantic relationship between each user query. The document to be processed is compressed based on the query semantic tree to obtain a compressed document; The compressed documents are hierarchically clustered to construct a preset document semantic tree.

[0137] A3. As described in A2, the step of compressing the document to be processed based on the query semantic tree to obtain the compressed document includes: A preset vector compression objective function is constructed based on the query vector in the query semantic tree and the text paragraph vector in the document to be processed. The document to be processed is compressed based on the preset vector compression objective function to obtain the compressed document.

[0138] A4. As described in A3, the step of constructing a preset vector compression objective function based on the query vector in the query semantic tree and the text paragraph vector in the document to be processed includes: The second-order momentum matrix of the query latent space is determined based on the query vector in the query semantic tree; A preset vector compression objective function is constructed based on the second-order momentum matrix and the text paragraph vectors in the document to be processed.

[0139] A5. The method described in A2, wherein the step of performing hierarchical clustering on the compressed document to construct a preset document semantic tree includes: The compressed document is segmented into paragraphs to obtain several text paragraphs; Hierarchical clustering is performed on each of the text paragraphs to determine the paragraph index corresponding to each text paragraph; A preset document semantic tree is constructed based on each of the text paragraphs and the paragraph index.

[0140] A6. As described in A5, the paragraph index includes a first-level paragraph index and a second-level paragraph index; the step of performing hierarchical clustering on each of the text paragraphs to determine the paragraph index corresponding to each of the text paragraphs includes: Perform a first-level clustering on each of the text paragraphs to obtain a first-level paragraph index, which is used to characterize at least one text paragraph in the compressed document. Cross-document text paragraphs are identified from each of the text paragraphs, and a second-level clustering is performed on the cross-document text paragraphs to obtain a second-level paragraph index, which is used to characterize text paragraphs in at least two different documents; The step of constructing a preset document semantic tree based on each of the text paragraphs and the paragraph index includes: A hierarchical document semantic tree corresponding to the compressed document is constructed based on each of the text paragraphs, the first-level paragraph index, and the second-level paragraph index.

[0141] A7. As described in A5, the step of performing a first-level clustering on each of the text paragraphs to obtain a first-level paragraph index includes: The first-level clustering of each text segment is performed using a residual clustering algorithm to determine the summary embedding vector corresponding to each text segment. The range of text paragraph identifiers is determined based on the text paragraph identifiers corresponding to each of the aforementioned text paragraphs; The first-level paragraph index is constructed based on the summary embedding vector, the tree height identifier, and the text paragraph identifier range.

[0142] A8. The method as described in any one of A2 to A7, after the step of performing hierarchical clustering on the compressed document to construct a preset document semantic tree, further includes: Using the query cluster center as the key, construct an inverted index linked list pointing to the second-level paragraph index in the preset document semantic tree; The step of determining the target compressed text paragraph in the preset document semantic tree based on the target paragraph index matched by the target user query includes: The target paragraph index matching the target user query is determined based on the inverted index linked list, and the target compressed text paragraph is determined in the preset document semantic tree based on the target paragraph index.

[0143] A9. The method as described in any one of A1 to A7, wherein the step of determining the target compressed text paragraph in the preset document semantic tree based on the target paragraph index matched by the target user query includes: The target matching document is determined in the preset document semantic tree based on the target paragraph index matched by the target user query. Based on the target matching document, the system queries the preset document semantic tree to find the target compressed text paragraph that matches the current user's query.

[0144] A10. The method as described in any one of A1 to A7, wherein the step of generating a target user query based on the current user query input by the user and the user's historical question-and-answer pair sequence includes: A pre-filled context is constructed based on the current user query input and the user's historical question-and-answer pair sequence; The document query results corresponding to the current user's query are generated based on the pre-populated context.

[0145] The present invention also discloses B11, a document retrieval device, the device comprising: The query generation module is used to generate a target user query based on the current user query input by the user and the user's historical question-and-answer pair sequence; The document query module is used to determine the target compressed text paragraph in a preset document semantic tree based on the target paragraph index matched by the target user's query. The preset document semantic tree stores the mapping relationship between the compressed text paragraphs in the document and the paragraph index. The result generation module is used to generate the document query result corresponding to the current user query based on the target compressed text paragraph.

[0146] B12. The device as described in B11, wherein the document query module is further configured to obtain a user query set and documents to be processed based on the query scenario to which the current user query belongs, as input by the user. The document query module is further configured to cluster each user query in the user query set to obtain query cluster centers, and construct a query semantic tree based on the query cluster centers. The query semantic tree is used to represent the semantic relationship between each user query. The document query module is also used to compress the document to be processed based on the query semantic tree to obtain the compressed document; The document query module is also used to perform hierarchical clustering on the compressed documents to construct a preset document semantic tree.

[0147] B13. The device as described in B12, wherein the document query module is further configured to construct a preset vector compression objective function based on the query vector in the query semantic tree and the text paragraph vector in the document to be processed; The document query module is also used to compress the document to be processed based on the preset vector compression objective function to obtain the compressed document.

[0148] B14. The apparatus as described in B13, wherein the document query module is further configured to determine the second-order momentum matrix of the query latent space based on the query vector in the query semantic tree; The document query module is also used to construct a preset vector compression objective function based on the second-order momentum matrix and the text paragraph vectors in the document to be processed.

[0149] B15. The device as described in B12, wherein the document query module is further configured to perform paragraph segmentation on the compressed document to obtain several text paragraphs; The document query module is also used to perform hierarchical clustering on each of the text paragraphs to determine the paragraph index corresponding to each of the text paragraphs; The document query module is also used to construct a preset document semantic tree based on each of the text paragraphs and the paragraph index.

[0150] B16. The apparatus as described in B15, wherein the paragraph index includes a first-level paragraph index and a second-level paragraph index; the document query module is further configured to perform a first-level clustering on each of the text paragraphs to obtain a first-level paragraph index, wherein the first-level paragraph index is used to characterize at least one text paragraph in the compressed document. The document query module is further configured to determine cross-document text paragraphs from each of the text paragraphs, and perform a second-level clustering on the cross-document text paragraphs to obtain a second-level paragraph index, wherein the second-level paragraph index is used to characterize text paragraphs in at least two different documents; The document query module is also used to construct a hierarchical document semantic tree corresponding to the compressed document based on each of the text paragraphs, the first-level paragraph index, and the second-level paragraph index.

[0151] B17. The apparatus as described in any one of B12 to B16, wherein the document query module is further configured to use the query cluster center as a key to construct an inverted index linked list pointing to the second-level paragraph index in the preset document semantic tree; The document query module is further configured to determine the target paragraph index matching the target user query based on the inverted index list, and determine the target compressed text paragraph in the preset document semantic tree based on the target paragraph index.

[0152] The present invention also discloses C18, a document query device, the device comprising: a memory, a processor, and a document query program stored in the memory and executable on the processor, the document query program being configured to implement the steps of the document query method described above.

[0153] The present invention also discloses D19, a storage medium storing a document query program, wherein the document query program, when executed by a processor, implements the steps of the document query method described above.

[0154] The present invention also discloses E20, a computer program product comprising a computer program that, when executed by a processor, implements the steps of the document query method described above.

Claims

1. A document retrieval method, characterized in that, The method includes: A target user query is generated based on the current user query input and the user's historical question-and-answer pair sequence. The target compressed text paragraph is determined in the preset document semantic tree based on the target paragraph index matched by the target user query. The preset document semantic tree stores the mapping relationship between the compressed text paragraphs in the target document and the paragraph index. The target document is the document that matches the query scenario to which the current user query belongs. Based on the target compressed text paragraph, the document query results corresponding to the current user's query are generated.

2. The method as described in claim 1, characterized in that, Before the step of generating a target user query based on the current user query and the user's historical question-and-answer pair sequence, the method further includes: Based on the query scenario of the current user query, obtain the user query set and the documents to be processed; Clustering is performed on each user query in the user query set to obtain query cluster centers, and a query semantic tree is constructed based on the query cluster centers. The query semantic tree is used to represent the semantic relationship between each user query. The document to be processed is compressed based on the query semantic tree to obtain a compressed document; The compressed documents are hierarchically clustered to construct a preset document semantic tree.

3. The method as described in claim 2, characterized in that, The step of compressing the document to be processed based on the query semantic tree to obtain the compressed document includes: A preset vector compression objective function is constructed based on the query vector in the query semantic tree and the text paragraph vector in the document to be processed. The document to be processed is compressed based on the preset vector compression objective function to obtain the compressed document.

4. The method as described in claim 3, characterized in that, The step of constructing a preset vector compression objective function based on the query vector in the query semantic tree and the text paragraph vector in the document to be processed includes: The second-order momentum matrix of the query latent space is determined based on the query vector in the query semantic tree; A preset vector compression objective function is constructed based on the second-order momentum matrix and the text paragraph vectors in the document to be processed.

5. The method as described in claim 2, characterized in that, The step of performing hierarchical clustering on the compressed document to construct a preset document semantic tree includes: The compressed document is segmented into paragraphs to obtain several text paragraphs; Hierarchical clustering is performed on each of the text paragraphs to determine the paragraph index corresponding to each text paragraph; A preset document semantic tree is constructed based on each of the text paragraphs and the paragraph index.

6. The method as described in claim 5, characterized in that, The paragraph index includes a first-level paragraph index and a second-level paragraph index; the step of performing hierarchical clustering on each of the text paragraphs to determine the paragraph index corresponding to each text paragraph includes: Perform a first-level clustering on each of the text paragraphs to obtain a first-level paragraph index, which is used to characterize at least one text paragraph in the compressed document. Cross-document text paragraphs are identified from each of the text paragraphs, and a second-level clustering is performed on the cross-document text paragraphs to obtain a second-level paragraph index, which is used to characterize text paragraphs in at least two different documents; The step of constructing a preset document semantic tree based on each of the text paragraphs and the paragraph index includes: A hierarchical document semantic tree corresponding to the compressed document is constructed based on each of the text paragraphs, the first-level paragraph index, and the second-level paragraph index.

7. A document retrieval device, characterized in that, The device includes: The query generation module is used to generate a target user query based on the current user query input by the user and the user's historical question-and-answer pair sequence; The document query module is used to determine the target compressed text paragraph in a preset document semantic tree based on the target paragraph index matched by the target user's query. The preset document semantic tree stores the mapping relationship between the compressed text paragraphs in the document and the paragraph index. The result generation module is used to generate the document query result corresponding to the current user query based on the target compressed text paragraph.

8. A document retrieval device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the document query method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the document query method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the document query method as described in any one of claims 1 to 6.

Images