Large scale data query method fusing deep semantic representation and explainable retrieval

Abstract

The application discloses a large-scale data query method fusing deep semantic representation and explainable retrieval, and belongs to the technical field of information retrieval. The method comprises the following steps: adopting a hierarchical Transformer encoder to perform word-level, phrase-level and sentence-level multi-granularity semantic feature extraction on a query text, and dynamically fusing to generate multi-scale semantic embedding through a gating mechanism; constructing a semantic-symbol dual-channel layered index structure, and dynamically switching a retrieval channel according to a semantic correlation threshold; identifying a query type through an intention classifier and selecting an optimal retrieval strategy; extracting attention weights of each layer of the encoder to generate a semantic correlation strength matrix through cross-layer aggregation, and outputting an explainable retrieval result description; and adopting a write-once-copy and block incremental reconstruction strategy to realize incremental index maintenance. The application realizes efficient, accurate and explainable large-scale data query.

Description

Technical Field

[0001] This invention belongs to the field of information retrieval technology, specifically relating to a large-scale data query method that integrates deep semantic representation and interpretable retrieval. Background Technology

[0002] With the explosive growth of internet data, efficiently and accurately retrieving the information users need from massive amounts of data has become a core challenge in the field of information retrieval. Traditional keyword-based retrieval methods rely on precise matching at the lexical level, making it difficult to handle semantic-level query needs. When there are lexical differences between the user's query and the target document, the retrieval effect drops significantly. In recent years, semantic retrieval technology based on deep learning has made great progress. By mapping queries and documents to a unified semantic vector space and using vector similarity for matching, it effectively solves the problem of lexical mismatch.

[0003] Chinese patent CN113705242A discloses an intelligent semantic matching method for educational consulting services. This method constructs a semantic matching model consisting of a multi-granularity embedding module, a dual-attention semantic matching module, a feature aggregation module, and a label prediction module. It performs self-attention operations on the word-level granularity of sentences to obtain key semantic feature representations and performs mutual attention operations between sentences to obtain semantic alignment feature representations. This method primarily targets sentence-pair semantic matching tasks, employing LSTM and BiLSTM encoders for feature extraction and realizing semantic interaction between sentences through self-attention and mutual attention mechanisms. However, this method has the following technical shortcomings: First, its semantic representation granularity is limited, considering only the character and word levels, failing to effectively capture hierarchical semantic information at the phrase and sentence levels, resulting in limited ability to understand complex semantic structures. Second, this method is mainly used for binary classification tasks to determine whether two sentences are semantically consistent, without addressing index building and retrieval efficiency issues in large-scale document retrieval scenarios, thus failing to meet the needs of rapid retrieval of massive amounts of data. Third, the retrieval process lacks interpretability design, failing to explain the basis of the matching results to users, reducing the system's credibility and user acceptance. Fourth, this method does not consider dynamic data update scenarios, lacks incremental index maintenance capabilities, and is difficult to adapt to the actual application needs of continuous data inflow.

[0004] Currently, researchers in the field of information retrieval are attempting to address the aforementioned problems using various technical approaches. In terms of semantic representation, Transformer-based pre-trained models such as BERT and DPR have become mainstream solutions, learning deep semantic representations through self-attention mechanisms. Regarding indexing efficiency, techniques such as product quantization and inverted indexes are widely used to accelerate nearest neighbor searches. In terms of retrieval strategies, hybrid retrieval methods attempt to combine the advantages of sparse and dense retrieval. Regarding interpretability, techniques such as attention visualization and feature attribution are used to explain model decisions. However, existing solutions still have shortcomings in organically integrating these technologies: there is a lack of collaborative design between multi-granular semantic representation and efficient indexing structures; the adaptability of retrieval strategies is insufficient; the coupling between interpretability mechanisms and the retrieval process is low; and it is difficult to balance incremental updates with retrieval performance. These problems lead to the poor performance of existing semantic retrieval systems when dealing with the complex needs of real-world scenarios. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a large-scale data query method that integrates deep semantic representation and interpretable retrieval. By constructing a multi-granularity semantic representation module, a semantic-symbolic dual-channel hierarchical index structure, an adaptive retrieval path planning mechanism that is aware of query intent, a semantic association interpretation generation module based on attention tracing, and an incremental index maintenance mechanism, it achieves efficient, accurate, and interpretable large-scale data query.

[0006] The large-scale data query method that integrates deep semantic representation and interpretable retrieval provided by this invention includes the following steps:

[0007] The multi-granular semantic representation steps involve acquiring the query text, using a hierarchical Transformer encoder to extract features layer by layer, capturing semantic boundary features at the lexical level through a local attention mechanism to obtain lexical-level feature vectors, introducing constituent syntactic constraints at the phrase level to aggregate semantic units of lexical-level features to obtain phrase-level feature vectors, and fusing global self-attention at the sentence level to generate context-aware dense vectors to obtain sentence-level feature vectors. Finally, a learnable gating mechanism is used to dynamically weight and fuse lexical-level feature vectors, phrase-level feature vectors, and sentence-level feature vectors to generate multi-scale semantic embedding vectors.

[0008] The dual-channel index construction and retrieval steps involve constructing a semantic-symbolic dual-channel hierarchical index structure. The semantic channel uses a product quantization and inverted file joint encoding strategy to encode the dense vector of the document, while the symbolic channel retains the inverted index of key entity terms in the document. Based on the comparison results between the semantic relevance score of the query and the preset semantic relevance threshold, the semantic channel and the symbolic channel are dynamically switched to retrieve candidate documents.

[0009] The query intent-aware adaptive retrieval path planning steps identify the query type of the query text through an intent classifier, select a retrieval strategy based on the query type, adopt a cascading retrieval mode of symbol first and then semantic for factual queries, adopt a semantic-driven parallel recall mode for exploratory queries, and dynamically allocate retrieval weights of semantic and symbolic channels for joint retrieval for mixed queries, outputting a set of candidate documents.

[0010] The semantic association interpretation generation step based on attention source tracing extracts the attention weight distribution of each layer of the hierarchical Transformer encoder, performs cross-layer aggregation on the attention weights of each layer, generates a semantic association strength matrix between query terms and candidate document terms, identifies key semantic matching paths based on the semantic association strength matrix and extracts supporting evidence fragments, and generates interpretability descriptions of the search results.

[0011] The incremental index maintenance process employs a copy-on-write strategy to temporarily store new documents in a memory buffer and build a temporary index. When the amount of data in the memory buffer reaches a preset buffer capacity threshold, a background merging thread is triggered to asynchronously merge the temporary index with the main index. The merging process uses a block-based incremental reconstruction strategy to update the dual-channel hierarchical index structure.

[0012] Preferably, in the multi-granularity semantic representation step, the attention window size of the local attention mechanism is 3 to 7 lexical units, the maximum phrase length used for the constituent syntactic constraints is 5 to 10 lexical units, and the sum of the fusion weights output by the learnable gating mechanism is 1.

[0013] Preferably, in the dual-channel index construction and retrieval step, the product quantization decomposes the high-dimensional dense vector into 8 to 16 low-dimensional subspaces, the codebook size of each subspace is 256, and the semantic relevance threshold ranges from 0.65 to 0.85.

[0014] Preferably, in the adaptive retrieval path planning step of query intent awareness, the classification confidence threshold of the intent classifier is 0.7 to 0.9, and a hybrid query retrieval strategy is adopted when the classification confidence is lower than the classification confidence threshold.

[0015] The beneficial effects of this invention are as follows:

[0016] First, this invention employs a hierarchical Transformer encoder to achieve multi-granular semantic representation at the word, phrase, and sentence levels. By dynamically fusing features at each granularity through a learnable gating mechanism, compared to existing schemes that only consider word and phrase granularity, this invention can more comprehensively capture the hierarchical structure of query semantics, improving the accuracy of semantic understanding by approximately 15% to 25%.

[0017] Second, the semantic-symbolic dual-channel hierarchical index structure designed in this invention organically combines dense semantic retrieval with sparse symbol retrieval. Through a dynamic channel switching mechanism, it achieves the optimal balance between retrieval efficiency and accuracy. Compared with a single-channel retrieval scheme, it reduces the average query latency by about 35% to 45% while ensuring a recall rate of no less than 95%.

[0018] Third, the query intent-aware adaptive retrieval path planning mechanism proposed in this invention can automatically select the optimal retrieval strategy according to the query type, effectively avoiding the efficiency loss caused by fixed retrieval paths. It improves the response speed of fact-based queries by about 50% and the result diversity of exploratory queries by about 30%.

[0019] Fourth, the semantic association explanation generation module based on attention tracing constructed in this invention generates interpretable search result descriptions through cross-layer attention aggregation and semantic association strength matrix, enabling users to understand the basis for search matching and significantly improving the credibility of the system and user satisfaction.

[0020] Fifth, the incremental index maintenance mechanism designed in this invention adopts a write-time copy and block-based incremental reconstruction strategy, which can maintain the stability of query services in scenarios with continuous data inflow, and control the query latency fluctuation during the index update process to within 10%. Attached Figure Description

[0021] Figure 1 This is an overall flowchart of the large-scale data query method that integrates deep semantic representation and interpretable retrieval according to the present invention.

[0022] Figure 2 This is a detailed flowchart of the multi-granularity semantic representation steps of the present invention.

[0023] Figure 3 This is a detailed flowchart of the dual-channel index construction and retrieval steps of the present invention.

[0024] Figure 4 This is a detailed flowchart of the adaptive retrieval path planning steps for query intent awareness in this invention.

[0025] Figure 5 This is a detailed flowchart of the semantic association interpretation generation steps based on attention tracing in this invention.

[0026] Figure 6 This is a detailed flowchart of the incremental index maintenance steps of the present invention. Detailed Implementation

[0027] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0028] This invention provides a method for large-scale data querying that integrates deep semantic representation and interpretable retrieval, such as... Figure 1 As shown, the method includes a multi-granularity semantic representation step, a dual-channel index construction and retrieval step, a query intent-aware adaptive retrieval path planning step, an attention-based semantic association interpretation and generation step, and an incremental index maintenance step. These five steps form a deeply coupled closed-loop collaborative process. The output of the multi-granularity semantic representation step serves as the input to the dual-channel index construction and retrieval step, and the candidate document set from the dual-channel index construction and retrieval step serves as the input to the interpretation and generation step. Feedback information from the interpretation and generation step is used to optimize the parameters of the multi-granularity semantic representation module, and the incremental index maintenance step ensures the continuous availability of the dual-channel index structure.

[0029] Step S1, multi-granular semantic representation steps, such as Figure 2 As shown, the multi-granularity semantic representation step is used to extract hierarchical semantic features from the query text, generating a multi-scale semantic embedding vector that balances local accuracy and global coherence. This invention employs a hierarchical Transformer encoder to achieve feature extraction and fusion at three granular levels.

[0030] In one embodiment of the present invention, the hierarchical Transformer encoder consists of 12 Transformer blocks, with a hidden layer dimension of 768 and 12 attention heads. During the pre-training phase, the encoder employs a large-scale text corpus for self-supervised learning, and during the fine-tuning phase, it uses a retrieval task dataset for supervised training, enabling the encoder to output semantic representations suitable for retrieval tasks.

[0031] Lexical-level feature extraction is achieved through a local attention mechanism. Unlike the global self-attention of the standard Transformer, the local attention mechanism restricts each lexical to only focus on lexicals within its neighborhood window, thereby capturing local semantic boundary features. In one embodiment of the present invention, the attention window size is set to 5, meaning that the attention range of each lexical covers the two adjacent lexicals on its left and right. The calculation process of local attention is as follows: First, the input lexical sequence is positionally encoded, and the lexical embedding vector is added to the positionally encoded vector to obtain the input representation; then, the attention weight of each lexical with the lexicals within the window is calculated. Specifically, the query vector of the lexical is multiplied by the key vectors of each lexical within the window and normalized using the Softmax function to obtain the attention weight; finally, the attention weight is weighted and summed with the value vector of the corresponding lexical to obtain the local attention output of that lexical. The lexical-level local attention weight calculation formula proposed in this invention is as follows:

[0032] ,

[0033] in, as a word element for lexical elements Local attention weights; as a word element The query vector, with dimensions of ; as a word element The key vector, with dimension ; In this embodiment, the dimension of the key vector is... ; as a word element Attention window containing lexical units The lexical index set within its neighborhood. The lexical-level feature vector is obtained by performing multi-layer nonlinear transformations on the local attention output.

[0034] Phrasal-level feature extraction introduces constituent syntactic constraints to achieve semantic unit aggregation. Based on lexical-level features, this invention utilizes a constituent syntactic analyzer to identify phrase boundaries in the query text and aggregates lexical features belonging to the same phrase. In one embodiment, the constituent syntactic analyzer employs a Transformer-based sequence labeling model to predict the phrase type and boundary marker for each lexical. The phrase aggregation process uses an attention pooling mechanism to perform a weighted average of the feature vectors of each lexical within the phrase; the weights are determined by the importance of the lexical in the phrase. The phrase-level feature aggregation formula proposed in this invention is:

[0035] ,

[0036] in, For the first The feature vectors of each phrase have a dimension of 768; For the first The set of lexical indexes contained in each phrase; as a word element The aggregation weights within a phrase are obtained through Softmax normalization. as a word element The word-level feature vectors have a dimension of 768. Aggregate weights. The calculation formula is:

[0037] ,

[0038] in, The weight vector is a learnable vector with a dimension of 768.

[0039] Sentence-level feature extraction is fused with global self-attention to generate context-aware dense vectors. Unlike local attention, global self-attention allows each lexical unit to attend to all other lexical units in the sequence, thereby capturing long-distance semantic dependencies. In one embodiment of the invention, sentence-level feature extraction is achieved through high-level layers (layers 9 to 12) of the Transformer encoder, where the attention mechanism employs standard global self-attention. The sentence-level feature vector is obtained by extracting a special position vector ([CLS] position) from the output of the last Transformer layer, which is then mapped through a multilayer perceptron to form a sentence-level semantic representation.

[0040] The feature vectors at three granularities are dynamically weighted and fused using a learnable gating mechanism. The gating mechanism adaptively adjusts the contribution weights of each granularity feature based on the semantic characteristics of the current query. In one embodiment of this invention, the gating network consists of a fully connected layer and a Softmax activation function. The input is the concatenation of the three granularity feature vectors, and the output is three normalized gating weights. The gating weight calculation formula proposed in this invention is as follows:

[0041] ,

[0042] in, , , The gating weights are for word-level, phrase-level, and sentence-level features, respectively, and the sum of the three is 1. Here is the weight matrix of the gated network, with dimension 1. ; This is the bias vector, with a dimension of 3; The average pooling result of the word-level feature vectors has a dimension of 768. The average pooling result of phrase-level feature vectors has a dimension of 768. The feature vectors are sentence-level features with a dimension of 768. This represents the vector concatenation operation. The formula for calculating multi-scale semantic embedding vectors is:

[0043] ,

[0044] in, The query is a multi-scale semantic embedding vector with a dimension of 768; This is the average value of all word-level feature vectors; This is the average of all phrase-level feature vectors.

[0045] In the experimental evaluation of this invention, the multi-granularity semantic representation module was tested using the MS MARCO and Natural Questions datasets. Experimental results show that, compared to benchmark methods that only use word-level representations, the multi-granularity semantic representation of this invention improves the MRR@10 metric by approximately 18.5% and the Recall@100 metric by approximately 12.3%. This improvement is mainly attributed to the accurate capture of entity boundaries by phrase-level features and the effective modeling of the overall query intent by sentence-level features.

[0046] Step S2, dual-channel index construction and retrieval steps, as follows: Figure 3 As shown, the dual-channel index construction and retrieval steps are used to construct a semantic-symbolic dual-channel hierarchical index structure and dynamically select the index channel for efficient retrieval based on query characteristics. The dual-channel index structure of this invention organically combines dense semantic retrieval with sparse symbolic retrieval, achieving an optimal balance between retrieval efficiency and accuracy through a dynamic channel switching mechanism.

[0047] The semantic channel employs a joint encoding strategy of product quantization and inverted index to encode the dense vector of the document. The core idea of ​​product quantization is to decompose a high-dimensional vector into multiple low-dimensional subspaces, quantizing independently in each subspace, thereby significantly reducing storage overhead and computational complexity while maintaining retrieval accuracy. In one embodiment of this invention, the document dense vector has a dimension of 768, which is divided into 12 equal subspaces, each with a dimension of 64. For each subspace, a codebook containing 256 cluster centers is constructed using the K-means clustering algorithm, with the clustering process completed offline. The quantization process of the document vector involves mapping each subvector to the nearest cluster center in the corresponding codebook, representing the subvector with an index (8-bit integer) of the cluster center. The indices of the 12 subspaces are combined to form the quantized code of the document, with an encoding length of 12 bytes. The approximate calculation formula for the product quantization distance proposed in this invention is as follows:

[0048] ,

[0049] in, For query vector Document Quantization Coding The approximate squared Euclidean distance between them; The number of subspaces is shown in this embodiment. ; For the query vector at the th Subvectors of a subspace, with a dimension of 64; For the first The codebook for each subspace contains 256 cluster center vectors; For the document in the first The quantization index of each subspace, with a value range of 0 to 255; Indicates the first The index in the codebook is The cluster center vector.

[0050] Building upon product quantization, this invention further constructs an inverted index to accelerate coarse-grained candidate selection. The inverted index construction process is as follows: First, coarse quantization is performed on the document vectors, i.e., documents are divided into different coarse-grained clusters using a smaller number of cluster centers (1024 in this embodiment). Then, an inverted chain is constructed for each coarse-grained cluster, storing all document identifiers belonging to that cluster and their complete product quantization codes. During retrieval, the query vector is first compared with the coarsely quantized cluster centers, and the closest clusters are selected (the first 32 clusters in this embodiment). Then, the inverted chains corresponding to these clusters are used for fine sorting using product quantization distance.

[0051] The symbolic channel retains an inverted index of key entity terms in the document for precise matching constraints and candidate set pruning. In one embodiment of the invention, key entity term extraction employs a combination of a named entity recognition model and a keyword extraction algorithm. The named entity recognition model identifies proper nouns such as names of people, places, and organizations in the document, while the keyword extraction algorithm uses a weighted fusion of TF-IDF and TextRank to identify the document's topic keywords. The inverted index structure of the symbolic channel is similar to that of traditional information retrieval systems, with each term corresponding to an inverted chain that stores the document identifier containing the term and the term's position information within the document.

[0052] The dynamic switching between the two channels is determined based on the comparison between the semantic relevance score of the query and a preset threshold. The semantic relevance score is obtained by calculating the cosine similarity between the query vector and the candidate document vector. In one embodiment of the present invention, the semantic relevance threshold is set to 0.75. When the query text contains explicit entity terms and the semantic relevance score is higher than the threshold, the symbolic channel is activated first for exact matching, and then the semantic channel is used to semantically expand the candidate set; when the query text is an open description and the semantic relevance score is lower than the threshold, the semantic channel is dominant for nearest neighbor search, and the symbolic channel is only used for result filtering. The channel selection decision function proposed in this invention is:

[0053] ,

[0054] Wherein, Channel is the selected search channel mode; The semantic relevance score for the query; In this embodiment, a preset semantic relevance threshold is used. ; To determine the number of entity terms identified in the query text; This indicates a cascading pattern where symbols precede semantics; This indicates a semantically driven retrieval pattern; This indicates a dual-channel parallel retrieval mode.

[0055] In the experimental evaluation of this invention, the dual-channel index structure was tested using the BEIR benchmark dataset. Experimental results show that, compared to a single dense retrieval scheme, the dual-channel index of this invention improves the NDCG@10 metric by approximately 8.7% and reduces the average query latency by approximately 42%. This improvement is mainly attributed to the efficient handling of exact matching scenarios by the symbolic channel and the rational allocation of computing resources through the dynamic channel switching mechanism.

[0056] Step S3, query intent-aware adaptive retrieval path planning steps, such as... Figure 4 As shown, the query intent-aware adaptive retrieval path planning step automatically selects the optimal retrieval strategy based on the query type, effectively reducing unnecessary computational overhead. This invention identifies query types using a lightweight intent classifier and plans the retrieval path based on the classification results.

[0057] The intent classifier is designed considering multi-dimensional query features. In one embodiment of the invention, the intent classifier employs a multilayer perceptron structure, with input features including three dimensions: lexical features, syntactic features, and semantic features. Lexical features include one-hot encoding of interrogative word types (such as what, how, why, etc.), the number of entity words in the query, and the proportion of proper nouns; syntactic features include sentence structure type (interrogative sentence, declarative sentence, etc.), clause nesting depth, and the proportion of modifying elements; semantic features include the cosine similarity between the query vector and prototype vectors of three predefined types. The three predefined query types are factual, exploratory, and hybrid.

[0058] Factual queries target specific factual information, typically including concrete entity names and limiting conditions such as time and location, aiming to obtain precise answers. Exploratory queries express open-ended information needs, not limiting the specific answer format, and aiming to retrieve multiple related documents for browsing and exploration. Hybrid queries combine the characteristics of factual and exploratory queries, including certain limiting conditions while allowing for diversity of results. The intent classification confidence calculation formula proposed in this invention is as follows:

[0059] ,

[0060] in, , , These represent the classification probabilities for factual, exploratory, and mixed queries, respectively. This is the weight matrix of the classifier; It is the bias vector; The query feature vector is obtained by concatenating lexical features, syntactic features, and semantic features. Classification confidence is defined as the highest classification probability value.

[0061] ,

[0062] When the classification confidence level is lower than the preset threshold (0.8 in this embodiment), the query is treated as a mixed type.

[0063] Based on the classification results of query types, this invention employs different retrieval strategies. For factual queries, a cascaded retrieval mode of symbol-first, semantic-second approach is adopted: first, precise matching based on entity terms is performed in the symbol channel to quickly obtain a set of candidate documents containing the target entity; then, the candidate documents are sorted by semantic similarity in the semantic channel to filter out semantically irrelevant documents; finally, the top-ranked documents are output as the retrieval results. The advantage of this mode is that it quickly narrows the search scope through precise matching in the symbol channel, avoiding the high computational cost of dense searching on the entire document set.

[0064] For exploratory queries, a semantic-driven parallel recall model is adopted: the semantic channel and the symbolic channel perform retrieval simultaneously. The semantic channel returns the candidate document set with the highest semantic similarity, while the symbolic channel returns the candidate document set containing high-frequency keywords. The results from the two channels are fused and deduplicated using a reciprocal ranking fusion method. The final output is a document set with high diversity. The advantage of this model is that it fully leverages the recall capability of semantic retrieval while utilizing symbolic retrieval to supplement relevant documents that are difficult to cover in the semantic space. The reciprocal ranking fusion formula proposed in this invention is as follows:

[0065] ,

[0066] in, For document The bottom ranking combined with the score; This is a set of ranking lists participating in the fusion, containing ranking results for both semantic and symbolic channels; For document In the ranking list Ranking position in; To smooth the parameters, in this embodiment .

[0067] For hybrid queries, a joint retrieval mode with dynamic weight allocation is adopted: the retrieval weights of the semantic and symbolic channels are dynamically adjusted based on the richness and semantic ambiguity of entity terms in the query; computing resources are allocated to the two channels according to their weights, and the retrieval is performed in parallel; the results are fused and sorted according to weighted similarity scores. The dynamic weight calculation formula proposed in this invention is as follows:

[0068] ,

[0069] in, The retrieval weight for the semantic channel ranges from 0 to 1; the retrieval weight for the symbolic channel is... ; Use the Sigmoid activation function; The parameter vector for weight calculation; For bias; This is the normalized value for the number of entity terms in the query; The semantic ambiguity score of the query is calculated by the distance between the query vector and its nearest neighbor document vector.

[0070] In the experimental evaluation of this invention, the adaptive retrieval path planning mechanism was tested using the MS MARCO and Natural Questions datasets. Experimental results show that, compared to a fixed retrieval path scheme, the adaptive retrieval mechanism of this invention reduces the average latency by approximately 52% for factual queries, improves the diversity index (Recall@1000) by approximately 28% for exploratory queries, and achieves an overall performance improvement of approximately 11.5% in the MRR@10 metric.

[0071] Step S4, the semantic association interpretation generation step based on attention source tracing, such as... Figure 5 As shown, the semantic association explanation generation step based on attention tracing is used to generate interpretability descriptions of search results, helping users understand the basis for search matching. This invention constructs a semantic association strength matrix between query terms and document terms by extracting and aggregating the attention weight distribution during the encoding stage, and generates hierarchical explanatory information based on this matrix.

[0072] Attention weight extraction covers all layers of the hierarchical Transformer encoder. During the forward propagation of the encoder, the self-attention mechanism of each layer calculates the attention weight matrix between words, which reflects the degree of association between words during semantic understanding. In one embodiment of the invention, the encoder has 12 layers, each containing 12 attention heads, thus extracting a total of 144 attention weight matrices. For each attention head, the dimension of the attention weight matrix is... ,in The length of the input sequence.

[0073] Cross-layer attention aggregation is implemented using a gradient-weighted averaging method. The attention weights of different layers contribute differently to the final semantic representation. This invention determines the aggregation weights based on the gradient contribution of each layer to the output. Specifically, during the backpropagation of the encoder, the gradient norm of the loss function with respect to the output of each layer is calculated; a larger gradient norm indicates a greater influence of that layer on the final output. The layer weight calculation formula proposed in this invention is as follows:

[0074] ,

[0075] in, For the first Layer aggregation weights; loss function For the first Layer output The gradient; Let Frobenius norm be represented. The formula for calculating the global attention matrix after cross-layer aggregation is:

[0076] ,

[0077] in, The aggregated global attention matrix has dimensions of . , The sequence length; For the first The average attention matrix of all attention heads in the layer.

[0078] The generation of the semantic association strength matrix involves correlation analysis of the attention distributions of the query sequence and the document sequence. In one embodiment of the invention, the query text and candidate document text are concatenated as encoder input. After obtaining the global attention matrix, the attention weight submatrix between the query word position and the document word position is extracted as the semantic association strength matrix. The element represents the query result. The word and document number The semantic association strength between words.

[0079] The identification of key semantic matching paths is achieved using a graph coloring algorithm. This invention models the semantic association strength matrix as a bipartite graph, with query terms and document terms as nodes on either side, and edges added between word pairs with association strength greater than a threshold. Then, a maximum weight matching algorithm is used to identify key matching paths, which consist of the sequence of word pairs with the highest association strength, representing the main semantic correspondence between the query and the document. The formula for calculating the matching path weight proposed in this invention is as follows:

[0080] ,

[0081] in, The total weight of the matching path; This is the set of word pairs contained in the matching path; Position in the semantic association strength matrix The value; For document words The inverse document frequency is used to emphasize the importance of matching rare words.

[0082] The hierarchical explanation generation strategy provides different levels of detail based on user needs. At the coarse-grained level, it calculates the overall semantic similarity score between candidate documents and the query, and identifies the main matching dimensions (such as topic matching, entity matching, intent matching, etc.). The output format is that the document is highly relevant to your query in terms of [main matching dimensions], and the similarity score is [score value]. At the fine-grained level, it identifies specific matching word pairs based on the semantic association strength matrix and traces the semantic association chain between matching word pairs. The output format is that the query term [query term] and the document term [document term] are highly relevant because [reason for association]. Users can choose to view coarse-grained explanations, fine-grained explanations, or a combination of both through the explanation detail parameter.

[0083] In the experimental evaluation of this invention, 50 users were invited to conduct a subjective evaluation of the explanation generation module. The evaluation results showed that 86% of users believed that the explanation information generated by this invention helped them understand the search results. Compared with the baseline system without explanation, users' trust in the search results increased by approximately 35%, and their satisfaction with the search system increased by approximately 28%.

[0084] Step S5, incremental index maintenance steps, as follows: Figure 6 As shown, the incremental index maintenance step is used to maintain the timeliness of the index and the stability of the query service in scenarios with dynamic data updates. This invention employs a copy-on-write strategy and a block-based incremental reconstruction strategy to achieve low-interference parallelism between index updates and query services.

[0085] The copy-on-write strategy temporarily stores newly added documents in a memory buffer and builds a temporary index. In one embodiment of the invention, the capacity threshold of the memory buffer is set to 10,000 documents. When a new document arrives, its semantic vector is extracted and keywords are extracted first, and then the processing results are written to a temporary data structure in the memory buffer. The temporary index adopts the same dual-channel structure as the main index, but is smaller in size, supporting fast incremental addition operations. Query requests are executed simultaneously on both the main index and the temporary index, and the results of the two indexes are merged, sorted, and returned.

[0086] When the amount of data in the memory buffer reaches the capacity threshold, a background merging thread is triggered to asynchronously merge the temporary index with the main index. The background merging thread runs in an independent worker thread and does not block the foreground query service. The first step of the merging process is to merge the document vectors in the temporary index into the product quantization structure of the main index, which requires updating the coarse quantization clustering assignment and the inverted chain of the inverted file; the second step is to merge the keyword inverted index in the temporary index into the symbolic channel of the main index, which requires updating the term inverted chain and document frequency statistics.

[0087] The block-based progressive reconstruction strategy divides the main index into multiple independent index blocks, and performs a fusion operation on each block sequentially. In one embodiment of the invention, the main index is divided into 16 blocks, each containing a similar number of documents. The fusion process is performed sequentially by block: once a block is fused, it immediately switches to the new version of the index, and subsequent query requests use the updated index data; blocks that have not yet been fused continue to use the original version of the index to respond to query requests. This block switching mechanism ensures the continuity of query services during index updates. The block fusion scheduling strategy proposed in this invention is as follows:

[0088] ,

[0089] in, For the first The point in time when each block begins to merge; This is the start time of the fusion process; The time interval for merging adjacent blocks is dynamically adjusted based on the actual merging time of the previous block to avoid resource contention caused by merging multiple blocks simultaneously.

[0090] Index consistency is ensured through a version number mechanism and read-write locks. Each index block maintains a monotonically increasing version number. Query requests start by recording a snapshot of the version number of each block, and the index data corresponding to the snapshot version is always used during the query process, ensuring the consistency of query results. Write operations are mutually exclusive using write locks, and read operations are concurrent using read locks. The granularity of read-write locks is at the index block level.

[0091] In the experimental evaluation of this invention, a simulated continuous data flow was used to stress test the incremental index maintenance mechanism. The test conditions were 100 new documents per second for 24 hours. The experimental results show that under the scenario of continuous data inflow, the query latency P99 percentile fluctuation of this invention is controlled within 12%, and the index update throughput reaches 85 documents per second, meeting the needs of most practical application scenarios.

[0092] In summary, the large-scale data query method integrating deep semantic representation and interpretable retrieval provided by this invention achieves efficient, accurate, and interpretable large-scale data querying through the deep coupling and synergy of five core steps: multi-granularity semantic representation, dual-channel index structure, adaptive retrieval path planning, attention-based source interpretation generation, and incremental index maintenance. This method outperforms existing technologies in terms of semantic understanding capability, retrieval efficiency, result interpretability, and system stability, demonstrating significant practical value and potential for wider application.

[0093] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, alterations, and equivalent transformations made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A large-scale data query method integrating deep semantic representation and interpretable retrieval, characterized in that, Includes the following steps: The multi-granular semantic representation step involves acquiring the query text, using a hierarchical Transformer encoder to extract features from the query text layer by layer, obtaining word-level feature vectors, phrase-level feature vectors, and sentence-level feature vectors respectively, and dynamically weighting and fusing the word-level feature vectors, phrase-level feature vectors, and sentence-level feature vectors through a learnable gating mechanism to generate multi-scale semantic embedding vectors. The dual-channel index construction and retrieval steps involve receiving the multi-scale semantic embedding vector as a query vector, constructing a semantic-symbolic dual-channel hierarchical index structure, encoding the dense vector of the document in the semantic channel, and retaining the inverted index of key entity terms in the document in the symbolic channel. The retrieval channel is dynamically switched according to the comparison result of the semantic relevance score of the query and the preset semantic relevance threshold, and a preliminary candidate document set is output. The query intent-aware adaptive retrieval path planning step, based on the multi-scale semantic embedding vector and the preliminary candidate document set, identifies the query type of the query text through an intent classifier. For factual queries, a cascading retrieval mode is adopted; for exploratory queries, a parallel recall mode is adopted; and for hybrid queries, a joint retrieval mode is adopted, outputting a candidate document set. The semantic association interpretation generation step based on attention tracing involves extracting the attention weight distribution of each layer of the hierarchical Transformer encoder for the candidate document set and performing cross-layer aggregation to generate a semantic association strength matrix. Based on the semantic association strength matrix, key semantic matching paths are identified to generate interpretability descriptions of the search results. The incremental index maintenance steps involve using a copy-on-write strategy to temporarily store new documents in a memory buffer and build a temporary index. When the amount of data in the memory buffer reaches a preset buffer capacity threshold, a background merging thread is triggered to asynchronously merge the temporary index with the main index. A block-based incremental reconstruction strategy is then used to update the dual-channel hierarchical index structure. In the query intent-aware adaptive retrieval path planning step, the cascading retrieval mode is a symbol-first, semantic-second retrieval mode, the parallel recall mode is a semantic-dominated dual-channel parallel retrieval mode, and the joint retrieval mode is a mode that dynamically allocates the retrieval weights of the semantic channel and the symbol channel for joint retrieval.

2. The large-scale data query method integrating deep semantic representation and interpretable retrieval according to claim 1, characterized in that, In the multi-granularity semantic representation step, the attention window size of the local attention mechanism is 3 to 7 lexical units, the maximum phrase length used for the constituent syntactic constraints is 5 to 10 lexical units, and the sum of the fusion weights output by the learnable gating mechanism is 1.

3. The large-scale data query method integrating deep semantic representation and interpretable retrieval according to claim 1, characterized in that, In the dual-channel index construction and retrieval steps, the product quantization decomposes the high-dimensional dense vector into 8 to 16 low-dimensional subspaces, with each subspace having a codebook size of 256, and the semantic relevance threshold ranges from 0.65 to 0.

85.

4. The large-scale data query method integrating deep semantic representation and interpretable retrieval according to claim 1, characterized in that, In the query intent-aware adaptive retrieval path planning step, the classification confidence threshold of the intent classifier is 0.7 to 0.

9. When the classification confidence is lower than the classification confidence threshold, the joint retrieval mode is adopted.

5. The large-scale data query method integrating deep semantic representation and interpretable retrieval according to claim 1, characterized in that, In the multi-granularity semantic representation step, the learnable gating mechanism includes: The word-level feature vector, phrase-level feature vector, and sentence-level feature vector are respectively input into the gating network, which contains a fully connected layer and a Softmax activation function; The gating network calculates the corresponding gating weights based on the semantic information content of each granularity feature vector, and the gating weights characterize the degree of contribution of each granularity feature vector to the semantic expression of the current query. The feature vectors at each granularity are multiplied by their corresponding gating weights and then summed in a weighted manner to obtain the fused multi-scale semantic embedding vector.

6. The large-scale data query method integrating deep semantic representation and interpretable retrieval according to claim 1, characterized in that, In the dual-channel index construction and retrieval steps, the joint encoding strategy of product quantization and inverted file includes: The dense vector of the document is divided into multiple sub-vectors, and K-means clustering is performed independently on each sub-vector space to obtain the sub-space codebook; Each subvector is quantized into the index of the nearest neighbor cluster center in the corresponding codebook, and the indices of multiple subspaces are combined to form the quantized encoding of the document; An inverted index is constructed based on the prefix of the quantization encoding. Each inverted chain in the inverted index corresponds to a coarse-grained cluster, storing the document identifier belonging to that cluster and its complete quantization encoding.

7. The large-scale data query method integrating deep semantic representation and interpretable retrieval according to claim 1, characterized in that, In the query intent-aware adaptive retrieval path planning step, the query type identification of the intent classifier includes: Extract lexical, syntactic, and semantic features from the query text as classification input; The lexical features include the type of interrogative words, the number of entity words, and the proportion of proper nouns; The syntactic features include sentence structure type, clause nesting depth, and the proportion of modifying elements; The semantic features include the cosine similarity between the query vector and the predefined type prototype vector; Lexical features, syntactic features, and semantic features are concatenated and input into a multilayer perceptron classifier, which outputs the probability distribution of query types.

8. The large-scale data query method integrating deep semantic representation and interpretable retrieval according to claim 1, characterized in that, In the attention-based semantic association interpretation generation step, the cross-layer aggregation includes: Extract the self-attention weight matrix of each layer in the hierarchical Transformer encoder; The self-attention weight matrices of each layer are weighted and averaged along the layer dimension. The layer weights are determined based on the gradient contribution of each layer to the final output. The weighted average attention matrix is ​​normalized to obtain the aggregated global attention distribution.

9. The large-scale data query method integrating deep semantic representation and interpretable retrieval according to claim 1, characterized in that, In the incremental index maintenance step, the block-based incremental reconstruction strategy includes: The main index is divided into multiple independent index blocks, each containing a preset number of document index records; When index merging is triggered, a temporary index merging operation is performed on each block in sequence, and the merged blocks are immediately available for query services. During the block merging process, blocks that have not yet been merged still use the original index data to respond to query requests, while blocks that have been merged use the updated index data to respond to query requests.

Images