A retrieval method, apparatus, device, medium and product

Abstract

The present application relates to the technical field of data retrieval, in particular to a retrieval method, device, equipment, medium and product, the method comprising: generating a query feature vector according to a user's query request; performing retrieval based on a text index structure, an entity index structure and a semantic index structure respectively to obtain different types of semantic units and corresponding relevance scores; performing retrieval through multi-level index structures can obtain comprehensive semantic units; determining the text index weight, the semantic index weight and the entity index weight corresponding to each semantic unit according to the query feature vector; then determining a comprehensive ranking score according to the weights and the relevance scores; and finally screening and outputting the ranked semantic units; the method comprehensively considers multiple factors such as text, entity and semantic, dynamically predicts the weights of multi-dimensional indexes and fuses them in combination with the query feature vector, so as to screen out the final semantic units, can more accurately match the user's query requirements, and greatly improves the accuracy of the retrieval results.

Description

Technical Field

[0001] This invention relates to the field of data retrieval technology, and in particular to a retrieval method, apparatus, device, medium, and product. Background Technology

[0002] Against the backdrop of the rapid development of big data and artificial intelligence, knowledge base retrieval systems have become an indispensable part of information management, knowledge discovery, and decision support in various fields.

[0003] In related technologies, conventional indexing methods, which construct specific index structures such as inverted indexes, can quickly locate preliminary results that may meet search requirements in massive amounts of data, thus satisfying the need for rapid retrieval to a certain extent. However, conventional indexing methods often focus on matching the surface features of the data, resulting in generally low accuracy. Therefore, how to improve retrieval accuracy is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a retrieval method, apparatus, device, medium, and product that can improve retrieval accuracy.

[0005] Firstly, a retrieval method is provided, comprising: generating a query feature vector based on a user's query request; retrieving a first semantic unit and its corresponding first relevance score based on a text index structure based on the query request; retrieving a second semantic unit and its corresponding second relevance score based on an entity index structure based on the query request; retrieving a third semantic unit and its corresponding third relevance score based on the query feature vector and a semantic index structure; determining the text index weight, semantic index weight, and entity index weight corresponding to each semantic unit in the semantic set based on the query feature vector, wherein the semantic set includes the first semantic unit, the second semantic unit, and the third semantic unit; determining a comprehensive ranking score based on the text index weight, semantic index weight, entity index weight, first relevance score, second relevance score, and third relevance score corresponding to each semantic unit; selecting multiple ranked semantic units from the semantic set based on the comprehensive ranking score of each semantic unit; and outputting the multiple ranked semantic units.

[0006] In a preferred embodiment, the present invention may further be configured to include: segmenting text data in a knowledge base to obtain multiple semantic units; generating semantic vectors corresponding to each of the multiple semantic units; constructing a text index structure based on the multiple semantic units; constructing an entity index structure based on the multiple semantic units; and constructing a semantic index structure based on the semantic vectors corresponding to each of the multiple semantic units.

[0007] In a preferred embodiment, the present invention can be further configured to: generate semantic vectors corresponding to each of the multiple semantic units, including: generating semantic vectors corresponding to each of the multiple semantic units; performing principal component analysis on the semantic vectors corresponding to each of the multiple semantic units to obtain the semantic vectors corresponding to each of the multiple semantic units.

[0008] In a preferred embodiment, the present invention can be further configured as follows: constructing a semantic index structure based on the semantic vectors corresponding to each of the plurality of semantic units, including: clustering the semantic vectors corresponding to each of the plurality of semantic units according to a density-based noisy spatial clustering algorithm to obtain a plurality of first clusters; determining the vector mean of the first clusters; using fuzzy C-means clustering based on the vector mean to cluster the semantic vectors corresponding to each of the plurality of semantic units to obtain a plurality of second clusters, wherein the vector mean is the initial cluster centroid of the fuzzy C-means clustering; and assigning topic labels to each second cluster to complete the construction of the semantic index structure.

[0009] In a preferred embodiment, the present invention may further be configured to include: determining the clustering radius of the region of the density-based noisy spatial clustering algorithm based on the basic clustering radius and the semantic density of the region.

[0010] In a preferred embodiment, the present invention can be further configured as follows: assigning topic labels to each second cluster, including: determining the first similarity between the cluster centroid of the target second cluster and the pre-trained word embeddings, wherein the target second cluster is any one of a plurality of second clusters; and using the pre-trained word embeddings with a first similarity greater than a preset similarity threshold as candidate topic labels for the target second cluster; determining the weights and keyword semantic vectors of a plurality of high-weight keywords of the target second cluster; for each candidate topic label of the target second cluster, determining the keyword semantic vectors of a plurality of high-weight keywords and the second similarity between the candidate topic label and the candidate topic label; determining a weighted average similarity for each candidate topic label based on the second similarity and the weights of the plurality of high-weight keywords; for each candidate topic label of the target second cluster, determining a first comprehensive score corresponding to each candidate topic label based on the first similarity, the first centroid weight, the weighted average similarity of the candidate topic label, and the keyword weight; and determining the topic label corresponding to the target second cluster from the candidate topic labels of the target second cluster based on the first comprehensive score corresponding to each candidate topic label.

[0011] In a preferred embodiment, the present invention can be further configured as follows: based on the query feature vector and a semantic index structure, a third semantic unit and its corresponding third relevance score are retrieved, including: determining the third similarity between the query feature vector and the cluster centroid of each second cluster; determining the second comprehensive score of the query feature vector and the topic tags of the second cluster; performing a weighted calculation based on the third similarity, the second centroid weight, the second comprehensive score, and the topic weight to obtain the comprehensive relevance score of each second cluster; determining a retrieval cluster from multiple second clusters based on the comprehensive relevance score of each second cluster; determining the fourth similarity between the semantic vector in the retrieval cluster and the query feature vector; selecting third semantic units from the semantic vectors of the retrieval cluster based on the fourth similarity and the topic tag weight; and using the fourth similarity corresponding to the third semantic unit as the third relevance score corresponding to the third semantic unit.

[0012] In a preferred embodiment, the present invention may be further configured to: determine a second comprehensive score for the query feature vector and the topic tags of the second cluster, including: determining the similarity between the query feature vector and the topic tags of the second cluster; and determining the second comprehensive score based on the similarity between the query feature vector and the topic tags of the second cluster, and the topic tag weights of the topic tags of the second cluster.

[0013] In a preferred embodiment, the present invention may be further configured as follows: after determining the topic label corresponding to the target second cluster from the candidate topic labels of the target second cluster based on the first comprehensive score corresponding to each candidate topic label, the invention further includes: performing normalized weight calculation on the first comprehensive score of the topic label corresponding to the target second cluster to obtain the topic label weight of the topic label corresponding to the target second cluster.

[0014] In a preferred embodiment, the present invention can be further configured as follows: determining the text index weight, semantic index weight, and entity index weight corresponding to each semantic unit in the semantic set based on the query feature vector, including: using a multilayer perceptron based on the query feature vector to determine the first text index weight, first semantic index weight, and first entity index weight corresponding to each semantic unit in the semantic set; and normalizing the first text index weight, first semantic index weight, and first entity index weight to obtain the text index weight, semantic index weight, and entity index weight corresponding to each semantic unit in the semantic set.

[0015] In a preferred embodiment, the present invention can be further configured as follows: the training process of the multilayer perceptron includes: acquiring an initial training dataset, the initial training dataset including labeled samples of training query feature vectors and corresponding index weights, the index weights including second text index weights, second semantic index weights, and second entity index weights of the labeled samples; training an initial multilayer perceptron based on the initial training dataset to obtain a multilayer perceptron. In a preferred embodiment, the present invention can be further configured as follows: the index weights of the labeled samples in the initial training dataset are obtained by adjusting the initial index weights based on user interaction data.

[0016] In a preferred embodiment, the present invention can be further configured as follows: based on the query feature vector, a multilayer perceptron is used to determine the first text index weight, the first semantic index weight, and the first entity index weight corresponding to each semantic unit in the semantic set, including: determining the interest vector for the current time period based on the user's interest vector, interaction data, and time decay weight in the previous time period; and using a multilayer perceptron, based on the query feature vector and the interest vector for the current time period, determining the first text index weight, the first semantic index weight, and the first entity index weight corresponding to each semantic unit in the semantic set.

[0017] In a preferred embodiment, the present invention may be further configured to include: determining whether a preset reconstruction condition has been met; if the preset reconstruction condition has been met, then reconstructing the index structure; wherein the preset reconstruction condition includes any one of the following: the global query hotspot entropy difference exceeds a preset entropy difference threshold; the proportion of modified documents exceeds a preset ratio; the modification rate of high-frequency topics exceeds a preset rate threshold; the query latency exceeds a performance benchmark value.

[0018] In a preferred embodiment, the present invention can be further configured to: reconstruct the index structure, including: reconstructing the index based on a double-buffered index mechanism.

[0019] In a preferred embodiment, the present invention can be further configured to: reconstruct the index structure, including: during the index reconstruction process, in a low-load window, dividing the reconstruction task into multiple independent subtasks to reconstruct the index structure; and in a high-load window, prioritizing the reconstruction of the target index to reconstruct the index structure.

[0020] Secondly, a retrieval device is provided, comprising: a vector generation module for generating a query feature vector based on a user's query request; a retrieval module for retrieving a first semantic unit and its corresponding first relevance score based on a text index structure according to the query request; retrieving a second semantic unit and its corresponding second relevance score based on an entity index structure according to the query request; and retrieving a third semantic unit and its corresponding third relevance score based on the query feature vector and a semantic index structure; a weight determination module for determining the text index weight, semantic index weight, and entity index weight corresponding to each semantic unit in the semantic set based on the query feature vector, wherein the semantic set includes the first semantic unit, the second semantic unit, and the third semantic unit; a comprehensive ranking score determination module for determining a comprehensive ranking score based on the text index weight, semantic index weight, entity index weight, first relevance score, second relevance score, and third relevance score corresponding to each semantic unit; a filtering module for filtering multiple ranked semantic units from the semantic set based on the comprehensive ranking score of each semantic unit; and an output module for outputting the multiple ranked semantic units.

[0021] Thirdly, an electronic device is provided, including a memory for storing a computer program; and a processor for executing the computer program to implement the retrieval method as described in any of the first aspects.

[0022] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the retrieval method as described in any of the first aspects.

[0023] Fifthly, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the retrieval method as described in any of the first aspects.

[0024] In summary, the retrieval method provided by this invention has the following beneficial technical effects: It generates a query feature vector based on the user's query request; it performs retrieval based on text index structure, entity index structure, and semantic index structure respectively, obtaining different types of semantic units and their corresponding relevance scores; through multi-level index structure retrieval, it can comprehensively obtain semantic units; it determines the text index weight, semantic index weight, and entity index weight corresponding to each semantic unit based on the query feature vector; it then determines the comprehensive ranking score based on the weight and relevance score, and finally filters and outputs the ranked semantic units; it comprehensively considers multiple factors such as text, entity, and semantics, and combines the query feature vector to dynamically predict and fuse the weights of multi-dimensional indexes, so as to filter out the final semantic units. Compared with single index retrieval, it can more accurately match the user's query needs and greatly improve the accuracy of retrieval results.

[0025] In addition, the present invention also provides a retrieval device, equipment, medium and product, all of which have the above-mentioned beneficial technical effects. Attached Figure Description

[0026] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating a retrieval method provided in an embodiment of the present invention.

[0028] Figure 2 This is a schematic diagram of dynamic weight fusion provided in an embodiment of the present invention.

[0029] Figure 3 This is a structural diagram of a retrieval method provided in an embodiment of the present invention.

[0030] Figure 4 This is a schematic flowchart of a retrieval device provided in an embodiment of the present invention.

[0031] Figure 5 This is a flowchart illustrating an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0033] It should be noted that, in the optional embodiments of the present invention, the data related to object information, etc., requires the permission or consent of the object when the embodiments of the present invention are applied to specific products or technologies. Furthermore, the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if the embodiments of the present invention involve data related to an object, it must be obtained with the permission and consent of the object, the permission and consent of relevant departments, and in accordance with the relevant laws, regulations, and standards of the country and region. If the embodiments involve personal information, the acquisition of all personal information requires the consent of the individual. If sensitive information is involved, the separate consent of the information subject is required. The embodiments also need to be implemented with the permission and consent of the object.

[0034] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.

[0035] To enable those skilled in the art to better understand the present invention, some technical methods are explained below.

[0036] RAG (Retrieval-Augmented Generation): Retrieval-enhanced generation refers to enhancing the output of generative models (such as language models) by combining data from external knowledge bases, thereby providing more accurate and information-rich results.

[0037] Indexing: The process of structuring data in documents or databases to facilitate fast searching and retrieval. This process typically involves mapping words in a document to document IDs or other identifiers.

[0038] Retrieval: The process of finding information or documents that match a user's query criteria in a knowledge base.

[0039] Inverted Index: An index structure that maps words in a document to a list of documents containing those words, often used for keyword matching.

[0040] Semantics refers to the meaning and usage of words, phrases, or sentences in a language, especially in understanding context and intent.

[0041] Entity: refers to a specific object in a document, such as a person's name, place name, company name, etc.

[0042] Relationship: In a knowledge base, the connection or association between entities, such as "born in" or "works at".

[0043] User Behavior Data: Data such as users' search history, click-through rates, and feedback within the system.

[0044] Hybrid Retrieval Model: A retrieval strategy that combines multiple retrieval methods (such as keyword matching, semantic retrieval, and entity relationship retrieval).

[0045] LSI (Latent Semantic Indexing): This index uses matrix factorization to uncover the latent semantic relationships between documents and queries.

[0046] BERT (Bidirectional Encoder Representations from Transformers): A pre-trained language model that uses deep learning techniques to perform deep semantic understanding of text.

[0047] Word2Vec and GloVe are word vector models used to represent the position of words in semantic space, thereby enabling semantic-based retrieval.

[0048] Against the backdrop of rapid development in big data and artificial intelligence, knowledge base retrieval systems have become an indispensable part of information management, knowledge discovery, and decision support across various fields. Traditional retrieval systems organize and store documents through index building, aiming to respond to user queries quickly and accurately. However, with the continuous expansion of knowledge bases and the diversification of user needs, various indexing and retrieval technologies have emerged.

[0049] 1. Traditional indexing techniques.

[0050] Inverted index: Traditional retrieval systems mostly use inverted indexes, which achieve fast keyword matching retrieval by mapping words in documents to a list of documents. However, when faced with large-scale document collections, the efficiency of building and updating the index becomes a bottleneck.

[0051] LSI (Latent Semantic Index): LSI uses matrix factorization techniques to mine the latent semantic relationships between documents and queries. Although it enhances the semantic understanding of retrieval, it has high computational complexity and is suitable for small datasets.

[0052] 2. Distributed and parallel index construction.

[0053] Distributed search engines like Elasticsearch: These systems employ a distributed architecture, utilizing multiple nodes to build and update indexes in parallel, thus improving the efficiency of index building. However, resource management and load balancing remain challenges when dealing with real-time updates and large-scale knowledge bases.

[0054] MapReduce: The MapReduce framework is widely used for large-scale data processing, including index building. However, this batch processing mode is not suitable for scenarios requiring real-time processing and dynamic updates.

[0055] 3. Semantic-based retrieval technology.

[0056] Word vector models (such as Word2Vec and GloVe): These models achieve semantic-based retrieval by learning the representation of words in a semantic space. However, their ability to understand long text documents is limited, and they rely on further algorithms to process document-level semantics.

[0057] Pre-trained language models such as BERT: The BERT model uses deep learning technology to perform deep semantic understanding of text, which improves the accuracy of retrieval, but it has high computational resource requirements and a long index building time.

[0058] 4. Entity relationships and knowledge graphs.

[0059] Knowledge graphs: Knowledge graphs built using entities and their relationships can enhance the depth and accuracy of retrieval, but existing technologies still face challenges in effectively integrating knowledge graphs into the indexing and querying processes of retrieval systems.

[0060] Entity linking: Entity linking technology is used to identify entities in text and link them to entities in a knowledge base, improving the accuracy of retrieval, but it relies on accurate entity identification and linking algorithms.

[0061] 5. Existing RAG system.

[0062] Traditional RAG systems: These systems generate model outputs by augmenting data from external knowledge bases. They rely mainly on keyword matching during the retrieval process and have limited semantic understanding capabilities.

[0063] Advanced RAG systems: Some systems are beginning to use more advanced technologies, such as neural network-based retrieval and generative models, but there is still room for improvement in indexing large-scale knowledge bases and real-time retrieval.

[0064] 6. Existing advanced indexing technologies.

[0065] Web Search: It utilizes advanced indexing technology, semantic understanding, and user behavior analysis, and is primarily geared towards web search rather than dedicated knowledge base retrieval.

[0066] Microsoft Azure Cognitive Search: The search service provides semantic-based retrieval capabilities, supports real-time index updates and knowledge base integration, but its implementation details and optimization strategies are difficult to understand in detail.

[0067] Elastic Search with Semantic Search combines traditional inverted indexes with modern semantic search technology, but there is still room for optimization in terms of search performance for dynamic updates of large-scale knowledge bases and high-concurrency environments.

[0068] These technologies provide a rich background and reference for this invention, but they still have shortcomings in terms of index building efficiency, retrieval accuracy and speed, adaptability, and dynamic optimization based on user behavior. This invention aims to provide a highly efficient, accurate, adaptable, and scalable knowledge base retrieval system by innovatively combining a multi-level index structure, a real-time index update mechanism, user behavior-based index optimization, and a hybrid retrieval model.

[0069] This invention proposes a high-performance knowledge base retrieval system and its retrieval method, aiming to address the shortcomings of traditional retrieval systems in terms of index building efficiency, retrieval accuracy, real-time performance, and user behavior-driven optimization. Combining various advanced technologies, this invention provides an innovative retrieval framework, with the following specific technical features: 1) Adaptive multi-level index construction.

[0070] The system integrates text inverted index, semantic index, and entity relation index to construct a multi-level index structure, supporting keyword matching, semantic association, and complex relationship queries. The text inverted index employs dynamic updates and a hierarchical B+ tree structure for efficient keyword matching; the semantic index combines improved DBSCAN and fuzzy C-means clustering (FCM), mining deep semantics from text based on BERT encoding, with DBSCAN dynamically adjusting clustering parameters to generate initial clusters, and FCM optimizing multi-topic allocation of segments; the entity relation index is based on a dynamic knowledge graph, supporting complex relationship queries through named entity recognition and dependency parsing. The three indexes dynamically predict weights and fuse them using a multi-layer perceptron (MLP), adjusting the contribution of each index based on query characteristics (such as length and semantic complexity) to ensure a balance between retrieval efficiency and accuracy. This technical feature provides an efficient indexing foundation for dynamic optimization and intelligent retrieval.

[0071] 2) Real-time index optimization based on user behavior.

[0072] The system analyzes user query history, click behavior, and dwell time through Gated Recurrent Units (GRUs) to construct a hierarchical user interest model. Combined with a Multilayer Perceptron (MLP), it dynamically adjusts index weights, prioritizing the preloading of highly relevant documents into the cache. For example, the index weight of frequently queried topics (such as "quantum computing") is increased, and related segments are prioritized. Hierarchical modeling addresses data sparsity issues through global, group, and individual models, quickly adapting to new user needs. This technical feature relies on multi-level indexing to dynamically respond to changes in user behavior, enhancing the personalized search experience.

[0073] 3) Evolution of automatic index structure and time series reconstruction.

[0074] The system dynamically restructures the index (e.g., sharding strategy, semantic clustering granularity) by analyzing retrieval logs and document update frequency to adapt to knowledge base growth and changing query trends. Key metrics include global query hotspot entropy (quantifying changes in query distribution, such as from "neural network" to "quantum computing"), modified document ratio, content update frequency, and query latency. Index restructuring is triggered based on business priorities (e.g., "image processing" has high priority) and metric thresholds. Restructuring employs a double-buffering mechanism: new indexes are built in parallel in the background and updated seamlessly through atomic switching; full restructuring is performed during low-load windows, using a phased strategy to decompose tasks (e.g., updating the "neural network" inverted index first, then re-clustering the "image processing" semantic vectors); and high-frequency or high-priority topics are prioritized for restructuring during high-load periods. This technical feature ensures the index structure adapts to dynamic changes in the knowledge base and optimizes long-term retrieval performance.

[0075] These technical features collectively construct a highly efficient and collaborative system architecture: Adaptive multi-level indexing integrates text inverted indexes, semantic indexes (combined with DBSCAN and FCM), and entity relationship indexes, providing an efficient foundation for keyword matching, semantic association, and complex relationship queries; Real-time index optimization based on user behavior utilizes GRU and MLP to build a hierarchical interest model, dynamically adjusts index weights, and preloads highly relevant documents to achieve personalized retrieval; Automatic index structure evolution dynamically reconstructs the index structure through global query hotspot entropy, business priorities, and a double buffering mechanism, adapting to knowledge base growth and changes in query trends. These features seamlessly connect index construction, real-time optimization, and long-term evolution, significantly improving the retrieval efficiency, accuracy, and personalized user experience of large-scale knowledge bases.

[0076] This invention provides a high-performance knowledge base augmentation generation (RAG) system. Through adaptive multi-level index construction, real-time index optimization based on user behavior, context-aware intelligent retrieval adjustment, automatic index structure evolution and time series reconstruction, and dynamic question-answering optimization based on user feedback, it achieves an efficient, accurate, and dynamically adaptable retrieval and question-answering system that meets user needs.

[0077] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0078] The following sections will describe in detail a retrieval method, apparatus, device, medium, and product provided by embodiments of the present invention.

[0079] This invention provides a retrieval method, such as... Figure 1As shown, the method provided in this embodiment of the invention can be executed by an electronic device, which is a server. This server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet computer, laptop computer, desktop computer, etc., but is not limited to these. The terminal device and electronic device can be directly or indirectly connected via wired or wireless communication. This embodiment of the invention does not impose any limitations on this connection. The method includes: S101, generating a query feature vector based on the user's query request.

[0080] The query request includes keywords and filtering conditions that the user expects to retrieve. The electronic device maps the user's query request to a unified speech space, encodes it into a high-dimensional semantic vector using a pre-trained language model such as BERT or its derivatives, and generates a compact vector representation using optimized semantic extraction techniques, thus obtaining the query feature vector.

[0081] S102. Based on the query request and the text index structure, retrieve the first semantic unit and its corresponding first relevance score; based on the query request and the entity index structure, retrieve the second semantic unit and its corresponding second relevance score; based on the query feature vector and the semantic index structure, retrieve the third semantic unit and its corresponding third relevance score.

[0082] A text index structure is a data structure built from semantic units in a knowledge base. It stores the content of semantic units through preprocessing, feature extraction, and mapping, representing the storage and retrieval architecture of semantic units. The first semantic unit refers to the semantic unit extracted from the knowledge base that corresponds to the query request. The first relevance score is a quantitative value calculated during retrieval, representing the degree of relevance between the first semantic unit and the query request.

[0083] An entity index structure is an index structure built for entity information (such as people, places, etc.) in text. It is a retrieval method that organizes entities and their related descriptions in a knowledge graph manner. The second semantic unit is a semantic unit corresponding to the query request based on the knowledge graph. This semantic unit focuses more on the semantic expression of the entity. The second relevance score is a quantitative indicator of the degree of relevance between the second semantic unit and the query request, calculated when performing a retrieval using the entity index structure.

[0084] A semantic index structure is an index structure built on a semantic level. A third semantic unit is a semantic unit retrieved from the semantic index structure that is semantically related to the query feature vector. The third relevance score is a numerical value used during the retrieval process to evaluate the degree of semantic relevance between the third semantic unit and the query feature vector.

[0085] In this embodiment of the invention, a multi-level index structure is constructed by integrating a text inverted index, a semantic index, and an entity relation index, supporting keyword matching, semantic association, and complex relation queries. The text inverted index can employ a hierarchical B+ tree and a dynamic update mechanism; the semantic index combines an improved DBSCAN to dynamically adjust clustering parameters and fuzzy C-means clustering (FCM) to optimize multi-topic allocation; the entity relation index can be based on a dynamic knowledge graph, supporting complex relation queries through named entity recognition and dependency parsing, achieving efficient and accurate retrieval.

[0086] Understandably, for each index structure, during retrieval based on a query request, a relevance score is calculated for the semantic units in the knowledge base. Then, based on the relevance score of each semantic unit, semantic units corresponding to the index structure are selected from a large number of semantic units. The semantic units retrieved by each index structure may not be exactly the same. For example, a text index structure may retrieve semantic units 1 and 2, an entity index structure may retrieve semantic units 2 and 3, and a semantic index structure may retrieve semantic units 4 and 5. The above is merely an example, and semantic units 1, 2, 3, 4, and 5 are considered as a semantic set.

[0087] The number of semantic units filtered out by various retrieval structures is not limited in this embodiment of the invention, and can be set according to actual needs.

[0088] S103. Based on the query feature vector, determine the text index weight, semantic index weight, and entity index weight corresponding to each semantic unit in the semantic set. The semantic set includes the first semantic unit, the second semantic unit, and the third semantic unit.

[0089] In one feasible approach, a pre-trained index weight model is used to dynamically generate scores for text inverted indexes, semantic indexes, and entity relation indexes through a multilayer perceptron (MLP) by combining query features / query features with interest vectors. The index weights are then adjusted based on these scores, thereby optimizing the retrieval ranking. Compared to traditional fixed-weight retrieval, this system adjusts MLP parameters in real-time through online learning, adapting to changes in user behavior and improving retrieval accuracy and personalization.

[0090] S104. Determine the comprehensive ranking score based on the text index weight, semantic index weight, entity index weight, first relevance score, second relevance score, and third relevance score corresponding to each semantic unit.

[0091] For each semantic unit, a weighted calculation can be performed by combining its index weights and corresponding relevance scores to obtain a comprehensive ranking score for each semantic unit.

[0092] Specifically, the total score corresponding to the overall ranking score: . The relevance score of the text index to the semantic unit document d is calculated based on the enhanced keyword matching algorithm. The semantic index relevance score for document d is calculated based on semantic similarity. The relevance score of the entity relationship index to document d is calculated based on entity association analysis.

[0093] S105. Based on the comprehensive ranking score of each semantic unit, select multiple ranked semantic units from the semantic set.

[0094] The number of sorting semantic units is not limited in this invention and can be customized by the user.

[0095] S106, Output multiple sorting semantic units.

[0096] Multiple semantic units are selected based on the comprehensive ranking score, and then the semantic units are output based on the ranking results.

[0097] As can be seen, in this embodiment of the invention, a query feature vector is generated based on the user's query request; retrieval is performed based on the text index structure, entity index structure, and semantic index structure respectively to obtain different types of semantic units and their corresponding relevance scores. Retrieval through a multi-level index structure can comprehensively obtain semantic units; the text index weight, semantic index weight, and entity index weight corresponding to each semantic unit are determined based on the query feature vector; then, a comprehensive ranking score is determined based on the weights and relevance scores; finally, the ranked semantic units are selected and output. This method comprehensively considers multiple factors such as text, entity, and semantics, and combines them with the query feature vector to dynamically predict and fuse the weights of the multi-dimensional indexes, thereby facilitating the selection of the final semantic units. Compared to single-index retrieval, this method can more accurately match the user's query needs and greatly improve the accuracy of the retrieval results.

[0098] One possible implementation of this invention further includes: segmenting the text data in the knowledge base to obtain multiple semantic units; generating semantic vectors corresponding to each of the multiple semantic units; constructing a text index structure based on the multiple semantic units; constructing an entity index structure based on the multiple semantic units; and constructing a semantic index structure based on the semantic vectors corresponding to each of the multiple semantic units.

[0099] The knowledge base stores multiple text data sets. The text data in the knowledge base is cleaned and segmented to provide high-quality input for subsequent vectorization and index construction. The cleaning process uses regular expressions to remove redundant characters (such as HTML tags and non-standard symbols), uniformly encodes them to UTF-8, and employs language detection and word segmentation techniques to segment the text into semantic units (such as paragraphs or sentences).

[0100] The preprocessed semantic units are mapped to a unified semantic space, providing a foundation for multi-level index construction and retrieval optimization. The preprocessed semantic units are encoded into high-dimensional semantic vectors through a pre-trained language model (such as BERT or its derivatives), and compact vector representations are generated using optimized semantic feature extraction techniques to obtain semantic vectors.

[0101] As can be seen, in this embodiment of the invention, by integrating text index, semantic index and entity relationship index, a multi-level index structure is constructed to support diverse retrieval needs such as keyword matching, semantic association and complex relationship query, thus realizing a complete adaptive multi-level index construction framework.

[0102] This invention provides a possible implementation of a text index structure. In this embodiment, an enhanced inverted index can be used to map keywords in preprocessed semantic units to document identifiers. A context-aware keyword extraction algorithm, combined with a sliding window and attention mechanism, captures composite concepts (e.g., treating "artificial intelligence" as a whole rather than splitting it into "artificial" and "intelligent"), improving keyword retrieval accuracy. The index uses a hierarchical B+ tree structure, maintaining efficient query performance through a dynamic balancing algorithm. This method enhances semantic integrity through an attention mechanism, avoiding the limitations of traditional TF-IDF. The hierarchical B+ tree ensures efficient querying (O(logn) complexity), and incremental updates reduce maintenance overhead, making it suitable for dynamic knowledge base scenarios. Specifically, semantic units are segmented to obtain multiple segments; a scanning window is set, and the segmentation results are traversed using a pre-trained language model (e.g., BERT) to obtain the context embedding vectors of words within the window; attention weights are calculated. Here, hi represents the hidden state of the i-th word. High-weight words are merged into compound concepts (e.g., when α "artificial intelligence" > α "artificial" + α "intelligent"), resulting in a set of keywords for semantic units, which includes both compound and non-compound concepts. Based on the resulting keyword combinations, an enhanced inverted index structure is constructed.

[0103] One possible implementation of this invention involves generating semantic vectors corresponding to multiple semantic units, including: generating semantic vectors corresponding to multiple semantic units; and performing principal component analysis on the semantic vectors corresponding to the multiple semantic units to obtain the semantic vectors corresponding to the multiple semantic units.

[0104] In this embodiment of the invention, in order to optimize storage and retrieval efficiency, the semantic unit system reduces the vector dimension by using dimensionality reduction techniques (such as principal component analysis, PCA) to retain the main semantic information while reducing storage overhead.

[0105] One possible implementation of this invention involves constructing a semantic index structure based on the semantic vectors corresponding to multiple semantic units. This includes: clustering the semantic vectors corresponding to the multiple semantic units using a density-based noisy spatial clustering algorithm (DBSCAN clustering algorithm) to obtain multiple first clusters; determining the vector mean of the first clusters; using fuzzy C-means clustering based on the vector mean to cluster the semantic vectors corresponding to the multiple semantic units to obtain multiple second clusters, where the vector mean is the initial cluster centroid of the fuzzy C-means clustering; and assigning topic labels to each second cluster to complete the construction of the semantic index structure.

[0106] In one feasible approach, an efficient semantic index is constructed by combining the DBSCAN clustering algorithm with fuzzy C-means clustering (FCM), organizing BERT-encoded semantic vectors into a structure that supports fast retrieval.

[0107] Fuzzy C-means clustering (FCM) is used to optimize multi-topic assignment of segments, allowing segments to belong to multiple clusters with different membership degrees, thereby improving semantic coverage. The goal of FCM is: The value is less than the threshold. The number of segments; Number of clusters; For segmented semantic vectors (BERT encoding, where segments are semantic units); It is the cluster centroid; For segmentation Cluster The membership degree reflects the extent to which a segment belongs to a certain cluster. A value between 0 and 1 satisfies ; This is the fuzzing parameter (usually 2).

[0108] The specific steps are as follows: Initialization: Set the number of clusters Randomly initialize the membership matrix Center of mass It is a cluster The semantic center is initially based on the vector mean of the DBSCAN cluster.

[0109] Membership update: . Let be the Euclidean distance from segment i to the centroid of cluster j. The membership degree is calculated by comparing the relative distances of the segments to the centroids of each cluster. The closer a segment is to the centroid of a cluster, the higher its membership degree. Summing the denominators ensures that the membership degrees are normalized. ).index Amplify the differences in distance to balance the distribution of membership degrees. For example, if a segment is closer to the centroid of the "neural network" cluster and farther from the centroid of the "image processing" cluster, its membership degree to the "neural network" cluster will be higher.

[0110] Centroid Update: . The value m represents the membership degree, used as the weight to emphasize the contribution of segments with high membership degrees to the centroid. The cluster centroid is recalculated based on the membership degrees of the segments to more accurately represent the semantic center of the cluster. Segments with high membership degrees have a greater impact on the centroid, and the denominator is normalized to ensure computational stability. The parameter m controls the degree of fuzziness, preventing the centroid from being overly biased towards a single segment.

[0111] Iterative optimization: Repeatedly update membership and centroid until... The change is less than the threshold.

[0112] One possible implementation of this invention involves determining the clustering radius of a region using a density-based noisy spatial clustering algorithm based on the basic clustering radius and the semantic density of the region.

[0113] The improved DBSCAN dynamically adjusts the clustering radius ε to adapt to the density distribution of different regions in the semantic vector space, generating initial clusters with similar semantics. Each cluster represents a group of semantically similar text segments, and the cluster center is used for fast query matching. Traditional DBSCAN relies on a fixed ε and a minimum number of points (minPts), which performs poorly on datasets with uneven semantic density: dense regions may become mixed with different semantics due to excessively large ε, while sparse regions may be ignored due to excessively small ε. In this embodiment, a density-aware mechanism is introduced to dynamically adjust ε. The adjustment formula is: ;in, Based on the threshold, To adjust the parameters, which control the degree of influence of density on radius, optimization is usually achieved through experiments; density is the semantic density, representing the density of vector distribution within a certain region, typically calculated using the average distance between local vectors or density estimation methods. In dense regions (such as popular technical topics), a smaller ε generates fine-grained clusters to avoid semantic clutter; in sparse regions (such as less popular technical documents), a larger ε improves coverage and ensures that no content is missed.

[0114] In summary, this invention combines the complementarity of the improved DBSCAN clustering algorithm and Fuzzy C-means (FCM) soft clustering to construct an efficient semantic index, organizing BERT-encoded semantic vectors into a structure that supports fast retrieval. DBSCAN generates initial clusters by dynamically adjusting the cluster radius ε, automatically determining the number of clusters to adapt to the complex density distribution of the semantic vector space, filtering noise points (such as irrelevant or semantically ambiguous segments), and providing initial cluster number estimation and centroid initialization for FCM, ensuring that FCM starts optimization from a high-quality starting point. Based on the initial clusters from DBSCAN, FCM optimizes the multi-topic allocation of segments, allowing segments to belong to multiple clusters with different membership degrees, capturing fuzzy semantic relationships, and providing support for multi-topic weighted label allocation. DBSCAN's hard clustering generates semantically consistent initial clusters, addressing FCM's sensitivity to cluster number presets and initial conditions; FCM refines the multi-topic allocation through soft clustering, compensating for DBSCAN's inability to handle segmented multi-topic membership. The two work together to achieve a process from coarse-grained clustering to refined multi-topic allocation, improving semantic coverage and retrieval accuracy.

[0115] One possible implementation of this invention involves assigning topic labels to each second cluster, including: SA1, determining the first similarity between the cluster centroid of the target second cluster and the pre-trained word embeddings, wherein the target second cluster is any one of a plurality of second clusters; and using pre-trained word embeddings with a first similarity greater than a preset similarity threshold as candidate topic labels for the target second cluster.

[0116] To capture the semantic diversity of clusters, the system assigns multiple topic tags to each cluster and assigns weights to the topic tags based on the semantic fusion comprehensive score, ensuring that the topic assignments are close to the cluster content.

[0117] First, centroid matching is performed. Specifically, the centroid of each cluster in the second cluster is calculated ( , The semantic vector is segmented within the cluster. The cosine similarity (i.e., the first similarity) between the target second cluster and the pre-trained word embeddings (a set of pre-defined word embeddings, such as "neural network", "deep learning", and "image processing") is used to rank them. Pre-trained word embeddings with similarity scores higher than a preset similarity threshold (e.g., 0.5) are selected as topic embeddings to form candidate topic labels. This target second cluster corresponds to a candidate topic class table and can include at least one topic label. For example, if the cluster centroid of the target second cluster has a similarity of 0.85 with the "neural network" embedding and 0.75 with the "deep learning" embedding, both exceeding the threshold of 0.5, then both are included in the candidate topics.

[0118] SA2. Determine the weights and semantic vectors of the keywords in the second target cluster.

[0119] In this step, keyword weight analysis is performed. Specifically, based on TextRank, high-weight keywords (such as "network" and "convolutional layer") are extracted from multiple semantic vectors of the target second cluster, their weights (i.e., TextRank scores) are calculated, and mapped to pre-trained word embeddings to generate keyword semantic vectors.

[0120] For example, the target second cluster j includes a text set Tj={t1,t2,...,tn}, where each text ti is a semantic unit. A co-occurrence graph is constructed based on the set of semantic units in cluster j; the edge weights in the co-occurrence graph represent the number of co-occurrences. Using TextRank, the score of all words is initialized to 1, and then iteratively updated until the score change is less than a threshold. The final scores are sorted, and a preset number are selected as high-weight keywords. Semantic processing is performed on each keyword to obtain a semantic vector.

[0121] SA3. For each candidate topic tag in the second target cluster, determine the keyword semantic vector of each of the multiple high-weight keywords and the second similarity of the candidate topic tag. Based on the second similarity and the weights of the multiple high-weight keywords, determine the weighted average similarity for each candidate topic tag.

[0122] In this embodiment of the invention, the average cosine similarity is calculated between the extracted high-weight keywords and the candidate topic embeddings (candidate topic tags such as "neural network" and "deep learning") generated by centroid matching. For example, if the high-weight keyword "network" (TextRank weight 0.6) has a similarity of 0.82 with the candidate topic tag "neural network" and 0.68 with the candidate topic tag "deep learning"; and the high-weight keyword "convolutional layer" (TextRank weight 0.4) has a similarity of 0.78 with the candidate topic tag "neural network" and 0.72 with the candidate topic tag "deep learning", then the weighted average similarity is: Candidate topic tag "neural network": Candidate topic tag "deep learning": .

[0123] SA4. For each candidate topic tag in the second target cluster, determine the first comprehensive score corresponding to each candidate topic tag based on the first similarity, the first centroid weight, the weighted average similarity of the candidate topic tags, and the keyword weight.

[0124] This step aims to perform semantic fusion. It combines centroid similarity (first centroid weight). The average similarity between keyword semantic vectors and keyword weights. ), calculate a first comprehensive score for each candidate topic tag: Where C is the cluster centroid, Embed for candidate topics, This is a semantic vector for keywords. For the number of keywords, (Optimize through experiments, for example) ).

[0125] The first overall score for the candidate topic tag "neural network": The first overall score for the candidate topic tag "deep learning": .

[0126] SA5. Based on the first comprehensive score corresponding to each candidate topic tag, determine the topic tag corresponding to the target second cluster from the candidate topic tags of the target second cluster.

[0127] This step aims to retain and weight multiple topics. Topic tags with a composite score above a threshold (e.g., 0.5) are retained.

[0128] One possible implementation of this invention involves retrieving a third semantic unit and its corresponding third relevance score based on a semantic index structure, according to a query feature vector, including: SC1, determining the third similarity between the query feature vector and the cluster centroid of each second cluster.

[0129] During the retrieval process, the system simultaneously recalls multiple relevant clusters using cluster centroids and topic weights to ensure semantic coverage. Based on the query vector, a third similarity value is calculated between the query vector and the cluster centroid of each second cluster. .

[0130] SC2 determines the second comprehensive score of the query feature vector and the topic tags of the second cluster.

[0131] Specifically, the similarity between the query feature vector and the topic tags of the second cluster is determined; based on the similarity between the query feature vector and the topic tags of the second cluster, and the topic tag weights of the topic tags of the second cluster, a second comprehensive score is determined.

[0132] Among them, the second comprehensive score = , The weight of topic tag t.

[0133] Furthermore, after determining the topic label corresponding to the target second cluster from the candidate topic labels of the target second cluster based on the first comprehensive score corresponding to each candidate topic label, the process also includes: calculating the normalized weight of the first comprehensive score of the topic label corresponding to the target second cluster to obtain the topic label weight of the topic label corresponding to the target second cluster.

[0134] After retaining topic labels with a comprehensive score higher than a threshold (e.g., 0.5) in SA5, they can be normalized to weights (summing to 1) to reflect the representativeness of each topic to the cluster. For example, the semantic fusion first comprehensive score for the target second cluster, the "neural network" cluster, is: "neural network" = 0.83, "deep learning" = 0.73, both higher than the threshold of 0.5; the normalized weights are calculated as: total score Therefore, the weights of the "neural network" "Deep Learning" Weight The final topic labels “neural network” (weight 0.55) and “deep learning” (weight 0.45) were assigned, reflecting that “neural network” has a higher semantic relevance.

[0135] SC3. The comprehensive relevance score of each second cluster is obtained by weighting the third similarity, the second centroid weight, the second comprehensive score and the topic weight.

[0136] The formula for the comprehensive relevance score is: ;in, As the second centroid weight, As the topic weight.

[0137] SC4. Based on the comprehensive relevance score of each second cluster, determine the search cluster from multiple second clusters.

[0138] Based on the comprehensive relevance score of each second cluster, the n clusters with the highest scores are selected from multiple second clusters as the retrieval clusters.

[0139] SC5. Determine the fourth similarity between the semantic vector and the query feature vector in the retrieval cluster.

[0140] SC6. Based on the fourth similarity and topic tag weight, select the third semantic unit from the semantic vector of the retrieval cluster.

[0141] Within the retrieval cluster, the cosine similarity between the query vector and the semantic vector is calculated as the fourth similarity. The semantic vectors with the highest fourth similarity are then retrieved and sorted in descending order of fourth similarity. In one possible implementation, semantic vectors with the same score are sorted according to the weight of their topic tags, with higher weights ranking higher. In another possible implementation, after obtaining the fourth similarity, the ranking priority can be adjusted based on the topic tag weight of each semantic vector as a weighting factor (e.g., the "neural network" topic-related segment receives a higher ranking due to a weight of 0.55). The specific method used is not limited in this embodiment; users can set it according to their actual needs.

[0142] SC7. The fourth similarity corresponding to the third semantic unit is used as the third relevance score corresponding to the third semantic unit.

[0143] For example, for the query "neural network optimization in image processing", first, a query feature vector (BERT encoding) is generated, and its cosine similarity (third similarity) with the centroids of each cluster is calculated. Then, a comprehensive relevance score is calculated by combining the topic weights, and one or more clusters with the highest scores (such as the "neural network" cluster and the "image processing" cluster) are selected. The formula for the comprehensive relevance score is: ,in For query vector, As the cluster centroid, For clusters Theme weight, For topic embeddings, for example, suppose the centroid similarity between the "Neural Network" cluster (topic weights: Neural Network 0.55, Deep Learning 0.45) and the query vector is 0.90, and the centroid similarity between the "Image Processing" cluster (topic weights: Image Processing 0.60, Feature Extraction 0.40) and the query vector is 0.85; the query vector has an embedding similarity of 0.80 with "Neural Network", 0.70 with "Deep Learning", 0.75 with "Image Processing", and 0.65 with "Feature Extraction". Then: Score for the "neural network" cluster: Score for the "Image Processing" cluster: .

[0144] Select the top n clusters with the highest scores (e.g., "Neural Networks" and "Image Processing"). Within each selected cluster, calculate the cosine similarity between the query vector and the segment vectors (vectors included within a cluster). Recall the segments with the highest similarity, sort them in descending order of similarity, and adjust the sorting priority using topic weights as a weighting factor (e.g., segments related to the "Neural Networks" topic receive a higher ranking due to a weight of 0.55). If too many segments are recalled, select the top... By segmenting data into segments or filtering low-similarity segments using thresholds, retrieval efficiency and accuracy can be ensured.

[0145] The construction of the entity relationship index is further elaborated. Entities and their relationships are stored based on a dynamic knowledge graph. Combined with BERT Named Entity Recognition (NER) and dependency parsing, complex relationships (such as multi-hop relationships, e.g., generating triples for "artificial intelligence is applied to medical imaging" using the formula (artificial intelligence, applied to, medical imaging)) are extracted. By capturing complex relationships through dependency parsing, the limitations of traditional triple extraction are overcome. An incremental update mechanism reduces maintenance costs, and the inference engine expands knowledge coverage, making it suitable for complex relationship query scenarios.

[0146] It is important to note that, see Figure 2 , Figure 2This is a schematic diagram of dynamic weight fusion provided by an embodiment of the present invention. The present invention can perform dynamic weight fusion. Dynamic weight fusion integrates the retrieval results of text inverted index, semantic index and entity relation index through multilayer perceptron (MLP), dynamically predicts the weight of each index according to query features, generates initial document ranking results, and provides an efficient and stable foundation for subsequent personalized optimization.

[0147] One possible implementation of this invention involves determining the text index weight, semantic index weight, and entity index weight corresponding to each semantic unit in the semantic set based on the query feature vector. This includes: using a multilayer perceptron based on the query feature vector to determine the first text index weight, first semantic index weight, and first entity index weight corresponding to each semantic unit in the semantic set; and normalizing the first text index weight, first semantic index weight, and first entity index weight to obtain the text index weight, semantic index weight, and entity index weight corresponding to each semantic unit in the semantic set.

[0148] In this embodiment of the invention, the user query is encoded using a pre-trained language model (such as BERT) to generate a query feature vector. This includes information such as query length, keyword distribution, and semantic complexity. MLP uses the initial training dataset (containing labeled samples of queries and their corresponding index weights) for supervised learning, constructing a mapping from query features to index weights, and then outputting the unnormalized score for each index. (such as the weight of the first text index) First semantic index weight First entity index weight These correspond to the text inverted index, semantic index, and entity relation index, respectively. The final weights (such as the text index weights) are generated through Softmax normalization. Semantic index weight Entity index weight This is used to merge the scores from various indexes to generate an initial document sorting list.

[0149] in, In the formula: Indicates the first The unnormalized score of each index (text, semantic, entity relation) represents its contribution to the retrieval results. It represents the query feature vector, which encodes the semantic and structural information of the query. This represents the weight matrix of the MLP, which controls the feature mapping. This represents the bias vector of the MLP, used to adjust the output offset. It is initialized to a zero vector. Softmax normalization is applied. The overall ranking score is ; The relevance score of the text inverted index to the semantic unit document d is based on the enhanced keyword matching algorithm. The semantic index is used to calculate the relevance score of document d based on semantic similarity. The relevance score of the entity relation index to document d is based on entity association analysis.

[0150] Furthermore, initially, the MLP is trained through supervised learning, based on an initial dataset containing queries and their corresponding index weight scores. The optimization objective is to optimize the relevance of the retrieval results. Training employs gradient descent, adjusting the weight matrix and bias vector based on a loss function (such as cross-entropy loss).

[0151] Specifically, one possible implementation of this invention, the training process of a multilayer perceptron, includes: obtaining an initial training dataset, which includes labeled samples of training query feature vectors and corresponding index weights, the index weights including the second text index weight, the second semantic index weight, and the second entity index weight of the labeled samples; and training an initial multilayer perceptron based on the initial training dataset to obtain a multilayer perceptron.

[0152] To ensure that the MLP has the ability to predict query features to index weights at startup, a normalization method is used to initialize parameters and supervised learning is employed for training. W is initialized using a normalization method to ensure a balanced weight distribution; the bias vector b is initialized to zero to avoid initial offset, and the weights are normalized. The initial setting is equal distribution (e.g.) This ensures that the text inverted index, semantic index, and entity relation index contribute equally when there is no training data. MLP is trained through supervised learning, based on an initial dataset containing the training query feature vector and the weight scores of the training semantic units of the corresponding indexes. The optimization objective is to optimize the relevance of the retrieval results. Gradient descent is used, and the weight matrix and bias vector are adjusted through a loss function (such as cross-entropy loss).

[0153] Specifically, the index weights of the labeled samples in the initial training dataset are obtained by adjusting the initial index weights based on user interaction data.

[0154] In this embodiment of the invention, the initial training dataset is generated through a hybrid strategy to reduce the subjectivity and cost of manual annotation.

[0155] First, for core query scenarios (such as short queries for "artificial intelligence", long queries for "the latest applications of artificial intelligence in the medical field", and entity relationship queries for "artificial intelligence in healthcare"), experts provide a small amount of high-quality weighted annotations as seed data to label the index weights of multiple semantic units. Second, by utilizing user interaction data (such as queries, clicks, and dwell time in existing retrieval system logs), the weights of clicked documents are increased by analyzing their index sources (e.g., keyword-matched documents increase weights). Entity relationship documents drive up Furthermore, based on the ranking and score of existing system retrieval results, heuristic rules are applied to infer weights (e.g., top results come from a text inverted index and are marked as high). This makes the weighting of the labels more accurate.

[0156] Furthermore, through weakly supervised learning, pseudo-labels are generated based on query features (such as length and number of entities) to expand the sample size. Training samples consist of the query feature vector Q and inferred or labeled weights. The data is paired and cleaned of noisy data through minimal manual verification or relevance metrics to ensure sample quality. The training process includes pre-training an MLP using seed data to learn the core mapping, fine-tuning with pseudo-labeled data to expand diversity, and online verification through subsequent user feedback to correct initial errors. The dataset is derived from general query sets, simulated user interaction data, or existing system logs to ensure coverage of diverse query scenarios and suitability for dynamic knowledge base environments.

[0157] The system generates an initial training dataset using a hybrid strategy, reducing reliance on manual annotation. It leverages user interaction, heuristic rules, and weakly supervised learning to generate high-quality samples, ensuring that the MLP dynamically allocates weights: short queries (e.g., 1-3 words, such as "artificial intelligence") have lower information content, resulting in higher MLP output. Using keyword matching Long queries (e.g., more than 4 words, such as "the latest applications of artificial intelligence in the medical field") have high MLP output due to semantic complexity or entity relationships. or It captures deep semantics or entity relationships. For example, when querying "AI in healthcare," which involves entity relationships, MLP prioritizes it. ,make Leading the effort to uncover documents related to AI medical applications.

[0158] One possible implementation of this invention involves using a multilayer perceptron based on the query feature vector to determine the first text index weight, first semantic index weight, and first entity index weight corresponding to each semantic unit in the semantic set. This includes: determining the interest vector for the current time period based on the user's interest vector, interaction data, and time decay weight from the previous time period; and using a multilayer perceptron based on the query feature vector and the interest vector for the current time period to determine the first text index weight, first semantic index weight, and first entity index weight corresponding to each semantic unit in the semantic set.

[0159] In this embodiment of the invention, by analyzing implicit user behaviors (such as search queries, clicks, and dwell time), a multi-timescale user interest model is constructed, and index weights are dynamically optimized to achieve personalized search ranking. When a user queries a specific topic, the index weights are optimized based on implicit feedback to improve the ranking results, ensuring search accuracy and a personalized experience.

[0160] Among them, user interest vector User preferences are represented by a gated recurrent unit (GRU) that integrates implicit user behaviors (such as search query keywords, clicked link URLs, page dwell time, mouse scrolling paths, and interaction timestamps). This data represents short-term, medium-term, and long-term interaction behaviors, reflecting changes in query intent and interests. Based on interaction data... and time decay weight Update the interest vector using the following formula: ,in, This is the current interest vector. This is the interest vector from the previous time step. To preprocess and encode implicit interaction data into high-dimensional feature vectors, The time decay weights are used for the initial interest vector. Generate based on initial user interaction or general knowledge base features.

[0161] Based on a pre-trained index weight model, the system utilizes interest vectors and query features The scores for text inverted index, semantic index, and entity relation index are continuously generated through a multilayer perceptron (MLP), and the formula is as follows: ,in This is a concatenated vector of query features and interest vectors. and These are the weight matrix and the bias vector, respectively. Based on the generated scores, the index weights are adjusted to optimize search ranking and improve search accuracy.

[0162] Through an online learning mechanism, MLP parameters are continuously optimized based on implicit user feedback, dynamically adapting to changes in user behavior. Interest Vector Real-time updates ensure the system responds quickly to diverse query needs, taking into account both short-term interest changes and long-term preference trends. By deeply integrating user interest vectors and query features, combined with GRU-modeled multi-timescale interest representations, MLP-driven index weight optimization, and an online learning mechanism based on implicit feedback, the system effectively improves retrieval accuracy and personalized recommendation capabilities, efficiently adapting to complex and ever-changing query scenarios and providing users with a high-quality personalized search experience.

[0163] As can be seen, in the embodiments of the present invention, the multi-timescale interest vector generation integrates implicit user behaviors (such as search query keywords, clicked links, page dwell time, mouse scroll trajectory, and interaction timestamps) through a gated recurrent unit (GRU) to generate multi-timescale interest vectors, dynamically capturing users' short-term, medium-term, and long-term intentions. Compared with traditional static user profiles, this significantly improves the accuracy of personalized retrieval.

[0164] Furthermore, the system dynamically optimizes the index structure and monitors performance by analyzing key operational data to adapt to changes in knowledge base growth and query trends.

[0165] One possible implementation of this invention further includes: determining whether a preset reconstruction condition has been met; if the preset reconstruction condition has been met, then reconstructing the index structure; wherein the preset reconstruction condition includes any one of the following: the global query hotspot entropy difference exceeds a preset entropy difference threshold; the proportion of modified documents exceeds a preset ratio; the modification rate of high-frequency topics exceeds a preset rate threshold; the query latency exceeds a performance benchmark value.

[0166] In this embodiment of the invention, the data collection layer uses a distributed log collection framework to collect the following key indicators in each indexing cycle (the time interval is dynamically determined according to system load and data changes): (1) Global query hotspot entropy, which quantifies the distribution change of the query ratio of all topic tags through information entropy, and calculates the difference between the current cycle entropy value and the previous cycle (the formula is...). ,in , , Let be the number of queries for the i-th topic tag. (t represents the total number of queries, and N represents the total number of topics). This metric was chosen because it comprehensively captures the shift of queries to new topic tags (such as from "neural network" to "quantum computing"), adapting to the dynamic number of topics.

[0167] (2) The percentage of modified documents is measured as the proportion of the number of document segments (semantic units of documents) modified in the current period to the total number of segments in the knowledge base (e.g., the percentage of modifications to the “blockchain technology” segment). This indicator is chosen because it standardizes the intensity of content updates, reflects the impact of semantic changes on the index, and is suitable for knowledge bases of different sizes.

[0168] (3) Content update frequency, which is measured by the modification rate of each topic document (for example, "Quantum Computing" is frequently modified and thus high frequency, while "Basic Theory" is modified less frequently and thus low frequency). This indicator is chosen because it distinguishes between rapidly changing content (such as technical hotspots) and stable content (such as basic theory), and guides the differentiated index update strategy.

[0169] (4) Query latency, defined as high percentile response time (in milliseconds), is chosen because it directly reflects index performance bottlenecks and indicates system optimization needs. These metrics are aggregated into a structured data stream and stored in a time-series database, providing a foundation for trend analysis and scheduling.

[0170] Furthermore, the triggering conditions for re-indexing are based on the following logic: assigning priority weights to topics according to business needs (e.g., "image processing" is given high priority due to strict response time requirements; "basic theory" has lower priority due to lower real-time requirements), combined with indicator analysis. If the global query hotspot entropy difference exceeds a preset threshold (e.g., the query distribution shifts from being concentrated on "neural networks" to including "quantum computing"), or the proportion of modified documents exceeds a preset ratio (e.g., the segmented modification ratio of "blockchain technology" is significant), or the content update frequency indicates that the modification rate of high-frequency topics exceeds the threshold, or the query latency exceeds the performance benchmark, then the relevant topics are marked as needing index reconstruction.

[0171] Furthermore, during the index reconstruction process, the inverted index adjusts its sharding strategy based on query hotspots, content update frequency, and business priorities: high-frequency or high-priority topics use small-granularity sharding to improve parallel retrieval, while low-frequency or low-priority topics use large-granularity sharding to optimize storage; the semantic index dynamically optimizes the neighborhood radius based on the semantic vector distribution density (estimated by the average distance of k-nearest neighbors) through adaptive DBSCAN parameter adjustment. High-density areas (such as popular technology topics) use smaller radii to improve clustering accuracy, while low-density areas (such as unpopular topics) use larger radii to ensure coverage, and multi-level semantic indexes are formed through hierarchical clustering.

[0172] As can be seen, in this embodiment of the invention, the index structure (such as sharding strategy and semantic clustering granularity) is dynamically reconstructed by analyzing indicators such as global query hotspot entropy, modified document ratio, content update frequency and query latency.

[0173] One possible implementation of this invention involves reconstructing the index structure, including: index reconstruction based on a double-buffered index mechanism. In this embodiment, the double-buffered index mechanism enables seamless updates by maintaining two index copies (the current index and the new index) in the background: the new index is built in parallel in the background, and upon completion, an atomic switch operation migrates query traffic from the current index to the new index in a very short time, ensuring uninterrupted online service. Simultaneously, the index version history is maintained to support rapid rollback in abnormal situations.

[0174] One possible implementation of this invention is to reconstruct the index structure, which includes: during the index reconstruction process, when the load window is low, dividing the reconstruction task into multiple independent subtasks to reconstruct the index structure; and when the load window is high, prioritizing the reconstruction of the target index to reconstruct the index structure.

[0175] The scheduling algorithm selects low-load time windows (assessed by the fuzzy logic controller based on CPU utilization, memory usage, disk I / O pressure, and network bandwidth, with a low load level) for reconstruction. During high load, it prioritizes the reconstruction of indexes for frequently queried or high-priority topics. Updates to high-priority indexes are prioritized in the scheduler to ensure rapid response to critical business needs and minimize impact on online queries. The scheduling algorithm performs a full reconstruction (inverted index, semantic vector re-clustering, and knowledge graph optimization) during low-load windows, employing a phased strategy to decompose the reconstruction task into multiple independent sub-tasks to reduce resource requirements per operation and ensure system stability. During high load, it prioritizes the reconstruction of target indexes to minimize impact on online queries.

[0176] As can be seen, in this embodiment of the invention, a double buffering mechanism and a phased reconstruction strategy are adopted. Full reconstruction is performed in the low-load window, and high-frequency or high-priority topics are optimized first when the load is high. This ensures that the index structure adapts to the growth of the knowledge base and changes in query trends, and optimizes long-term retrieval performance.

[0177] In this embodiment of the invention, compared with traditional fixed-weight retrieval systems, the present invention has technical advantages in user behavior analysis and dynamic retrieval optimization. In the user behavior analysis stage, the system uses a gated recurrent unit (GRU) to fuse implicit user behaviors (such as search query keywords, clicked links, page dwell time, mouse scrolling trajectory, and interaction timestamps) to generate multi-timescale interest vectors, capturing dynamic user intent and providing a precise foundation for personalized retrieval. The dynamic retrieval optimization module, based on a pre-trained index weight model, utilizes interest vectors and query features to dynamically generate scores for text inverted indexes, semantic indexes, and entity relation indexes through a multilayer perceptron (MLP), and adjusts index weights according to the scores to optimize retrieval ranking. Through continuous online learning to adjust MLP parameters, combined with time-series reconstructed interest vectors, the system drives the automatic evolution of the index structure, adapting in real time to changes in user behavior, improving retrieval accuracy and personalization, and overcoming the problem of insufficient adaptation to diverse queries in traditional systems. This invention, through the collaborative work of GRU and MLP, achieves dynamic evolution of the index structure and efficient retrieval optimization, and has significant industrial application value in the fields of accurate retrieval and personalized question answering in large-scale knowledge bases.

[0178] Based on any of the above embodiments, a knowledge base enhancement generation system based on a hierarchical index structure includes: a user behavior analysis module configured to fuse implicit user behavior through a gated loop unit to generate multi-timescale interest vectors, representing the user's short-term, medium-term, and long-term query intentions; that is, see [link to documentation]. Figure 3 , Figure 3 This is a structural diagram of a retrieval method provided in an embodiment of the present invention, based on the output vector of a user behavior analyzer at the retrieval layer.

[0179] The dynamic retrieval optimization module is configured to be based on a pre-trained index weight model. It uses interest vectors and query features to dynamically generate scores for the text inverted index, semantic index, and entity relationship index through a multilayer perceptron, and adjusts the index weights according to the scores to optimize the retrieval ranking.

[0180] The adaptive multi-level index building module is configured to integrate text inverted index, semantic index, and entity relationship index to build a multi-level index structure, supporting keyword matching, semantic association, and complex relationship queries; that is, it builds a multi-level index structure based on the inverted index builder, semantic index builder, and entity relationship index builder of the index layer.

[0181] The automatic index structure evolution module is configured to dynamically adjust the index structure by analyzing global query hotspot entropy, modifying document proportions, content update frequency, and query latency to adapt to knowledge base growth and changes in query trends. This is essentially the automatic index evolution module at the optimization layer.

[0182] The user behavior analysis module includes: a data collection unit, configured to collect implicit user behaviors, including search query keywords, clicked links, page dwell time, mouse scroll trajectory, and interaction timestamps; an interest vector generation unit, configured to process implicit behaviors through a gated loop unit, integrate time decay weights, and generate multi-time-scale interest vectors, which represent changes in user query intent and preferences over different time spans; and an initialization unit, configured to generate initial interest vectors based on initial user interactions or general knowledge base features, and set time decay weights to emphasize initial data.

[0183] The dynamic retrieval optimization module includes: a score generation unit, configured to dynamically generate scores for the text inverted index, semantic index, and entity relationship index based on the concatenation vector of query features and interest vectors using a multilayer perceptron; a weight adjustment unit, configured to adjust the weights of each index according to the scores, and merge the relevance scores of each index to generate an optimized retrieval ranking; and an online learning unit, configured to continuously adjust the parameters of the multilayer perceptron based on implicit user feedback to adapt to changes in user behavior in real time.

[0184] The adaptive multi-level index building module includes: a text inverted index unit, configured to map keywords to document identifiers through a hierarchical B+ tree structure and dynamic update mechanism, supporting efficient keyword matching; a semantic index unit, configured to combine an improved DBSCAN clustering algorithm and fuzzy C-means clustering to build a semantic index based on semantic vectors encoded by a pre-trained language model, supporting semantic association queries; and an entity relation index unit, configured to extract entities and their relations based on a dynamic knowledge graph through named entity recognition and dependency parsing, supporting complex relation queries.

[0185] The semantic indexing unit further includes: a dynamic clustering subunit, configured to dynamically adjust the clustering radius through an improved DBSCAN algorithm to adapt to the density distribution of the semantic vector space and generate semantically similar initial clusters; a multi-topic assignment subunit, configured to optimize the multi-topic membership of segments through fuzzy C-means clustering, allowing segments to belong to multiple clusters with different membership degrees; and a topic tag assignment subunit, configured to generate multi-topic tags and assign weights based on the cosine similarity between the cluster centroid and the pre-trained word embeddings and keyword weight analysis.

[0186] The automatic index structure evolution module includes: a data analysis unit, configured to collect and analyze global query hotspot entropy, modified document ratio, content update frequency, and query latency to determine the triggering conditions for index reconstruction; an index reconstruction unit, configured to dynamically adjust the index sharding strategy, semantic clustering granularity, and entity relationship weights based on business priorities and triggering conditions; and a double buffering mechanism unit, configured to build new indexes in parallel in the background, achieve seamless updates through atomic switching, and maintain index version history to support abnormal rollback.

[0187] The index reconstruction unit is further configured to: perform full index reconstruction during low-load time windows, decompose reconstruction tasks using a phased strategy, and execute them in order of priority; and prioritize the reconstruction of index portions of high-frequency queries or high-priority topics during high-load periods to reduce the impact on online queries.

[0188] Correspondingly, a knowledge base retrieval method based on a multi-level index structure includes the following steps: collecting implicit user behaviors, including search query keywords, clicked links, page dwell time, mouse scroll trajectory, and interaction timestamps; fusing implicit behaviors through gated recurrent units to generate multi-timescale interest vectors; dynamically generating scores for text inverted index, semantic index, and entity relation index using a multi-layer perceptron based on a pre-trained index weight model and utilizing interest vectors and query features, and adjusting index weights according to the scores to optimize retrieval ranking; constructing a multi-level index structure that integrates text inverted index, semantic index, and entity relation index, supporting keyword matching, semantic association, and complex relationship queries; and dynamically adjusting the index structure by analyzing global query hotspot entropy, modified document ratio, content update frequency, and query latency to adapt to knowledge base growth and changes in query trends.

[0189] The following describes a retrieval device provided by an embodiment of the present invention. The retrieval device described below can be referred to in correspondence with the method described above. The retrieval device of this embodiment is installed in an electronic device. Figure 4 , Figure 4 This is a structural block diagram of a retrieval device according to one embodiment of the present invention, comprising: a vector generation module 210, used to generate a query feature vector based on a user's query request; a retrieval module 220, used to retrieve a first semantic unit and its corresponding first relevance score based on a text index structure based on a query request; retrieve a second semantic unit and its corresponding second relevance score based on an entity index structure based on a query request; retrieve a third semantic unit and its corresponding third relevance score based on a semantic index structure based on the query feature vector; a weight determination module 230, used to determine the text index weight, semantic index weight, and entity index weight corresponding to each semantic unit in the semantic set based on the query feature vector, wherein the semantic set includes a first semantic unit, a second semantic unit, and a third semantic unit; a comprehensive ranking score determination module 240, used to determine a comprehensive ranking score based on the text index weight, semantic index weight, entity index weight, first relevance score, second relevance score, and third relevance score corresponding to each semantic unit; a filtering module 250, used to filter out multiple ranked semantic units from the semantic set based on the comprehensive ranking score of each semantic unit; and an output module 260, used to output multiple ranked semantic units.

[0190] In one possible implementation, the method further includes: a retrieval structure construction module for segmenting text data in the knowledge base to obtain multiple semantic units; generating semantic vectors corresponding to each of the multiple semantic units; constructing a text index structure based on the multiple semantic units; constructing an entity index structure based on the multiple semantic units; and constructing a semantic index structure based on the semantic vectors corresponding to each of the multiple semantic units.

[0191] In one feasible approach, the retrieval structure building module generates semantic vectors corresponding to multiple semantic units based on multiple semantic units, which are used for: generating semantic vectors corresponding to multiple semantic units based on multiple semantic units; and performing principal component analysis on the semantic vectors corresponding to multiple semantic units to obtain the semantic vectors corresponding to multiple semantic units.

[0192] In one feasible approach, the retrieval structure building module constructs a semantic index structure based on the semantic vectors corresponding to multiple semantic units. This structure is used to: cluster the semantic vectors corresponding to multiple semantic units using a density-based noisy spatial clustering algorithm to obtain multiple first clusters; determine the vector mean of the first clusters; cluster the semantic vectors corresponding to multiple semantic units using fuzzy C-means clustering based on the vector mean to obtain multiple second clusters, where the vector mean is the initial cluster centroid of the fuzzy C-means clustering; and assign topic labels to each second cluster to complete the construction of the semantic index structure.

[0193] In one feasible approach, the retrieval structure building module is also used to: determine the cluster radius of a region using a density-based noisy spatial clustering algorithm based on the basic cluster radius and the semantic density of the region.

[0194] In one feasible approach, the retrieval structure building module assigns topic tags to each second cluster for: determining the first similarity between the cluster centroid of the target second cluster and the pre-trained word embeddings, wherein the target second cluster is any one of multiple second clusters; and using pre-trained word embeddings with a first similarity greater than a preset similarity threshold as candidate topic tags for the target second cluster; determining the weights and keyword semantic vectors of multiple high-weight keywords for the target second cluster; for each candidate topic tag of the target second cluster, determining the keyword semantic vectors of multiple high-weight keywords and the second similarity between the candidate topic tag and the candidate topic tag; and determining a weighted average similarity for each candidate topic tag based on the second similarity and the weights of the multiple high-weight keywords; for each candidate topic tag of the target second cluster, determining a first comprehensive score corresponding to each candidate topic tag based on the first similarity, the first centroid weight, the weighted average similarity of the candidate topic tags, and the keyword weights; and determining the topic tag corresponding to the target second cluster from the candidate topic tags of the target second cluster based on the first comprehensive score corresponding to each candidate topic tag.

[0195] In one feasible approach, the retrieval module 220 retrieves a third semantic unit and its corresponding third relevance score based on the query feature vector and a semantic index structure. This is used to: determine the third similarity between the query feature vector and the cluster centroid of each second cluster; determine the second comprehensive score of the query feature vector and the topic tags of the second cluster; perform a weighted calculation based on the third similarity, the second centroid weight, the second comprehensive score, and the topic weight to obtain a comprehensive relevance score for each second cluster; determine a retrieval cluster from multiple second clusters based on the comprehensive relevance score of each second cluster; determine the fourth similarity between the semantic vector in the retrieval cluster and the query feature vector; filter out third semantic units from the semantic vectors of the retrieval cluster based on the fourth similarity and the topic tag weight; and use the fourth similarity corresponding to the third semantic unit as the third relevance score corresponding to the third semantic unit.

[0196] In one possible implementation, the retrieval module 220 determines a second comprehensive score for the query feature vector and the topic tags of the second cluster, which is used to: determine the similarity between the query feature vector and the topic tags of the second cluster; and determine the second comprehensive score based on the similarity between the query feature vector and the topic tags of the second cluster, as well as the topic tag weights of the topic tags of the second cluster.

[0197] In one feasible approach, the retrieval structure building module is also used to: calculate the normalized weights of the first comprehensive scores of the topic tags corresponding to the second target cluster, thereby obtaining the topic tag weights of the topic tags corresponding to the second target cluster.

[0198] In one feasible approach, the weight determination module 230, based on the query feature vector, determines the text index weight, semantic index weight, and entity index weight corresponding to each semantic unit in the semantic set. This is done by: using a multilayer perceptron based on the query feature vector to determine the first text index weight, first semantic index weight, and first entity index weight corresponding to each semantic unit in the semantic set; and normalizing the first text index weight, first semantic index weight, and first entity index weight to obtain the text index weight, semantic index weight, and entity index weight corresponding to each semantic unit in the semantic set.

[0199] In one possible implementation, the module further includes: a multilayer perceptron training module for obtaining an initial training dataset, which includes labeled samples of training query feature vectors and corresponding index weights, wherein the index weights include the second text index weight, the second semantic index weight, and the second entity index weight of the labeled samples; and training the initial multilayer perceptron based on the initial training dataset to obtain the multilayer perceptron.

[0200] In one feasible approach, the index weights of the labeled samples in the initial training dataset are obtained by adjusting the initial index weights based on user interaction data.

[0201] In one possible implementation, the weight determination module 230 determines the first text index weight, first semantic index weight, and first entity index weight corresponding to each semantic unit in the semantic set using a multilayer perceptron based on the query feature vector. This is used to: determine the interest vector for the current time period based on the user's interest vector, interaction data, and time decay weight from the previous time period; and determine the first text index weight, first semantic index weight, and first entity index weight corresponding to each semantic unit in the semantic set using a multilayer perceptron based on the query feature vector and the interest vector for the current time period.

[0202] In one possible implementation, the system further includes: a reconstruction module for determining whether preset reconstruction conditions have been met; if the preset reconstruction conditions have been met, then the index structure is reconstructed; wherein the preset reconstruction conditions include any of the following: the global query hotspot entropy difference exceeds a preset entropy difference threshold; the proportion of modified documents exceeds a preset ratio; the modification rate of high-frequency topics exceeds a preset rate threshold; the query latency exceeds a performance benchmark value.

[0203] In one feasible approach, a reconstruction module is used to: reconstruct the index based on a double-buffered index mechanism.

[0204] In one feasible approach, the refactoring module is used to: during the index refactoring process, divide the refactoring task into multiple independent subtasks to refactor the index structure during a low-load window; and during a high-load window, prioritize the refactoring of the target index to refactor the index structure.

[0205] Figure 5 A structural diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 5 As shown, the electronic device includes: a memory 60 for storing a computer program; and a processor 61 for executing the computer program to implement the steps of the method as described in the above embodiments.

[0206] The electronic devices provided in this embodiment may include, but are not limited to, smartphones, tablets, laptops, or desktop computers.

[0207] The processor 61 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 61 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 61 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 61 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0208] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 60 is used to store at least the following computer program 601, which, after being loaded and executed by the processor 61, is capable of implementing the relevant steps of the method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. The operating system 602 may include Windows, Unix, Linux, etc.

[0209] In some embodiments, the electronic device may further include a display screen 62, an input / output interface 63, a communication interface 64, a power supply 65, and a communication bus 66.

[0210] Those skilled in the art will understand that Figure 5 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.

[0211] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, magnetic disks, or optical disks, and other media capable of storing program code.

[0212] Based on this, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described above.

[0213] Based on this, embodiments of the present invention also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-described method.

[0214] The foregoing has provided a detailed description of a retrieval method, apparatus, device, medium, and product provided by embodiments of the present invention. The various embodiments are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0215] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0216] The foregoing has provided a detailed description of the retrieval method, apparatus, device, medium, and product provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A retrieval method, characterized in that, include: Generate query feature vectors based on the user's query request; Based on the query request and the text index structure, the first semantic unit and its corresponding first relevance score are retrieved. Based on the query request and the entity index structure, the second semantic unit and its corresponding second relevance score are retrieved. Based on the query feature vector and the semantic index structure, the third semantic unit and its corresponding third relevance score are retrieved. Based on the user's interest vector, interaction data, and time decay weight from the previous time period, the interest vector for the current time period is determined; based on the query feature vector and the interest vector for the current time period, a multilayer perceptron is used to determine the first text index weight, the first semantic index weight, and the first entity index weight corresponding to each semantic unit in the semantic set. The first text index weight, the first semantic index weight, and the first entity index weight are normalized to obtain the text index weight, semantic index weight, and entity index weight corresponding to each semantic unit in the semantic set. The semantic set includes the first semantic unit, the second semantic unit, and the third semantic unit. The comprehensive ranking score is determined based on the text index weight, semantic index weight, entity index weight, first relevance score, second relevance score, and third relevance score corresponding to each semantic unit. Based on the comprehensive ranking score of each semantic unit, multiple ranked semantic units are selected from the semantic set; Output the plurality of sorting semantic units; The method further includes: clustering the semantic vectors corresponding to multiple semantic units obtained by segmenting text data in a knowledge base using a density-based noisy spatial clustering algorithm to obtain multiple first clusters; determining the vector mean of the first clusters; using fuzzy C-means clustering based on the vector mean to cluster the semantic vectors corresponding to the multiple semantic units to obtain multiple second clusters, wherein the vector mean is the initial cluster centroid of the fuzzy C-means clustering; determining the first similarity between the cluster centroid of the target second cluster and the pre-trained word embedding, wherein the target second cluster is any one of the multiple second clusters; and using pre-trained word embeddings with a first similarity greater than a preset similarity threshold as candidate topic tags for the target second cluster; and determining multiple high-weight keywords for the target second cluster. The semantic index is constructed by determining the weights and semantic vectors of each keyword; for each candidate topic tag in the target second cluster, the semantic vectors of multiple high-weight keywords and the second similarity of the candidate topic tags are determined; based on the second similarity and the weights of the multiple high-weight keywords, the weighted average similarity of each candidate topic tag is determined; for each candidate topic tag in the target second cluster, the first comprehensive score is determined based on the first similarity, the first centroid weight, the weighted average similarity of the candidate topic tags, and the keyword weight; based on the first comprehensive score of each candidate topic tag, the topic tag corresponding to the target second cluster is determined from the candidate topic tags of the target second cluster to complete the construction of the semantic index structure. Accordingly, based on the query feature vector and the semantic index structure, the third semantic unit and its corresponding third relevance score are retrieved, including: determining the third similarity between the query feature vector and the cluster centroid of each second cluster; determining the second comprehensive score of the query feature vector and the topic tags of the second cluster; calculating the comprehensive relevance score of each second cluster by weighting the third similarity, the second centroid weight, the second comprehensive score, and the topic weight; determining the retrieval cluster from multiple second clusters based on the comprehensive relevance score of each second cluster; determining the fourth similarity between the semantic vector in the retrieval cluster and the query feature vector; selecting the third semantic unit from the semantic vector of the retrieval cluster based on the fourth similarity and the topic tag weight; and using the fourth similarity corresponding to the third semantic unit as the third relevance score corresponding to the third semantic unit.

2. The retrieval method according to claim 1, characterized in that, Also includes: The text data in the knowledge base is segmented to obtain multiple semantic units; based on the multiple semantic units, a semantic vector corresponding to each of the multiple semantic units is generated; Construct a text index structure based on the multiple semantic units described above; An entity index structure is constructed based on the multiple semantic units.

3. The retrieval method according to claim 2, characterized in that, Based on the multiple semantic units, generate semantic vectors corresponding to each of the multiple semantic units, including: Based on the multiple semantic units, generate semantic vectors corresponding to each of the multiple semantic units; Principal component analysis is performed on the semantic vectors corresponding to each of the multiple semantic units to obtain the semantic vectors corresponding to each of the multiple semantic units.

4. The retrieval method according to claim 1, characterized in that, Also includes: The clustering radius of the region in the density-based noisy spatial clustering algorithm is determined based on the basic clustering radius and the semantic density of the region.

5. The retrieval method according to claim 1, characterized in that, The second comprehensive score for determining the query feature vector and the topic tags of the second cluster includes: Determine the similarity between the query feature vector and the topic tags of the second cluster; The second comprehensive score is determined based on the similarity between the query feature vector and the topic tags of the second cluster, as well as the topic tag weights of the topic tags of the second cluster.

6. The retrieval method according to claim 5, characterized in that, After determining the topic tags corresponding to the target second cluster from the candidate topic tags based on the first comprehensive score corresponding to each candidate topic tag, the process also includes: The first comprehensive score of the topic tags corresponding to the second cluster of the target is normalized and weighted to obtain the topic tag weight of the topic tags corresponding to the second cluster of the target.

7. The retrieval method according to claim 1, characterized in that, The training process of the multilayer perceptron includes: Obtain an initial training dataset, which includes labeled samples of training query feature vectors and corresponding index weights, wherein the index weights include the second text index weight, the second semantic index weight, and the second entity index weight of the labeled samples; The initial multilayer perceptron is trained based on the initial training dataset to obtain the multilayer perceptron.

8. The retrieval method according to claim 7, characterized in that, The index weights of the labeled samples in the initial training dataset are obtained by adjusting the initial index weights based on user interaction data.

9. The retrieval method according to claim 1, characterized in that, Also includes: Determine whether the preset conditions for reconstruction have been met; If the preset conditions for reconstruction are met, then the index structure will be reconstructed. The pre-defined conditions for reconstruction include any of the following: the entropy difference of global query hotspots exceeds the pre-defined entropy difference threshold; the proportion of modified documents exceeds the pre-defined proportion; the modification rate of high-frequency topics exceeds the pre-defined rate threshold; and the query latency exceeds the performance benchmark value.

10. The retrieval method according to claim 9, characterized in that, Restructuring the index structure includes: Index reconstruction is performed based on a double-buffered index mechanism.

11. The retrieval method according to claim 9, characterized in that, Restructuring the index structure includes: During index reconstruction, in a low-load window, the reconstruction task is divided into multiple independent subtasks to reconstruct the index structure; in a high-load window, the target index is reconstructed first to reconstruct the index structure.

12. A retrieval device, characterized in that, include: The vector generation module is used to generate query feature vectors based on the user's query request; The retrieval module is used to retrieve the first semantic unit and its corresponding first relevance score based on the text index structure according to the query request. Based on the query request and the entity index structure, the second semantic unit and its corresponding second relevance score are retrieved; based on the query feature vector and the semantic index structure, the third semantic unit and its corresponding third relevance score are retrieved. The weight determination module is used to determine the text index weight, semantic index weight and entity index weight corresponding to each semantic unit in the semantic set based on the query feature vector. The semantic set includes a first semantic unit, a second semantic unit and a third semantic unit. The comprehensive ranking score determination module is used to determine the comprehensive ranking score based on the text index weight, semantic index weight, entity index weight, first relevance score, second relevance score, and third relevance score corresponding to each semantic unit. The filtering module is used to filter out multiple ranked semantic units from the semantic set based on the comprehensive ranking score of each semantic unit. The output module is used to output the plurality of sorting semantic units; The system further includes: a retrieval structure construction module, used to cluster the semantic vectors corresponding to multiple semantic units obtained by segmenting text data in a knowledge base using a density-based noisy spatial clustering algorithm to obtain multiple first clusters; determine the vector mean of the first clusters; use fuzzy C-means clustering based on the vector mean to cluster the semantic vectors corresponding to the multiple semantic units to obtain multiple second clusters, wherein the vector mean is the initial cluster centroid of the fuzzy C-means clustering; determine the first similarity between the cluster centroid of the target second cluster and the pre-trained word embedding, wherein the target second cluster is any one of the multiple second clusters; and use pre-trained word embeddings with a first similarity greater than a preset similarity threshold as candidate topic tags for the target second cluster; and determine the target second cluster. The system first determines the weights and semantic vectors of multiple high-weight keywords; for each candidate topic tag in the target second cluster, it determines the semantic vectors of multiple high-weight keywords and the second similarity of the candidate topic tag; based on the second similarity and the weights of multiple high-weight keywords, it determines the weighted average similarity for each candidate topic tag; for each candidate topic tag in the target second cluster, it determines the first comprehensive score for each candidate topic tag based on the first similarity, the first centroid weight, the weighted average similarity of the candidate topic tags, and the keyword weight; based on the first comprehensive score for each candidate topic tag, it determines the topic tag corresponding to the target second cluster from the candidate topic tags of the target second cluster, thus completing the construction of the semantic index structure. The retrieval module is used to: determine the third similarity between the query feature vector and the cluster centroid of each second cluster; determine the second comprehensive score of the query feature vector and the topic tags of the second cluster; perform a weighted calculation based on the third similarity, the second centroid weight, the second comprehensive score, and the topic weight to obtain the comprehensive relevance score of each second cluster; determine the retrieval cluster from multiple second clusters based on the comprehensive relevance score of each second cluster; determine the fourth similarity between the semantic vector in the retrieval cluster and the query feature vector; filter out third semantic units from the semantic vector of the retrieval cluster based on the fourth similarity and the topic tag weight; and use the fourth similarity corresponding to the third semantic unit as the third relevance score corresponding to the third semantic unit. The weight determination module is used to determine the interest vector for the current time period based on the user's interest vector, interaction data, and time decay weight from the previous time period; based on the query feature vector and the interest vector for the current time period, it uses a multilayer perceptron to determine the first text index weight, the first semantic index weight, and the first entity index weight corresponding to each semantic unit in the semantic set; and it performs normalization processing on the first text index weight, the first semantic index weight, and the first entity index weight to obtain the text index weight, the semantic index weight, and the entity index weight corresponding to each semantic unit in the semantic set.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the retrieval method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the retrieval method as described in any one of claims 1 to 11.

15. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the retrieval method according to any one of claims 1 to 11.

Images