Large-scale semantic search optimization method based on deep learning
By constructing a multi-level index through deep learning and topology analysis, the problem of balancing semantic understanding and efficiency in traditional semantic retrieval methods is solved, achieving high-precision, high-efficiency, and scalable large-scale semantic retrieval.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GANSU COMM IND SERVICE CO LTD
- Filing Date
- 2025-12-22
- Publication Date
- 2026-06-16
AI Technical Summary
Traditional semantic retrieval methods struggle to effectively capture semantic relationships between texts. Deep learning models suffer from the curse of dimensionality in large-scale semantic vector retrieval, and existing index structures struggle to balance retrieval efficiency and accuracy in high-dimensional spaces.
By converting text data into semantic vectors through deep learning models, a topology-aware hierarchical navigable small world graph index is constructed. Combined with multi-level structure and dynamic optimization, efficient semantic retrieval is achieved.
It significantly improves retrieval accuracy by 15%–20%, increases query efficiency by 30%–50%, reduces resource consumption by 20%–25%, enhances adaptability, and greatly improves scalability.
Smart Images
Figure CN121765032B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information retrieval technology, and in particular to a large-scale semantic retrieval optimization method based on deep learning, applicable to application scenarios such as search engines, recommendation systems, knowledge graph question answering, and text analysis. Background Technology
[0002] With the explosive growth of the internet and digital information, traditional keyword-based retrieval methods are no longer sufficient to meet users' needs for deep semantic understanding. Existing semantic retrieval technologies mainly suffer from the following problems: On the one hand, traditional inverted indexes struggle to effectively capture semantic relationships between texts, resulting in a semantic gap between search results and user intent; on the other hand, although deep learning models can generate high-quality semantic representations, in large-scale semantic vector retrieval, commonly used structures such as KD-trees and LSH (Locality Sensitive Hash) face the curse of dimensionality in high-dimensional spaces, causing retrieval efficiency to drop sharply as the data scale increases.
[0003] In recent years, the HNSW (Hierarchical Navigable Small World) index structure has shown good performance in the field of near nearest neighbor search, but its standard implementation still has many limitations: First, the traditional HNSW uses simple Euclidean distance or cosine distance, which cannot fully express complex semantic relationships; second, the fixed index structure cannot adapt to dynamically changing query patterns; and third, it is difficult to balance retrieval accuracy and efficiency when the semantic space density is uneven.
[0004] Therefore, how to combine deep learning technology with efficient index structures to build an optimized method suitable for large-scale semantic retrieval is a technical problem that urgently needs to be solved. Summary of the Invention
[0005] The purpose of this invention is to provide a large-scale semantic retrieval optimization method based on deep learning, which aims to solve the problem that it is difficult to achieve both deep semantic understanding and efficient retrieval in the existing technology, and to realize high accuracy, high efficiency and scalability of semantic retrieval.
[0006] This invention proposes a large-scale semantic retrieval optimization method based on deep learning, including:
[0007] The text data to be retrieved is obtained, and the text data to be retrieved is converted into a semantic vector representation through a deep learning model;
[0008] Perform topological structure analysis on the semantic vector representation to obtain topological characteristic information of the semantic space;
[0009] Based on the aforementioned topological characteristic information, a topology-aware hierarchical navigable small-world graph index with a multi-level structure is constructed, wherein each level contains a set of nodes and their connection relationships.
[0010] Based on actual query patterns and system performance feedback, the topology-aware hierarchical navigable small world graph index is dynamically optimized and adjusted.
[0011] Receive a query request, convert the query request into a query vector, perform a multi-level navigation search in the topology-aware hierarchical navigable small world graph index, and return retrieval results that are semantically related to the query vector.
[0012] Preferably, the step of converting the text data to be retrieved into a semantic vector representation using a deep learning model specifically includes:
[0013] The text data to be retrieved is encoded using a pre-trained language model to obtain a context-sensitive representation of the text;
[0014] The context-sensitive representation is integrated into a fixed-dimensional semantic vector through weighted averaging, special label extraction, or attention pooling mechanisms.
[0015] The semantic vectors are normalized, and their dimensions are adjusted or features are selected as needed to balance representational power and computational efficiency.
[0016] Preferably, the step of performing topological structure analysis on the semantic vector representation specifically includes:
[0017] Calculate the local density distribution and global density gradient in the semantic space to identify high-density and sparse regions;
[0018] Analyze the dimension, curvature properties, and connectivity of the semantic space to determine its intrinsic structure;
[0019] Hierarchical clustering and density clustering methods are used to identify naturally formed semantic clusters and boundary points in the semantic space;
[0020] Identify key points, skeletal structures, and bottleneck regions of semantic flow in the semantic space to form a topological descriptor.
[0021] As a preferred embodiment, the steps for constructing a topology-aware hierarchical navigable small world graph index with a multi-level structure specifically include:
[0022] Build a complete underlying index, including all semantic vectors and their initial connections;
[0023] Based on the importance index of nodes, select some nodes to be promoted to the upper level, forming a decreasing hierarchical structure;
[0024] Within each layer, the connection relationships between nodes are determined based on topological characteristics.
[0025] Establish inter-layer connections to ensure that queries can efficiently navigate from the top layer to the target node at the bottom layer.
[0026] Preferably, determining the connection relationships between nodes includes:
[0027] Basic nearest neighbor connections enable each node to establish basic connections with its K nearest neighbors.
[0028] Topology-aware connectivity increases the number of connections for nodes located on the semantic skeleton and increases connection density in semantic bottleneck regions.
[0029] Density-adaptive connections make the number of connections proportional to the local density, allowing high-density areas to receive more connections to improve retrieval accuracy.
[0030] Long-distance shortcut connections establish direct connections between semantic regions over long distances, optimizing global navigation efficiency.
[0031] Preferably, the step of dynamically optimizing and adjusting the topology-aware hierarchical navigable small world graph index specifically includes:
[0032] Monitor system performance metrics, including query path length, node access frequency, query response time, and resource usage;
[0033] Based on the aforementioned performance metrics, identify hotspot paths and inefficient areas that require optimization;
[0034] Add direct connections or increase connection weights for hotspot paths, and simplify the connection structure for low-frequency access areas;
[0035] Regularly assess the overall topology quality of the index and perform a global structure rebalancing when necessary.
[0036] Preferably, the dynamic optimization adjustment also includes an incremental update mechanism:
[0037] For newly added vectors, the optimal entry point is selected based on topological characteristics, their hierarchical distribution is determined, and topology-aware connections are constructed.
[0038] For deletion operations, the connections of the deleted nodes are reconstructed, the affected navigation paths are updated, and topological integrity is maintained.
[0039] For vector updates, identify vectors with significant semantic changes, reconstruct only the affected parts of the structure, and aggregate multiple small updates using a batch processing approach.
[0040] Preferably, the step of performing multi-level navigation search in the topology-aware hierarchical navigable small world graph index specifically includes:
[0041] Start from the top-level entry point, or choose an optimized entry point based on the query context;
[0042] Find the node in the current layer that is closest to the query vector, and then move to that node;
[0043] Explore the neighboring nodes along the connections of the current node to find the node that is closer to the query vector;
[0044] When no closer node can be found in the current layer, descend to the next layer to continue the search;
[0045] An extended search is performed at the bottom layer to ensure the accuracy of the results, and the candidate results are finally verified and sorted.
[0046] Preferably, the implementation of the method includes the following parameter configuration:
[0047] The number of index levels is proportional to the logarithm of the total number of vectors;
[0048] The maximum number of connections per node is 16 to 32 at the bottom layer and 64 to 128 at the top layer;
[0049] Topology parameters include density threshold, skeleton node ratio, and shortcut connection ratio;
[0050] Optimize the trigger threshold to a fixed number of queries or a fixed time interval;
[0051] The hotspot determination threshold is based on the multiple of the node access frequency relative to the average value.
[0052] Preferably, the method also includes distributed scalability:
[0053] A data sharding strategy based on semantic clustering distributes semantic vectors to different computing nodes;
[0054] The multi-machine index distribution mechanism enables each computing node to maintain a portion of the index structure;
[0055] The intelligent query routing system sends query requests to the most relevant computing nodes;
[0056] The result merging mechanism integrates search results from multiple computing nodes and sorts them uniformly.
[0057] A global topology maintenance strategy ensures the consistency and integrity of the index structure in a distributed environment.
[0058] This invention deeply optimizes the HNSW index structure by introducing topology space theory, constructing a topology-aware multi-level index framework, and achieving the following beneficial effects:
[0059] 1. Significantly improved retrieval accuracy: By combining deep learning with topological structure analysis, the system can more accurately capture deep semantic relationships between texts, improving retrieval accuracy by 15% to 20% while maintaining the same recall rate.
[0060] 2. Significantly improved query efficiency: Based on the topology-aware multi-layer navigation strategy, the system query time complexity is reduced to the logarithmic level, which is 30% to 50% faster than the traditional HNSW. Especially for high-frequency query paths, the response time is reduced by 40% to 50%.
[0061] 3. Significantly reduced resource consumption: Through dynamic optimization and topology entropy-guided structural optimization, system memory consumption is reduced by 20% to 25%, and index building time is reduced by 15% to 20%.
[0062] 4. Enhanced Adaptability: The system can dynamically adjust the index structure according to the actual query pattern, create optimized connections for high-frequency query paths, form a semantic highway, and continuously optimize system performance according to usage patterns.
[0063] 5. Significantly improved scalability: Through distributed architecture design, the system can be scaled up to a scale of billions of vectors, supporting continuous data growth while maintaining high-efficiency retrieval performance. Attached Figure Description
[0064] Figure 1 This is a flowchart of the large-scale semantic retrieval optimization method based on deep learning of the present invention. Detailed Implementation
[0065] Please refer to Figure 1 The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the scope of the invention.
[0066] like Figure 1 As shown, the large-scale semantic retrieval optimization method based on deep learning provided by this invention includes the following steps:
[0067] Step 1: Obtain the text data to be retrieved, and convert the text data to be retrieved into a semantic vector representation using a deep learning model;
[0068] Step 2: Perform topological structure analysis on the semantic vector representation to obtain topological characteristic information of the semantic space;
[0069] Step 3: Based on the aforementioned topological characteristic information, construct a topology-aware hierarchical navigable small-world graph index with a multi-level structure, wherein each level contains a set of nodes and their connection relationships;
[0070] Step 4: Based on the actual query patterns and system performance feedback, dynamically optimize and adjust the topology-aware hierarchical navigable small world graph index;
[0071] Step 5: Receive the query request, convert the query request into a query vector, perform a multi-level navigation search in the topology-aware hierarchical navigable small world graph index, and return the retrieval results that are semantically related to the query vector.
[0072] In a preferred embodiment of the present invention, the specific process of converting the text data to be retrieved into a semantic vector representation using a deep learning model in step one is as follows:
[0073] First, a pre-trained language model is used to encode the retrieved text data to obtain a context-sensitive representation of the text. This invention preferably uses BERT or its variants as the basic semantic encoder. The BERT model encodes the input text through a multi-layer bidirectional Transformer structure, effectively capturing the contextual relationships between words. For text data in specific domains, domain-adaptive fine-tuning can be performed to improve the accuracy of semantic representation.
[0074] Secondly, the aforementioned context-sensitive representations are integrated into a fixed-dimensional semantic vector through weighted averaging, special label extraction, or attention pooling mechanisms. In a preferred embodiment, the following attention pooling mechanism can be used:
[0075] ,
[0076] in: The final semantic vector representation typically has a dimension of 768; Let represent the hidden state of the i-th token, with dimensions and . same; The attention weight is a scalar value between 0 and 1. The sequence length represents the number of tokens after the input text is segmented. This represents a weighted summation of the hidden states of all tokens. Attention weights. Calculated in the following way:
[0077] ,
[0078] in: Let i be the attention weight of the i-th token. For a learnable parameter vector, the dimension and same; Representing vectors Transpose of; It is a natural exponential function; This is a normalization factor that ensures the sum of all weights is 1. This calculation method is actually an application of the softmax function, making the sum of the attention weights of all tokens equal to 1, forming a probability distribution.
[0079] Finally, the resulting semantic vectors are normalized, and dimensionality adjustments or feature selections are performed as needed. L2 normalization is typically used to ensure uniform vector length.
[0080] ,
[0081] in: This is the normalized vector; This is the original semantic vector; For vectors The L2 norm is calculated using the following formula: ,in For vector dimensions, For vectors The Each element. L2 normalization makes the length (modulus) of the vector equal to 1, which facilitates subsequent calculations using cosine similarity.
[0082] In practical applications, semantic vectors of different dimensions can be selected based on the scale and performance requirements of the retrieval system. For general application scenarios, 768-dimensional vectors can provide good semantic representation capabilities; for scenarios requiring higher efficiency, the vectors can be reduced to 128 or 256 dimensions through methods such as principal component analysis (PCA) or autoencoders, thereby reducing computational overhead while retaining the main semantic information.
[0083] In another preferred embodiment of the present invention, the specific process of performing topological structure analysis on the semantic vector representation in step two is as follows:
[0084] First, the local density distribution and global density gradient in the semantic space are calculated to identify high-density and sparse regions. Local density can be estimated as follows:
[0085] ,
[0086] in: For vectors The local density at a given location is a non-negative scalar value; For vectors and The distance between them is usually calculated using cosine distance or Euclidean distance; This is the Gaussian kernel width parameter, which controls the range of density calculation. The total number of vectors; This indicates summing over all vectors; This is the natural exponential function. The formula essentially applies the Gaussian kernel function to a point... Estimate the density of the surrounding data, point The more points around it, the higher its density value. The larger. Preferably, The value can be set to 0.1 to 0.3 times the average of all distances, and the specific value can be adjusted according to the characteristics of the data distribution. The density gradient represents the direction and rate of density change and is used to identify the transition zone between high-density and sparse regions.
[0087] Secondly, the intrinsic structure of the semantic space is determined by analyzing its dimension, curvature properties, and connectivity. The intrinsic dimension can be estimated using the following methods:
[0088] ,
[0089] in: is the intrinsic dimension, representing the minimum dimension required for the actual distribution of the data; For radius The number of data points contained within the hypersphere; Represents the natural logarithm; Indicates when The limit is approaching 0. This formula is based on dimensionality estimation theory, estimating the intrinsic dimension by observing the distribution characteristics of data points at different scales. In practical calculations, multiple different radius values can be taken, and the dimension can be estimated by fitting the slope through linear regression.
[0090] Furthermore, hierarchical clustering and density clustering methods are used to identify naturally formed semantic clusters and boundary points in the semantic space. In a preferred embodiment, density peak clustering can be used to identify the centers of semantic clusters.
[0091] ,
[0092] in: For vectors The centrality measure is such that a larger value indicates that the point is more likely to be a cluster center. This represents the local density calculated previously; For distance metric, representing vectors Distance to the nearest vector with a higher density. The calculation formula is:
[0093] ,
[0094] in: For vectors Distance metric; For vectors and The distance between; Indicates all densities higher than The minimum distance is selected from the points. For the point with the highest density, since there are no points with higher density, a special setting is used. This is the maximum distance from that point to all other points. Cluster centers typically have high [values / values]. Values can be set by thresholding. To identify, preferably, Can be set to The top 5 percentiles of the value.
[0095] Finally, keypoints, skeleton structures, and bottleneck regions of semantic flow in the semantic space are identified to form a topological descriptor. The skeleton structure can be identified by connecting keypoints on cluster centers and density paths, while bottleneck regions are narrow connection channels between different clusters.
[0096] In yet another preferred embodiment of the present invention, the specific process of constructing a topology-aware hierarchical navigable small-world graph index with a multi-level structure in step three is as follows:
[0097] First, a complete underlying index is constructed, containing all semantic vectors and their initial connections. In the underlying index, each node represents a semantic vector, and connections between nodes are established based on semantic similarity. Preferably, the initial connections are constructed using a k-NNgraph, where each node is connected to its k nearest neighbors, and the value of k is typically set between 16 and 32.
[0098] Secondly, based on the importance metrics of nodes, some nodes are selected and promoted to higher levels, forming a decreasing hierarchical structure. Node importance can be evaluated by combining centrality measures and local density.
[0099]
[0100] in: This is a node importance indicator; the larger the value, the more important the node. The centrality measure has already been calculated. The density is local, as previously calculated; is a weighting coefficient that controls the relative weight of centrality and density in importance calculation, with a preferred value range of 0.3 to 0.7. The probability of a node being promoted is proportional to its importance index, and the number of nodes in each layer follows an exponential decay law, typically with the number of nodes in the upper layer being 1 / 4 to 1 / 7 of that in the lower layer.
[0101] Then, within each layer, the connection relationships between nodes are determined based on topological characteristics. This invention innovatively introduces multiple connection strategies, including basic nearest neighbor connections, topology-aware connections, density-adaptive connections, and long-distance shortcut connections.
[0102] Basic nearest neighbor connections ensure that each node establishes basic connections with its K nearest neighbors, and the connection weights can be dynamically adjusted based on semantic distance.
[0103] ,
[0104] in: For nodes With nodes The connection weights between nodes, with values between 0 and 1; This represents the distance between corresponding semantic vectors; This is the average distance between all connected nodes; It is a natural exponential function. This formula results in a greater weight for connections between nodes that are closer together and a smaller weight for connections between nodes that are farther apart, creating a smooth decay of weights.
[0105] Topology-aware connectivity increases the number of connections to nodes on the semantic skeleton and increases connection density in semantic bottleneck regions. The number of connections to skeleton nodes can be set to 1.5 to 2 times the base number of connections, and the connection density in bottleneck regions can be increased by 30% to 50%.
[0106] Density-adaptive connectivity makes the number of connections proportional to local density, allowing higher-density regions to receive more connections to improve retrieval accuracy.
[0107] ,
[0108] in: For nodes The target number of connections, rounded up to the nearest integer; The basic connection number is typically 16 to 32; As an adjustment coefficient, to control the degree of influence of density on the number of connections, the preferred value range is 0.5 to 1.5; For nodes Local density; and These are the minimum and maximum local densities of all nodes, respectively. The density values are normalized to a range of 0 to 1. This formula allows nodes in high-density regions to have more connections, while nodes in low-density regions have fewer connections, thus improving the retrieval accuracy in high-density regions while keeping the overall number of connections under control.
[0109] Long-distance shortcut connections establish direct connections between distant semantic regions, optimizing global navigation efficiency. The number of shortcut connections typically accounts for 1% to 5% of the total connections, and they are preferentially established between different cluster centers.
[0110] Finally, inter-layer connections are established to ensure that queries can efficiently navigate from the top layer to the target node at the bottom layer. Inter-layer connections follow the funnel principle, that is, upper-layer nodes are connected to multiple related nodes in the lower layer to form a navigation path.
[0111] In another preferred embodiment of the present invention, the specific process of dynamically optimizing and adjusting the topology-aware hierarchical navigable small world graph index in step four is as follows:
[0112] First, monitor system performance metrics, including query path length, node access frequency, query response time, and resource usage. The system records the complete path information for each query, calculates the access frequency of each node, and monitors the query response time distribution. Resource usage monitoring includes the memory usage of the index structure and CPU load.
[0113] Secondly, based on the aforementioned performance metrics, identify the hot paths and inefficient areas that require optimization. Hot paths refer to paths that are frequently queried and accessed, and can be identified in the following ways:
[0114]
[0115] in: Representing a path The popularity index is a dimensionless ratio; For path Access frequency, in times per hour; Average access frequency for all paths, in units of same; The threshold for determining hotspots is a constant greater than 1, preferably between 3 and 5. The inequality expresses the condition that when the access frequency of a path exceeds its average access frequency... When the length of a query path is doubled, it is identified as a hotspot path. Inefficient areas refer to areas with abnormally long query paths or long response times.
[0116] Then, add direct connections or increase connection weights for hotspot paths to simplify the connection structure in low-frequency access areas. For non-adjacent node pairs on hotspot paths... If their common access frequency exceeds a threshold, then a direct connection is added:
[0117]
[0118] in: For node pairs The common access frequency, expressed in times per hour, represents the frequency of access to nodes during the query process. and nodes The frequency of consecutive accesses; Add a threshold to the connection, unit and Similarly, it is preferable to set it to twice the average access frequency of the node. This indicates a logical deduction; if the condition on the left is true, the operation on the right is executed. For low-frequency access areas, the number of connections can be appropriately reduced, adjusting it to 60% to 80% of the original value.
[0119] Finally, regularly evaluate the overall index topology quality and perform a global structure rebalancing if necessary. Topology quality can be evaluated using topology entropy.
[0120] This represents the topological entropy; a lower value indicates a more efficient structure. The total number of nodes; For nodes The set of neighbors; For the node Transfer to node The probability of; It is the natural logarithm; This represents summing over all nodes; Indicates a node Summing all of its neighbors. It is usually proportional to the connection weight:
[0121] ,
[0122] in: For the node Transfer to node The probability of; For nodes With nodes The connection weights between them have already been calculated. For nodes The sum of all connection weights is used as a normalization factor. This formula ensures that from the node... The sum of all transition probabilities from the starting point is 1, forming a probability distribution. The lower the topological entropy, the more efficient the index structure. When the topological entropy exceeds a threshold or fails to decrease after multiple consecutive optimizations, a global structure rebalancing is triggered.
[0123] Furthermore, this invention includes an incremental update mechanism for handling the operations of adding, deleting, and updating vectors. For newly added vectors, the optimal entry point is selected based on topological characteristics, its hierarchical distribution is determined, and topology-aware connections are constructed. The entry point selection can be based on a weighted combination of semantic similarity and centrality metrics:
[0124] ,
[0125] in: For nodes The higher the score of the entry point, the more suitable it is as an entry point. For new vectors and nodes The similarity between corresponding vectors is usually calculated using cosine similarity. For nodes The centrality measure of has been calculated previously; The weighting coefficient controls the relative importance of similarity and centrality in the selection of entry points, with an optimal value range of 0.6 to 0.8. This formula balances the two factors of similarity and centrality, taking into account both the semantic relationship between the new vector and existing nodes, and the importance of nodes in the overall structure.
[0126] For deletion operations, it is necessary to reconstruct the connections of the deleted node, update the affected navigation paths, and maintain topological integrity. When a node... When deleted, all its neighboring nodes The connectivity needs to be reassessed.
[0127] .
[0128] in: Indicates "for all"; Represents a node and nodes All are deleted nodes The neighbors; For nodes and nodes The distance between corresponding vectors; This is a distance threshold, typically set to 1.2 to 1.5 times the original connection distance. This condition means that if two nodes originally connected to the deleted node are close to each other, a direct connection is established between them to maintain network connectivity and navigation efficiency.
[0129] For vector updates, vectors with significant semantic changes are first identified. Only the affected parts of the structure are reconstructed, and multiple small updates are aggregated using batch processing. The degree of semantic change can be evaluated using the cosine distance between the old and new vectors.
[0130] ,
[0131] in: For vectors The degree of semantic change, with values ranging from 0 to 2, where 0 indicates exactly the same and 2 indicates completely opposite directions; The updated vector; The vector before the update; Let represent the cosine similarity function, which calculates the cosine of the angle between two vectors. When Exceeding the threshold When the value is typically set to 0.1 to 0.2, it is considered a significant change and the connection structure of the node needs to be updated.
[0132] In yet another preferred embodiment of the present invention, the specific process of performing multi-level navigation search in the topology-aware hierarchical navigable small world graph index in step five is as follows:
[0133] First, start with the top-level entry point, or select an optimized entry point based on the query context. For the first query, the top-level global entry point is usually used; for consecutive queries with contextual relationships, nodes related to the results of previous queries can be selected as entry points to improve retrieval efficiency.
[0134] Next, find the node closest to the query vector in the current layer, and then move to that node. This step uses a greedy search strategy, calculating the distance between the query vector and all neighboring nodes of the current node, and selecting the neighbor with the smallest distance as the next node to visit.
[0135] ,
[0136] Where: next is the next node to be visited; Indicates at node All neighbors Find the node that minimizes the subsequent expression. query vector With nodes Corresponding vector The distance between them. This formula means selecting the neighbor node with the smallest distance to the query vector as the next node to be visited.
[0137] Then, explore the neighboring nodes along the connections of the current node to find nodes closer to the query vector. This step uses a beam search strategy, maintaining a candidate node set C, initially containing all the neighbors of the current node. In each iteration, calculate the distance between the query vector and all nodes in C, and retain the nodes with the smallest distances (ef is the expansion factor, usually set to 64 to 256). The iteration process continues until no closer nodes can be found or the maximum number of iterations is reached.
[0138] When no closer node can be found in the current layer, the search descents to the next layer to continue. The descent strategy follows the nearest-first principle, starting from the best node in the current layer and descending to the corresponding region in the next layer through inter-layer connections to continue the search process.
[0139] Finally, an extended search is performed at the bottom layer to ensure result accuracy, and the candidate results are then validated and ranked. The bottom-layer search expands the exploration scope, typically increasing the expansion factor ef to 2 to 4 times the original value to improve recall. The final results are ranked according to their semantic similarity to the query vector, returning the k most similar results.
[0140] In one embodiment of the present invention, the implementation of the above method includes a series of parameter configurations, which have a significant impact on system performance:
[0141] The number of index levels is proportional to the logarithm of the total number of vectors, and can be determined by the following formula:
[0142] ,
[0143] in: The index level is rounded up to the nearest integer. The total number of vectors; This represents the attenuation ratio of the number of nodes between layers, typically ranging from 4 to 7. This represents the floor function; Indicated by Logarithm to the base. For example, for an index containing 100 million vectors, use... When, the number of levels should be .
[0144] The maximum number of connections per node is 16 to 32 at the bottom layer and 64 to 128 at the top layer. The number of connections increases linearly with each layer. The number of connections in a layer can be set as follows:
[0145] ,
[0146] in: For the first The number of connections in a layer, rounded up to the nearest integer; This refers to the number of underlying connections, typically ranging from 16 to 32. This represents the number of top-level connections, typically ranging from 64 to 128. For the overall level; This is a hierarchical index, ranging from 0 to... This formula achieves a linear increase in the number of connections from the bottom layer to the top layer, allowing top-level nodes to have more connections to improve global navigation efficiency.
[0147] Topology parameters include density threshold, skeleton node ratio, and shortcut connection ratio. Density threshold To identify high-density regions, it is typically set to the 70% to 80% quantile of all node density values. The skeleton node ratio is usually 5% to 10% of the total nodes, and the shortcut connection ratio is 1% to 5% of the total connections.
[0148] The optimization trigger threshold is set to a fixed number of queries or a fixed time interval, such as triggering an optimization every 10,000 queries or every 24 hours. The hotspot determination threshold is based on the multiple of the node access frequency relative to the average value, and is usually set to 3 to 5 times.
[0149] In another embodiment of the invention, the method further includes distributed scalability, enabling the system to handle larger-scale data:
[0150] First, a data sharding strategy based on semantic clustering is used to distribute semantic vectors to different computing nodes. Semantic clustering employs K-means or spectral clustering algorithms to divide the vector space into K clusters, with each cluster assigned to a computing node. Preferably, the value of K can be set to 1 to 2 times the number of computing nodes to achieve load balancing.
[0151] Secondly, the multi-machine index distribution mechanism enables each compute node to maintain a portion of the index structure. Each compute node is responsible for building and maintaining the index structure of its assigned subset of vectors, while maintaining a small number of cross-node connections to ensure global reachability.
[0152] Then, the intelligent query routing system sends the query request to the most relevant computing nodes. Query routing is based on the similarity between the query vector and the semantic cluster centers managed by each computing node, sending the query request to the nodes with the highest similarity:
[0153] ,
[0154] in: query vector The target node set contains the indices of all compute nodes that meet the conditions; For the index of the compute node; query vector With computing nodes Semantic cluster center of management The similarity between them is usually calculated using cosine similarity; This is the routing threshold, typically set to 80% to 90% of the maximum similarity. This indicates all conditions that are met. Node index The set constitutes the set. This formula enables the routing of query requests to the computation nodes most relevant to the semantics of the query vector.
[0155] Next, the result merging mechanism integrates the search results from multiple computing nodes and performs a unified sorting. Result merging employs a "score normalization" strategy, normalizing the similarity scores returned by different nodes before comparison.
[0156] ,
[0157] in: Normalized fractions; query vector With the result vector The original similarity between them; For computing nodes Return the mean of all scores; For computing nodes The standard deviation of all returned scores. This formula uses Z-score normalization to make the similarity scores returned by different computing nodes comparable, thus enabling fair merging of results from multiple nodes.
[0158] Finally, a global topology maintenance strategy ensures the consistency and integrity of the index structure in a distributed environment. The system periodically exchanges information about the boundary regions between nodes, updates cross-node connections, and performs global topology optimization, typically every 1 to 7 days, with the specific frequency depending on the data update speed.
[0159] Through the above steps and strategies, this invention realizes a large-scale semantic retrieval optimization method based on deep learning, successfully solving the problems of traditional retrieval methods in terms of semantic understanding, retrieval efficiency and scalability, and providing an efficient, accurate and scalable technical solution for large-scale semantic retrieval.
[0160] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.
Claims
1. A large-scale semantic retrieval optimization method based on deep learning, characterized in that, include: The text data to be retrieved is obtained, and the text data to be retrieved is converted into a semantic vector representation through a deep learning model; Perform topological structure analysis on the semantic vector representation to obtain topological characteristic information of the semantic space; Based on the aforementioned topological characteristic information, a topology-aware hierarchical navigable small-world graph index with a multi-level structure is constructed, wherein each level contains a set of nodes and their connection relationships. Based on actual query patterns and system performance feedback, the topology-aware hierarchical navigable small world graph index is dynamically optimized and adjusted. Receive a query request, convert the query request into a query vector, perform a multi-level navigation search in the topology-aware hierarchical navigable small world graph index, and return retrieval results that are semantically related to the query vector; The specific steps for constructing a topology-aware hierarchical navigable small-world graph index with a multi-level structure include: Build a complete underlying index, including all semantic vectors and their initial connections; Based on the importance index of nodes, the importance of nodes is evaluated by combining centrality measure and local density, and some nodes are selected to be promoted to the upper level to form a decreasing hierarchical structure. Within each layer, the connection relationships between nodes are determined based on topological characteristics. Establish inter-layer connections to ensure that queries can efficiently navigate from the top layer to the target node at the bottom layer; Determining the connections between nodes includes: Basic nearest neighbor connections enable each node to establish basic connections with its K nearest neighbors. Topology-aware connectivity increases the number of connections for nodes located on the semantic skeleton and increases connection density in semantic bottleneck regions. Density-adaptive connections make the number of connections proportional to the local density, allowing high-density areas to receive more connections to improve retrieval accuracy. Long-distance shortcut connections establish direct connections between semantic regions over long distances, optimizing global navigation efficiency; The specific steps for dynamically optimizing and adjusting the topology-aware hierarchical navigable small world graph index include: Monitor system performance metrics, including query path length, node access frequency, query response time, and resource usage; Based on the aforementioned performance metrics, identify hotspot paths and inefficient areas that require optimization; Add direct connections or increase connection weights for hotspot paths, and simplify the connection structure for low-frequency access areas; Regularly evaluate the overall topology quality of the index. Topology quality is evaluated by topology entropy. When the topology entropy exceeds the threshold or does not decrease after multiple optimizations, perform a global structure rebalancing.
2. The large-scale semantic retrieval optimization method based on deep learning according to claim 1, characterized in that, The steps of converting the text data to be retrieved into a semantic vector representation using a deep learning model specifically include: The text data to be retrieved is encoded using a pre-trained language model to obtain a context-sensitive representation of the text; The context-sensitive representation is integrated into a fixed-dimensional semantic vector through weighted averaging, special label extraction, or attention pooling mechanisms. The semantic vectors are normalized, and their dimensions are adjusted or features are selected as needed to balance representational power and computational efficiency.
3. The large-scale semantic retrieval optimization method based on deep learning according to claim 1, characterized in that, The specific steps for performing topological structure analysis on the semantic vector representation include: Calculate the local density distribution and global density gradient in the semantic space to identify high-density and sparse regions; Analyze the dimension, curvature properties, and connectivity of the semantic space to determine its intrinsic structure; Hierarchical clustering and density clustering methods are used to identify naturally formed semantic clusters and boundary points in the semantic space; Identify key points, skeletal structures, and bottleneck regions of semantic flow in the semantic space to form a topological descriptor.
4. The large-scale semantic retrieval optimization method based on deep learning according to claim 1, characterized in that, The dynamic optimization adjustment also includes an incremental update mechanism: For newly added vectors, the optimal entry point is selected based on topological characteristics, their hierarchical distribution is determined, and topology-aware connections are constructed. For deletion operations, the connections of the deleted nodes are reconstructed, the affected navigation paths are updated, and topological integrity is maintained. For vector updates, identify vectors with significant semantic changes, reconstruct only the affected parts of the structure, and aggregate multiple small updates using a batch processing approach.
5. The large-scale semantic retrieval optimization method based on deep learning according to claim 1, characterized in that, The steps for performing multi-level navigation search in the topology-aware hierarchical navigable small-world graph index specifically include: Start from the top-level entry point, or choose an optimized entry point based on the query context; Find the node in the current layer that is closest to the query vector, and then move to that node; Explore the neighboring nodes along the connections of the current node to find the node that is closer to the query vector; When no closer node can be found in the current layer, descend to the next layer to continue the search; An extended search is performed at the bottom layer to ensure the accuracy of the results, and the candidate results are finally verified and sorted.
6. The large-scale semantic retrieval optimization method based on deep learning according to claim 1, characterized in that, The implementation of the method includes the following parameter configuration: The number of index levels is proportional to the logarithm of the total number of vectors; The maximum number of connections per node is 16 to 32 at the bottom layer and 64 to 128 at the top layer; Topology parameters include density threshold, skeleton node ratio, and shortcut connection ratio; Optimize the trigger threshold to a fixed number of queries or a fixed time interval; The hotspot determination threshold is based on the multiple of the node access frequency relative to the average value.
7. The large-scale semantic retrieval optimization method based on deep learning according to claim 1, characterized in that, The method also includes distributed scalability: A data sharding strategy based on semantic clustering distributes semantic vectors to different computing nodes; The multi-machine index distribution mechanism enables each computing node to maintain a portion of the index structure; The intelligent query routing system sends query requests to the most relevant computing nodes; The result merging mechanism integrates search results from multiple computing nodes and sorts them uniformly. A global topology maintenance strategy ensures the consistency and integrity of the index structure in a distributed environment.