Cluster-based data retrieval method, electronic device, and computer program product

Abstract

The present disclosure provides a cluster-based data retrieval method, an electronic device and a computer program product. The method of the present disclosure first determines at least one data category with the highest semantic similarity to the data to be retrieved from a plurality of established data categories, wherein the data categories are obtained by clustering existing data. Then, a multi-layer index structure corresponding to the determined data category is searched layer by layer to obtain a plurality of existing data most similar to the data to be retrieved and to provide feedback, wherein the searchable data range of the upper index structure searched first in the two index structures adjacent in the retrieval order in the same multi-layer index structure is contained in the searchable data range of the lower index structure searched later, and the retrieval of the lower index structure is based on the retrieval result of the upper index structure.

Description

Technical Field

[0001] This disclosure specifically relates to clustering-based data retrieval methods, electronic devices, readable storage media, and computer program products. Background Technology

[0002] With the rapid development of artificial intelligence and big data technologies, semantic retrieval has become a key technology in many application scenarios such as information retrieval, intelligent question answering, and recommendation systems. Semantic retrieval enables the rapid retrieval of the most semantically relevant candidate results from a knowledge base.

[0003] Currently, some common semantic retrieval methods can reduce query time by narrowing the search space, but the retrieval accuracy cannot meet expectations in some business scenarios, especially when the knowledge base data is unevenly distributed and the category span is large, the recall results are prone to deviating from the true optimal answer. Summary of the Invention

[0004] This disclosure provides clustering-based data retrieval methods, electronic devices, readable storage media, and computer program products.

[0005] The first aspect of this disclosure proposes a clustering-based data retrieval method, comprising: determining at least one data category with the highest semantic similarity to the data to be retrieved from multiple established data categories, wherein the data category is obtained by clustering existing data; and performing a layer-by-layer retrieval of the multi-level index structure corresponding to the determined data category to obtain multiple existing data most similar to the data to be retrieved and providing feedback, wherein in the same multi-level index structure, among two index structures that are adjacent in retrieval order, the retrieval range of the lower index structure retrieved later is included in the retrieval range of the upper index structure retrieved earlier, and the retrieval of the lower index structure is based on the retrieval result of the upper index structure.

[0006] According to some embodiments of this disclosure, determining at least one data category with the highest semantic similarity to the data to be retrieved from a plurality of established data categories includes: determining at least one target reference point with the highest semantic similarity to the data to be retrieved from reference points corresponding to the plurality of established data categories, wherein the reference point can represent the central tendency of existing data in the corresponding data category in the semantic space; and determining the data category corresponding to the target reference point as the target data category with the highest semantic similarity to the data to be retrieved.

[0007] According to some embodiments of this disclosure, a layer-by-layer retrieval is performed on the multi-level index structure corresponding to the determined data category, including: taking the target reference point of the target data category as the starting index node of the index structure retrieved first in the retrieval order, and performing a layer-by-layer retrieval on the multi-level index structure corresponding to the target data category. Each layer of the index structure includes multiple index nodes, and the index nodes correspond to the existing data. The range of retrievalable data is represented by the distribution of index nodes contained in the index structure of the corresponding layer.

[0008] According to some embodiments of this disclosure, there are multiple target data categories; performing a layer-by-layer retrieval of the multi-level index structure corresponding to the target data category includes: taking the multiple multi-level index structures to which the multiple starting index nodes belong as multiple retrieval tasks, processing the multiple retrieval tasks in parallel, and retrieving the multi-level index structures contained in the retrieval tasks in a layer-by-layer retrieval manner.

[0009] According to some embodiments of this disclosure, each index node in at least some of the index nodes corresponds to at least one neighboring index node, and the target index node of the lower-level index structure used as the starting point for retrieval at this level includes the index node in the retrieval result of the upper-level index structure. The retrieval result of the current-level index structure is determined by retrieving the neighboring index nodes of the target index node.

[0010] According to some embodiments of this disclosure, determining at least one target reference point with the highest semantic similarity to the data to be retrieved includes: determining the distance between the target semantic vector of the data to be retrieved and the reference semantic vector of the reference point; and determining at least one reference semantic vector that is closest to the target semantic vector, and taking the reference point corresponding to the determined reference semantic vector as the target reference point.

[0011] According to some embodiments of this disclosure, the reference point includes the data category as the cluster center of the cluster.

[0012] According to some embodiments of this disclosure, multiple existing data that are most similar to the data to be retrieved are determined in the index structure that is retrieved last in the retrieval order.

[0013] A second aspect of this disclosure provides an electronic device, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform the data retrieval method described in any of the above embodiments.

[0014] A third aspect of this disclosure provides a readable storage medium storing a computer program, which, when executed by a processor, is used to implement the data retrieval method described in any of the above embodiments.

[0015] This disclosure provides a fourth aspect of a computer program product, the computer program product comprising a computer program, which, when executed by a processor, is used to implement the data retrieval method described in any of the above embodiments. Attached Figure Description

[0016] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0017] Figure 1 The diagram illustrates an application scenario of a cluster-based retrieval method according to some embodiments of the present disclosure.

[0018] Figure 2 A schematic diagram of the overall process of a clustering-based data retrieval method M100 according to some embodiments of the present disclosure is shown.

[0019] Figure 3 A schematic diagram of the overall process for retrieving text W using data retrieval method M100 in some embodiments is shown.

[0020] Figure 4 A schematic diagram of the overall flow of a clustering-based data retrieval method M100 according to other embodiments of this disclosure is shown.

[0021] Figure 5 This diagram illustrates a step-by-step retrieval of the multi-level index structure for data category K1.

[0022] Figure 6 A schematic diagram of the overall flow of a clustering-based data retrieval method M100 according to other embodiments of this disclosure is shown.

[0023] Figure 7 A schematic diagram of the overall process for retrieving text W using data retrieval method M100 in some other embodiments is shown.

[0024] Figure 8 A schematic diagram of the overall flow of a clustering-based data retrieval method M100 according to other embodiments of this disclosure is shown.

[0025] Figure 9 This is a schematic block diagram of an electronic device 1000 according to one embodiment of the present disclosure. Detailed Implementation

[0026] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0027] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0028] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0029] The terminology used herein is for the purpose of describing particular embodiments and is not restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0030] As service providers offer different types of services to customers in various business scenarios, these services may generate a large amount of interactive information, such as business inquiries and feedback. This interactive information is submitted to the service provider's service platform. To improve the efficiency of processing this information, the service platform can structure and organize it into a knowledge base covering multiple business lines, which can be used by an automated question-and-answer system.

[0031] In real-world business scenarios, users of the service provider need to obtain accurate answers within a short period of time using the semantic retrieval function of the aforementioned question-and-answer system. Common semantic retrieval methods include traditional information retrieval enhancement based on keyword expansion and semantic rules, or semantic fingerprint matching based on hash signatures. These methods efficiently but with limited precision generate a recall set. Then, secondary retrieval of the recall results is performed using methods such as brute-force search, cosine similarity, and inner product calculation to improve the final accuracy. This multi-stage retrieval strategy of "rough screening + precise ranking" is used to obtain the answer.

[0032] Because fine-ranking algorithms incur additional computational overhead and significantly increase retrieval time, they cannot meet the demands of high concurrency and low latency real-time interaction, making them difficult to apply in business scenarios requiring millisecond-level response times. Especially when the data distribution in the knowledge base is uneven and the category range is large, the retrieved results are prone to deviating from the true optimal answer, causing the retrieval accuracy to fail to meet user requirements in some business scenarios.

[0033] Therefore, this disclosure proposes a cluster-based retrieval method.

[0034] Figure 1 The diagram illustrates an application scenario of a clustering-based retrieval method according to some embodiments of this disclosure. In this application scenario, a query terminal 10 and a server 20 may be included. The query terminal 10 is capable of communicating with the server 20 to send and receive data or commands. In this disclosure, both the query terminal 10 and the server 20 each include at least one processor and at least one memory.

[0035] For example, query terminal 10 can be a terminal device held by an employee (user) of an enterprise, and server 20 can be a server of the enterprise's business system. Query terminal 10 receives user input, generates a corresponding query request, and sends it to server 20. After receiving the query request, server 20 converts the text contained in the query request into a vector to obtain the vector to be retrieved. First, it determines the multiple data categories with the most similar semantics from the multiple vector data categories established in the knowledge base through semantic similarity calculation. Each data category corresponds to a multi-level index structure. By performing a layer-by-layer retrieval on the multi-level index structure of each determined data category, multiple recall vectors most similar to the vector to be retrieved can be obtained after completing the layer-by-layer retrieval. Since the recall vectors are obtained by vectorizing existing text (knowledge text), the existing text corresponding to each recall vector is the multiple texts most similar to the text in the query request, thereby obtaining the text retrieval results and feeding them back to query terminal 10.

[0036] exist Figure 1The shape and structure of the query terminal 10 and server 20 shown should not be construed as a limitation on the scope of protection of this disclosure. In this disclosure, the query terminal 10 can be different types of electronic devices, such as mobile phones, tablets, laptops, or desktop computers. Furthermore, the server 20 can be a physical server or a cloud server; this disclosure does not limit the type of server.

[0037] Figure 2 A schematic diagram of the overall flow of a clustering-based data retrieval method M100 according to some embodiments of this disclosure is shown. Figure 2 The method shown includes steps S110 and S120. This method can be executed by electronic devices such as computers and servers.

[0038] S110, from the multiple established data categories, determine at least one data category with the highest semantic similarity to the data to be retrieved. The data categories are obtained by clustering existing data.

[0039] S120: Perform a layer-by-layer search on the multi-level index structure corresponding to the determined data category to obtain multiple existing data that are most similar to the data to be searched and provide feedback. Specifically, within the same multi-level index structure, among two adjacent index structures in the search order, the searchable data range of the later-searched lower-level index structure is included within the searchable data range of the earlier-searched upper-level index structure, and the search of the lower-level index structure is based on the search results of the upper-level index structure.

[0040] The data retrieval method M100 disclosed herein is used to retrieve multiple data sets that are semantically most similar to the data to be retrieved from existing data. Existing data may include vector data of multiple text or images, or may directly include the text or image data itself. Vectors can be obtained by converting natural language text into fixed-dimensional numerical vectors, and text may include textual information such as business inquiries and feedback generated by the service provider during interactions with customers. Existing data is stored in a database to form a knowledge base; the existing data constitutes the knowledge data in the knowledge base.

[0041] Data categories refer to semantically similar subsets of data partitioned by clustering algorithms applied to existing data. Each data category exhibits high semantic consistency. Examples include customer demand types, business indicator types, etc. The clustering algorithms mentioned can be K-means, hierarchical clustering, density-based clustering, or other types of algorithms.

[0042] The semantic similarity calculated in step S110 can be achieved by calculating the cosine similarity, Euclidean distance, or other parameters between the vector of the data to be retrieved (i.e., the vector to be retrieved) and the reference vector. The reference vector can represent the overall semantic features of the data category. A data category may include one or more reference vectors.

[0043] An index structure corresponds to multiple existing data sets, including indexes on those existing data sets. A data category can correspond to one multi-level index structure, and all multi-level index structures for all data categories can correspond to all existing data sets, meaning all existing data sets constitute the searchable data range. The index structure can be graph-based, search tree-based, or hash table-based.

[0044] Any multi-level index structure can have a predefined retrieval order. Taking a three-level index structure as an example, the retrieval order is top-level, middle-level, and bottom-level. When retrieving from this three-level index structure, the top-level index structure is retrieved first, resulting in n1 retrieval results, where n1 ≥ 1. At this point, the top-level index structure is the upper-level index structure of the middle-level index structure, and the middle-level index structure is the lower-level index structure of the top-level index structure. In the obtained retrieval results, each retrieval result corresponds to a vector of existing data.

[0045] The searchable data range differs at each level of the index structure and increases sequentially according to the retrieval order. For example, the searchable data range of the intermediate index structure includes that of the top-level index structure; that is, the searchable data of the top-level structure is a subset of the searchable data of the intermediate layer. Therefore, the aforementioned n1 search results are all contained within the intermediate index structure. Based on these n1 search results, a search is performed within the intermediate index structure to obtain n2 search results, where n2 ≥ n1. At this point, the intermediate layer becomes the upper-level index structure of the bottom layer, and the bottom layer becomes the lower-level index structure of the intermediate layer. This hierarchical jump gradually narrows the search space.

[0046] Similarly, based on the n² search results mentioned above, a search is performed in the lowest-level index structure to obtain n³ search results, where n³ ≥ n². Obtaining n³ search results is equivalent to obtaining n³ vectors of existing data; these vectors are the most semantically similar to the vector to be searched, and the value of n³ is usually greater than 1. After obtaining the n³ search results, the corresponding text or image data can be fed back to the client as a response to the client's question.

[0047] Figure 3 A schematic diagram illustrating the overall process of retrieving text W using data retrieval method M100 in some embodiments is shown. (See also...) Figure 3The text W is the text to be retrieved. Text W is vectorized to obtain a semantic vector y1. From the three established data categories K1, K2, and K3, the semantic similarity between semantic vector y1 and data categories K1, K2, and K3 is calculated respectively, and the data category with the highest semantic similarity is determined to be K1. A layer-by-layer retrieval is performed on the three-layer index structure contained in data category K1. The retrieval result q11 is obtained by searching for vector y1 in the top-level index structure c11. Based on result q11, a retrieval is performed in the middle-level index structure c12 to obtain retrieval result q12. Based on result q12, a retrieval is performed in the bottom-level index structure c13 to obtain retrieval result Q1, which is then used as the recall result and fed back to the client.

[0048] The clustering-based data retrieval method proposed in this disclosure first pre-clusters knowledge data to group semantically similar knowledge data into the same data category. Then, it narrows the retrieval scope by calculating the semantic similarity between the vector to be retrieved and each data category. Finally, it performs a fast retrieval only on vectors contained in one or more data categories with the highest similarity to obtain the recall results. The recall results can be directly fed back to the customer without the need for precise ranking of the recall results using a fine-ranking algorithm, thus reducing computational resource overhead and retrieval time.

[0049] This implementation method organically combines global semantic partitioning with efficient local retrieval. It only requires searching knowledge data within specific data categories, significantly reducing the search space for a single retrieval and avoiding blind searching of irrelevant data, thus improving retrieval speed. Simultaneously, by leveraging the hierarchical inclusion characteristics of multi-level indexes, it achieves efficient retrieval navigation from coarse to fine, further enhancing retrieval speed. Furthermore, due to the high semantic consistency within the same data category, it improves the accuracy and robustness of the retrieval results.

[0050] Therefore, compared with traditional semantic retrieval methods and two-stage retrieval methods that use coarse screening and fine ranking, this implementation method can meet the real-time interaction requirements of high concurrency and low latency, and can be applied in business scenarios with millisecond-level response. Even in scenarios where knowledge data is unevenly distributed and the categories are wide, it can still obtain recall results that are close to the true optimal answer.

[0051] Figure 4 A schematic diagram of the overall flow of a clustering-based data retrieval method M100 according to other embodiments of this disclosure is shown. See also... Figure 4 In step S110, the method of determining at least one data category with the highest semantic similarity to the data to be retrieved from multiple established data categories may specifically include steps S111 and S112.

[0052] S111, from the reference points corresponding to the multiple established data categories, determine at least one target reference point with the highest semantic similarity to the data to be retrieved. Here, the reference point can represent the central tendency of existing data in the corresponding data category in the semantic space.

[0053] S112, determine the data category corresponding to the target reference point as the target data category with the highest semantic similarity to the data to be retrieved.

[0054] A data category can correspond to one or more reference points. A reference point corresponds to a reference vector, which can reflect the central tendency of multiple vectors (such as text vectors) corresponding to the data category in the semantic space. That is, the reference vector is a representative vector that can characterize the overall semantic position of the data category.

[0055] In some implementations, reference points may include data categories as cluster centers. Since data categories are obtained through clustering, each data category corresponds to a cluster center. The vector at the cluster center location can serve as a reference point for that data category. The cluster center vector can be the centroid or median of the data category. It is understood that a data category can have multiple cluster center vectors; for example, if both the centroid and median are used as cluster centers, there will also be multiple reference points, and all reference points can be cluster centers, thereby achieving efficient vector retrieval based on clustering and cluster center navigation.

[0056] The centroid can be obtained by calculating the arithmetic mean of all vectors in the data category. For example, the mean can be calculated dimension by dimension for each vector to obtain the multidimensional centroid as the cluster center vector. The median can be obtained by calculating the position with the minimum sum of distances to all vectors. For example, the distances between each point in space and each vector can be calculated, the sum of the distances to each point can be calculated, and the point corresponding to the minimum sum of distances can be taken as the median.

[0057] In step S111, the distance between each reference point and the vector to be retrieved is calculated, and the one or more reference points with the closest distance are selected as the target reference points with the highest semantic similarity to the data to be retrieved. The distance between vectors can be calculated by methods such as cosine similarity. In step S112, the data category corresponding to the target reference point is the target data category to be retrieved. The multi-level index structure of the target data category will be retrieved layer by layer to form the final retrieval result. One or more target data categories can be determined, for example... Figure 3 In this context, the target data category includes only data category K1.

[0058] This implementation introduces reference points as representatives and references for categories, simplifying the category selection process to vector distance comparison. This quickly locates the data category to be retrieved, significantly reducing the search space and computational overhead. Simultaneously, the reference points accurately reflect the semantic center of the category, greatly reducing the probability of mistakenly entering an irrelevant category at the beginning of the overall retrieval, laying the foundation for subsequent efficient and accurate retrieval.

[0059] Continue reading Figure 4 In step S120, the method of performing a layer-by-layer retrieval of the multi-level index structure corresponding to the determined data category can be as follows: The target reference point of the target data category is used as the starting index node of the index structure to be retrieved first in the retrieval order, and the multi-level index structure corresponding to the target data category is retrieved layer by layer. Each layer of the index structure includes multiple index nodes. The index nodes correspond to existing data. The range of retrievable data is represented by the distribution of index nodes contained in the corresponding layer's index structure.

[0060] Figure 5 This diagram illustrates a hierarchical retrieval of the multi-level index structure for data category K1. (See also...) Figure 5 The multi-level index structure corresponding to the target data category K1 consists of three levels: c11, c12, and c13, retrieved in the order of retrieval. The first index structure retrieved in the retrieval order is the top-level index structure, c11. p0 is the vector to be retrieved. p1 is the target reference point for category K1.

[0061] In the multi-level index structure corresponding to the target data category, the target reference point is the index node in the index structure that is retrieved first in the retrieval order. That is, point p1 is an index node in the top-level c11. c11, c12, and c13 each include multiple index nodes, represented by solid circles and hollow circles. Each multi-level index structure corresponding to each data category can be constructed separately for the corresponding reference point.

[0062] The searchable data range is equivalent to the index range, which is represented by the set and distribution of nodes contained in this layer of the index structure. The position of each index node represents its semantic region, and the number of index nodes represents the coverage density. Figure 5 In the multi-level index structure shown, the higher levels have fewer nodes and a sparser distribution, resulting in a smaller index range suitable for fast navigation; the lower levels have more nodes and a denser distribution, resulting in a larger index range suitable for exact matching. Index nodes in the upper-level index structure also exist in the lower-level index structure. The number of index nodes in the index structure increases sequentially from the beginning to the end, following the retrieval order. The search proceeds from the higher levels to the lower levels using the hierarchical graph index structure and a layer-by-layer retrieval method.

[0063] The starting index node of the top-level c11 is point p1. Point p1 is used as the entry point for the overall multi-level index structure, and the search is performed layer by layer. Point p1 is taken as the search result in the top-level c11 and enters the intermediate layer c12 for retrieval using a greedy search algorithm. In the intermediate layer c12, the distances between point p0 and the index nodes p2, p3, and p4 associated with point p1 are calculated. It is determined that among point p1 and its associated nodes, point p1 is still the closest to point p0. Point p1 is taken as the search result in the intermediate layer c12 and enters the bottom-level c13 for retrieval using a greedy search algorithm. In the bottom-level c13, the distances between point p0 and the index nodes p2, p3, p4, and p5 associated with point p1 are calculated, filtered, and sorted in ascending order of distance to obtain the recall results {p1, p2, p5}.

[0064] This implementation directly uses reference points as index nodes in the top-level structure and as the retrieval entry point for the overall multi-level index structure. By comparing the distances between the vector to be retrieved and each reference point, the data category most semantically similar to the data to be retrieved is determined, realizing a cross-category navigation mechanism. This dynamically and accurately determines which categories should be retrieved, satisfying both coverage requirements and improving retrieval efficiency (without traversing all categories). Furthermore, a separate multi-level index structure is built for each data category. Compared to building a unified index structure for all data, this reduces the amount of data in the index structure and lowers memory usage.

[0065] In some implementations, multiple target data categories are identified in step S112, and correspondingly, multiple target reference points (reference points with the highest semantic similarity to the data to be retrieved) are also identified. Multiple multi-level index structures corresponding to multiple target reference points can be searched sequentially to obtain multiple search results for multiple multi-level index structures.

[0066] Figure 6 A schematic diagram of the overall flow of a clustering-based data retrieval method M100 according to other embodiments of this disclosure is shown. See also... Figure 6 In step S120, the method of performing a layer-by-layer retrieval of the multi-level index structure corresponding to the target data category can be specifically as follows: multiple multi-level index structures belonging to multiple starting index nodes are treated as multiple retrieval tasks, and multiple retrieval tasks are processed in parallel. The multi-level index structures contained in the retrieval tasks are retrieved in a layer-by-layer retrieval manner.

[0067] k starting index nodes (k≥2) are reference points with the highest semantic similarity (ranked highest) to the data to be retrieved. This allows the construction of k corresponding retrieval tasks, each corresponding to a multi-level index structure. Parallel processing is used to perform layer-by-layer searches on the multiple multi-level index structures using multiple independent threads or processes.

[0068] For the coarse screening + fine ranking retrieval method, in the coarse screening stage, due to the large span of data categories, the data of different categories may be far apart in semantic space, and the data density and distribution pattern within each category are significantly different. Therefore, a unified global index structure is difficult to take into account both the local structural characteristics within each category and the global distinguishing ability across categories. This causes the query vector to be easily guided to semantically irrelevant areas during the retrieval process, resulting in the omission of relevant results or the false recall of irrelevant results. Furthermore, the omission cannot be compensated for by fine ranking.

[0069] This implementation uses parallel processing to handle k retrieval tasks, which effectively avoids missed recall due to ambiguous cluster boundaries or cross-class query semantics, further ensuring the accuracy of retrieval results. It also utilizes the parallel capabilities of modern multi-core CPUs / GPUs, so that the overall response time does not increase linearly with the number of categories, thus balancing recall and real-time response.

[0070] Figure 7 A schematic diagram illustrating the overall process of retrieving text W using data retrieval method M100 in some other embodiments is shown. Figure 7 For example, in step S110, data categories K1 and K3 are first identified as target data categories. The cluster center vector of data category K2 has low similarity to the vector of text W and is therefore not identified as a target data category. Then, in step S120, two parallel searches are performed on the two multi-level index structures (category K1 and category K3). The first search thread sequentially searches index structures c11, c12, and c13 in category K1, obtaining search result Q1 for category K1. Simultaneously, the second search thread sequentially searches index structures c31, c32, and c33 in category K3, obtaining search result Q3 for category K3. From results Q1 and Q3, multiple vectors with the highest similarity (closest distance) to the vector to be searched are identified, resulting in multiple existing text data most similar to text W (recall result Q).

[0071] In some implementations, each index node in at least a subset of index nodes may have at least one neighboring index node. See also Figure 5 In the middle layer c12, node p1's neighbor index nodes include nodes p2, p3, and p4, while node p4's neighbor index nodes include nodes p1 and p2. The neighbor index nodes are represented by lines in the diagram. In the bottom layer c13, node p1's neighbor index nodes include node p5. Each node has a corresponding neighbor list in different layers.

[0072] The establishment of neighboring index nodes for an index node can be done in the same way as inserting a new node and establishing neighbor relationships in the ANN (Approximate Nearest Neighbor Search) algorithm or HNSW (Hierarchical Navigable Small World). The target index node used as the starting point for retrieval in a lower-level index structure can include index nodes from the retrieval results of the upper-level index structure. Each index structure has an index node used as the starting point for retrieval in its current level. The index node used as the starting point for retrieval in the top-level index structure is the starting index node of the entire multi-level index structure, i.e., the target reference point determined in step S110. For non-top-level index structures, such as intermediate and bottom levels, the index node used as the starting point for retrieval in this level can be called the target index node, which is the node passed from the upper-level index structure, i.e., a node from the retrieval results of the upper level.

[0073] In actual retrieval, the retrieval results of the current index structure can be determined by searching the neighboring index nodes of the target index node. For example, in the intermediate layer c12, for a node p1 passed from the previous layer, the neighboring nodes p2, p3, and p4 of node p1 can be searched (vector distance calculation) to obtain the retrieval results (determined by the magnitude of the vector distance).

[0074] Multiple existing data points most similar to the data to be retrieved can be identified from the index structure retrieved last in the search order. Figure 7 For example, search result Q1 is determined from the lowest level c13, search result Q3 is determined from the lowest level c33, and the final recall result Q is obtained from the lowest level index structure of one or more data categories.

[0075] By using a greedy algorithm to search for index nodes within a layer and their neighboring index nodes, we can avoid getting trapped in local optima and improve robustness. Even if the starting point is off-center, it can approach the correct node after a few jumps. Passing the search results layer by layer from the higher to the lower layers can avoid blindly exploring in dense graphs.

[0076] Figure 8 A schematic diagram of the overall flow of a clustering-based data retrieval method M100 according to other embodiments of this disclosure is shown. See also... Figure 8 Step S111 (determining at least one target reference point with the highest semantic similarity to the data to be retrieved from the reference points corresponding to the multiple established data categories) may specifically include steps S111a and S111b.

[0077] S111a, from the reference points corresponding to the multiple established data categories, determine the distance between the target semantic vector of the data to be retrieved and the reference semantic vector of the reference points.

[0078] S111b, determine at least one reference semantic vector that is closest to the target semantic vector, and use the reference point corresponding to the determined reference semantic vector as the target reference point.

[0079] The reference semantic vector can be the cluster center vector of the data category, such as the vector at the centroid or median. The target semantic vector is the vector obtained by encoding the data to be retrieved through an encoding model (such as the BERT model). The distance between the two vectors is calculated by methods such as Euclidean distance or cosine distance, and the target reference point is obtained by using the nearest reference semantic vector. Steps S111a and S111b provide a quantitative and computable way to select the most relevant category, ensuring the objectivity and reproducibility of the starting node selection.

[0080] The execution subject of the cluster-based data retrieval method in the specific embodiments of this disclosure can be an electronic device such as a computer or server.

[0081] Therefore, based on any of the above embodiments, this disclosure also provides an electronic device that can execute the clustering-based data retrieval method of any of the embodiments described above.

[0082] Figure 9 This is a schematic block diagram of an electronic device 1000 according to one embodiment of the present disclosure.

[0083] The hardware architecture of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0084] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but this does not imply that there is only one bus or one type of bus.

[0085] The processor 1200 can be a central processing unit (CPU). The processor 1200 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0086] The memory 1300 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions of the computer program in the embodiments of this disclosure. The processor 1200 implements a clustering-based data retrieval method by running the non-transitory software programs, instructions, and modules stored in the memory 1300.

[0087] The memory 1300 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created by the processor 1200. Furthermore, the memory 1300 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1300 may optionally include memory remotely located relative to the processor 1200, and these remote memories may be connected to the processor 1200 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0088] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0089] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the processes or functions of this disclosure are performed wholly or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, OAM, or other programmable device.

[0090] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0091] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0096] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0097] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A clustering-based data retrieval method, characterized in that, include: From multiple established data categories, determine at least one data category with the highest semantic similarity to the data to be retrieved, wherein the data category is obtained by clustering existing data; as well as The multi-level index structure corresponding to the determined data category is searched layer by layer to obtain multiple existing data that are most similar to the data to be searched and feedback is provided. Among two index structures that are adjacent in the search order within the same multi-level index structure, the searchable data range of the lower index structure searched later is included in the searchable data range of the upper index structure searched earlier. The search of the lower index structure is based on the search results of the upper index structure.

2. The data retrieval method according to claim 1, characterized in that, From multiple established data categories, identify at least one data category with the highest semantic similarity to the data to be retrieved, including: From multiple established reference points corresponding to data categories, at least one target reference point with the highest semantic similarity to the data to be retrieved is determined. This reference point represents the central tendency of existing data within the corresponding data category in the semantic space. The data category corresponding to the target reference point is determined to be the target data category with the highest semantic similarity to the data to be retrieved.

3. The data retrieval method according to claim 2, characterized in that, A layer-by-layer retrieval is performed on the multi-level index structure corresponding to the identified data categories, including: The target reference point of the target data category is used as the starting index node of the index structure that is searched first in the search order. The multi-level index structure corresponding to the target data category is searched layer by layer. Each index structure includes multiple index nodes, and the index nodes correspond to the existing data. The range of searchable data is represented by the distribution of index nodes contained in the index structure of the corresponding layer.

4. The data retrieval method according to claim 3, characterized in that, There are multiple target data categories; a layer-by-layer retrieval is performed on the multi-level index structure corresponding to the target data categories, including: The multiple multi-level index structures to which the multiple starting index nodes belong are used as multiple retrieval tasks. The multiple retrieval tasks are processed in parallel, and the multi-level index structures contained in the retrieval tasks are retrieved in a layer-by-layer retrieval manner.

5. The retrieval method according to claim 3, characterized in that, At least some of the index nodes have at least one neighboring index node. The target index node of the lower-level index structure used as the starting point for retrieval at this level includes the index node in the retrieval result of the upper-level index structure. The retrieval result of the current-level index structure is determined by retrieving the neighboring index nodes of the target index node.

6. The data retrieval method according to claim 2, characterized in that, Identify at least one target reference point with the highest semantic similarity to the data to be retrieved, including: Determine the distance between the target semantic vector of the data to be retrieved and the reference semantic vector of the reference point; and Determine at least one reference semantic vector that is closest to the target semantic vector, and use the reference point corresponding to the determined reference semantic vector as the target reference point.

7. The data retrieval method according to claim 2, characterized in that, The reference point includes the data category as the cluster center of the cluster.

8. The data retrieval method according to claim 1 or 2, characterized in that, The multiple existing data that are most similar to the data to be retrieved are all determined from the index structure that is retrieved last in the retrieval order.

9. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the data retrieval method according to any one of claims 1 to 8.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, is used to implement the data retrieval method according to any one of claims 1 to 8.

Images