Data query methods and apparatus, electronic devices, computer-readable storage media

Abstract

This disclosure relates to a data query method and apparatus, electronic device, and computer-readable storage medium. The data query method includes: encoding attribute information of a vector to be retrieved to obtain corresponding encoded data, which serves as the retrieval encoded data; obtaining a set of reference vectors, including multiple reference vectors, each associated with reference encoded data, wherein the reference encoded data is encoded data obtained by encoding the attribute information of the associated reference vectors; and combining the retrieval encoded data and the reference encoded data to determine a target vector corresponding to the vector to be retrieved from the set of reference vectors, wherein the target vector is used for data query. By encoding attribute information into encoded data, the encoded data can participate in the retrieval calculation, optimizing the retrieval results using attributes, improving recall, and transforming multi-attribute constraint problems into single-attribute constraint problems, thus ensuring retrieval efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning technology, and in particular to a data query method and apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Thanks to the rapid development of deep neural networks, we now have the ability to abstract unstructured data into vectors, enabling us to query unstructured data through vector retrieval. However, performing precise retrieval on such massive amounts of high-dimensional data requires significant time investment. To address this, many researchers have proposed vector retrieval algorithms based on approximate nearest neighbor retrieval. Approximate nearest neighbor retrieval algorithms can perform k-nearest neighbor retrieval on vectors in a very short time.

[0003] However, with the development of modern recommender systems, we need to perform retrieval not only on vectors but also on attributes, such as data categories or, in e-commerce, the merchant to which a product belongs. Existing attribute-based vector retrieval methods mainly fall into two categories. One is to first calculate similarity based on vectors and then filter by attributes. However, since similar vectors do not necessarily have the same attribute labels, the recall rate is often relatively low. The other is to first filter by attributes and then only retrieve data within the vector domain that meets the conditions. While this can guarantee a high recall rate, it relies on pre-filtering and selection of attributes, requiring additional attribute retrieval, incurring speed overhead, and reducing retrieval efficiency. Furthermore, as the number of attributes increases, the filtering efficiency further decreases, failing to guarantee retrieval efficiency. Therefore, existing attribute-based vector retrieval methods often cannot balance recall and retrieval efficiency, and the overall retrieval performance needs improvement. Summary of the Invention

[0004] This disclosure provides a data query method and apparatus, electronic device, and computer-readable storage medium to at least solve the problem of how to improve the performance of attribute vector retrieval in the related art, or it may not solve any of the above problems.

[0005] According to a first aspect of this disclosure, a data query method is provided, comprising: encoding attribute information of a vector to be retrieved to obtain corresponding encoded data, which serves as the retrieval encoded data; obtaining a set of reference vectors, the set of reference vectors including multiple reference vectors, each reference vector being associated with reference encoded data, the reference encoded data being encoded data obtained by encoding the attribute information of the associated reference vector; and combining the retrieval encoded data and the reference encoded data to determine a target vector corresponding to the vector to be retrieved from the set of reference vectors, the target vector being used for data query.

[0006] Optionally, the step of encoding the attribute information of the vector to be retrieved to obtain a corresponding encoded data as the retrieval encoded data includes: querying the encoded data set for the attribute combination formed by the attribute information of the vector to be retrieved, and obtaining the encoded data corresponding to the attribute information of the vector to be retrieved, as the retrieval encoded data, wherein the encoded data set stores attribute combinations and encoded data in a one-to-one correspondence.

[0007] Optionally, the encoded data set is generated through the following steps: obtaining multiple attribute combinations; determining the similarity between the multiple attribute combinations; assigning corresponding encoded data to each attribute combination, wherein the difference between the encoded data of any two attribute combinations is negatively correlated with the similarity between the two attribute combinations.

[0008] Optionally, the reference vector set is generated through the following steps: obtaining the plurality of reference vectors; constructing a composite nearest neighbor graph based on the plurality of reference vectors as the reference vector set, wherein: the plurality of reference vectors represent a plurality of image points in the composite nearest neighbor graph, denoted as reference points; each reference point is a neighbor point of at least one other reference point, the other reference points being reference points in the composite nearest neighbor graph other than the current reference point; each reference point and at least one neighbor point of each reference point are similar points, the similar points being points whose encoded data satisfy a preset relationship, the preset relationship being used to indicate that two encoded data are the same or similar.

[0009] Optionally, the step of combining the coded data to be retrieved and the reference coded data to determine the target vector corresponding to the target vector from the set of reference vectors includes: taking any reference point in the composite nearest neighbor graph as a parent candidate point, and determining the vector distance between the reference vectors corresponding to all neighbor points of the parent candidate point and the target vector; based on the vector distance, selecting a preset number of neighbor points from all neighbor points as child candidate points of the parent candidate point; taking each child candidate point that does not meet the truncation condition as a new parent candidate point, and for each new parent candidate point, repeatedly performing the process of determining the reference vectors corresponding to all neighbor points of the parent candidate point and the target vector; The process involves determining the vector distance between the vectors to be retrieved, and selecting a preset number of neighboring points from all neighboring points based on the vector distance as child candidate points of the parent candidate point. The truncation condition indicates that the difference between the reference encoded data corresponding to the child candidate point and the encoded data to be retrieved is greater than a difference threshold, and the difference between the reference encoded data corresponding to the N consecutive parent candidate points preceding the child candidate point and the encoded data to be retrieved is also greater than the difference threshold, where N is a preset value. The process continues until a termination condition is met, at which point the target vector is selected from all the reference vectors corresponding to the parent candidate points. The termination condition indicates that no new child candidate points can be obtained.

[0010] Optionally, the composite nearest neighbor graph is constructed through the following steps: constructing a nearest neighbor graph as a baseline graph based on the vector distances between the plurality of reference vectors; dividing the set of reference vectors into multiple subsets based on the reference encoded data, constructing a nearest neighbor graph for each subset as an attribute graph; and combining the neighbor relationships in the baseline graph and the attribute graph to obtain the composite nearest neighbor graph.

[0011] Optionally, dividing the reference vector set into multiple subsets based on the reference encoded data includes: assigning reference vectors with the same reference encoded data in the reference vector set to the same subset to obtain multiple initial subsets; merging the initial subsets with fewer than a number threshold into corresponding approximate subsets to obtain the multiple subsets, wherein the attribute similarity between the reference encoded data of the initial subset and the reference encoded data of the corresponding approximate subset is greater than or equal to a similarity threshold.

[0012] Optionally, the step of combining the coded data to be retrieved and the reference coded data to determine the target vector corresponding to the vector to be retrieved from the reference vector set includes: determining reference coded data in the reference vector set that matches the coded data to be retrieved based on the coded data to be retrieved; obtaining a candidate vector set based on the reference vectors corresponding to the matching reference coded data; and determining the target vector corresponding to the vector to be retrieved from the candidate vector set.

[0013] According to a second aspect of this disclosure, a data query apparatus is provided, comprising: an encoding unit configured to perform encoding processing on attribute information of a vector to be retrieved to obtain corresponding encoded data as the retrieval encoded data; an acquisition unit configured to acquire a set of reference vectors, the set of reference vectors including multiple reference vectors, each reference vector being associated with reference encoded data, wherein a set of reference encoded data is encoded data obtained by encoding the attribute information of the associated reference vector; and a retrieval unit configured to combine the retrieval encoded data and the reference encoded data to determine a target vector corresponding to the vector to be retrieved from the set of reference vectors, the target vector being used for data query.

[0014] Optionally, the encoding unit is further configured to perform an attribute combination based on the attribute information of the vector to be retrieved, query the encoded data set, and obtain encoded data corresponding to the attribute information of the vector to be retrieved, as the encoded data to be retrieved, wherein the encoded data set stores attribute combinations and encoded data in a one-to-one correspondence.

[0015] Optionally, the encoded data set is generated through the following steps: obtaining multiple attribute combinations; determining the similarity between the multiple attribute combinations; assigning corresponding encoded data to each attribute combination, wherein the difference between the encoded data of any two attribute combinations is negatively correlated with the similarity between the two attribute combinations.

[0016] Optionally, the reference vector set is generated through the following steps: obtaining the plurality of reference vectors; constructing a composite nearest neighbor graph based on the plurality of reference vectors as the reference vector set, wherein: the plurality of reference vectors represent a plurality of image points in the composite nearest neighbor graph, denoted as reference points; each reference point is a neighbor point of at least one other reference point, the other reference points being reference points in the composite nearest neighbor graph other than the current reference point; each reference point and at least one neighbor point of each reference point are similar points, the similar points being points whose encoded data satisfy a preset relationship, the preset relationship being used to indicate that two encoded data are the same or similar.

[0017] Optionally, the retrieval unit is further configured to perform the following steps: taking any reference point in the composite nearest neighbor graph as a parent candidate point, determining the vector distance between the reference vectors corresponding to all neighbor points of the parent candidate point and the vector to be retrieved; selecting a preset number of neighbor points from all neighbor points as child candidate points of the parent candidate point based on the vector distance; taking each child candidate point that does not meet the truncation condition as a new parent candidate point, and for each new parent candidate point, repeatedly performing the steps of determining the vector distance between the reference vectors corresponding to all neighbor points of the parent candidate point and the vector to be retrieved, and selecting a preset number of neighbor points from all neighbor points as child candidate points of the parent candidate point; taking each child candidate point that does not meet the truncation condition as a new parent candidate point, and repeatedly performing the following steps based on the vector distance: The step involves selecting a preset number of neighboring points from all neighboring points as child candidate points of the parent candidate point. The truncation condition indicates that the difference between the reference encoded data corresponding to the child candidate point and the encoded data to be retrieved is greater than a difference threshold, and the difference between the reference encoded data corresponding to the N consecutive parent candidate points preceding the child candidate point and the encoded data to be retrieved is also greater than the difference threshold, where N is a preset value. The process continues until a termination condition is met, at which point the target vector is selected from the reference vectors corresponding to all the parent candidate points. The termination condition indicates that no new child candidate points can be obtained.

[0018] Optionally, the composite nearest neighbor graph is constructed through the following steps: constructing a nearest neighbor graph as a baseline graph based on the vector distances between the plurality of reference vectors; dividing the set of reference vectors into multiple subsets based on the reference encoded data, constructing a nearest neighbor graph for each subset as an attribute graph; and combining the neighbor relationships in the baseline graph and the attribute graph to obtain the composite nearest neighbor graph.

[0019] Optionally, dividing the reference vector set into multiple subsets based on the reference encoded data includes: assigning reference vectors with the same reference encoded data in the reference vector set to the same subset to obtain multiple initial subsets; merging the initial subsets with fewer than a number threshold into corresponding approximate subsets to obtain the multiple subsets, wherein the attribute similarity between the reference encoded data of the initial subset and the reference encoded data of the corresponding approximate subset is greater than or equal to a similarity threshold.

[0020] Optionally, the retrieval unit is further configured to perform the following operations: determining reference encoded data in the reference vector set that matches the coded data to be retrieved based on the coded data to be retrieved; obtaining a candidate vector set based on the reference vectors corresponding to the matching reference encoded data; and determining the target vector corresponding to the vector to be retrieved from the candidate vector set.

[0021] According to a third aspect of this disclosure, an electronic device is provided, the electronic device comprising: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform a data query method according to this disclosure.

[0022] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by at least one processor, causes the at least one processor to perform a data query method according to this disclosure.

[0023] According to a fifth aspect of this disclosure, a computer program product is provided, including computer instructions that, when executed by at least one processor, implement a data query method according to this disclosure.

[0024] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0025] According to the data query method and data query apparatus of the embodiments of this disclosure, by encoding the attribute information of a vector into encoded data, the encoded data can be made to participate in the retrieval calculation, thereby optimizing the retrieval results using attributes and improving the recall rate. Furthermore, since all attributes of the vector can be represented by a single encoded data regardless of the number of attributes involved, a multi-attribute constraint problem can be transformed into a single-attribute constraint problem, allowing the retrieval efficiency to remain at a high level as the number of attributes increases. Therefore, this disclosure can balance recall and retrieval efficiency.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0028] Figure 1 This is a flowchart illustrating an attribute-based vector retrieval method in related technologies;

[0029] Figure 2 This is a flowchart illustrating another attribute-based vector retrieval method in related technologies;

[0030] Figure 3 This is a flowchart illustrating another attribute-based vector retrieval method in related technologies;

[0031] Figure 4This is a flowchart illustrating a data query method according to an exemplary embodiment of the present disclosure;

[0032] Figure 5 This is a schematic diagram illustrating a reference diagram according to a specific embodiment of the present disclosure;

[0033] Figures 6a to 6c This is a schematic diagram illustrating a property diagram according to a specific embodiment of the present disclosure;

[0034] Figure 7 This is a schematic diagram illustrating a composite nearest neighbor graph according to a specific embodiment of the present disclosure;

[0035] Figure 8 This is a block diagram illustrating a data query apparatus according to an exemplary embodiment of the present disclosure;

[0036] Figure 9 This is a block diagram illustrating an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0037] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0038] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0039] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0040] It should be noted that the user information involved in this disclosure (including but not limited to user device information, user personal information, etc.) is all information authorized by the user or fully authorized by all parties.

[0041] Thanks to the rapid development of deep neural networks, we now have the ability to abstract unstructured data into vectors, enabling us to query unstructured data through vector retrieval. However, performing precise retrieval on such massive amounts of high-dimensional data requires significant time investment. To address this, many researchers have proposed vector retrieval algorithms such as Approximate Nearest Neighbor Search (ANNS), which retrieves the top-k nearest neighbors of a given query vector from an existing database. ANNS algorithms can perform k-nearest neighbor retrieval on a vector in a very short time. However, with the development of modern recommender systems, we need to incorporate not only vector retrieval but also attribute constraints, such as data categories or, in e-commerce, the merchant to which a product belongs.

[0042] For ordinary vector retrieval, there are currently two main methods. One is to quantize and compress the vectors to reduce the data size of each vector, and then calculate the distance between vectors, speeding up data processing by sacrificing some data precision. The other is to use a graph indexing method, constructing a nearest neighbor graph. A vector is represented by a point in the nearest neighbor graph, and for each vector, other vectors with the closest distance are considered its nearest neighbors (also called neighboring points), connected by edges. Retrieval is performed in the nearest neighbor graph, thus speeding up the retrieval process. A typical graph indexing method involves first randomly selecting a point in the nearest neighbor graph as the entry point, then calculating the distances between the entry point and all its neighbors to the vector to be retrieved. Several nearest neighboring points are selected from all neighbors, and the entry point, the selected neighbors, and their distance values ​​are added to a queue. Then, the distances between the neighbors of these neighbors and the vector to be retrieved are calculated again, iterating through the points in the graph index until a termination condition is met. The termination condition is usually that the distances between all neighbors of the current neighbor and the vector to be retrieved are greater than the minimum distance in the queue. During this process, points whose distances have already been calculated are not recalculated.

[0043] For vector retrieval with attributes, there are currently two main methods. One is as follows: Figure 1 As shown in the figure (hollow circles represent the vector to be retrieved, and solid circles represent the attributes of the vector to be retrieved), Figure 2 and Figure 3Similarly (and without further explanation), this approach still uses vector similarity calculations to perform ordinary vector retrieval, and then filters based on attributes to obtain the top-k vectors as the returned results. This method only requires expanding the candidate results of vector retrieval and can be easily extended to existing vector retrieval schemes, using either quantization compression methods or graph indexes. However, many existing vector retrieval methods are very sensitive to the size of the candidate set; an expanded candidate set size will lead to a significant decrease in search performance. Furthermore, since similar vectors are not necessarily data with the same attribute labels, this approach will result in a relatively low recall rate (the probability of finding the top-1 vector, i.e., the vector closest to the target vector), and the recall capability will significantly decrease as the number of attribute combinations increases. Another approach... Figure 2 As shown, this method first filters using attributes, then searches only within the vector domain that meets the conditions. For this approach, some scholars have proposed an attribute query processing framework based on PQ (Product Quantization), which pre-compresses and stores vectors using PQ. During the query, all vectors that meet the attribute conditions are first scanned and filtered out before the search is performed. While this method can guarantee a high recall rate, it relies on pre-filtering and selection of attributes, requiring additional attribute searches, incurring speed overhead, reducing search efficiency, and impacting performance to some extent. This performance impact increases significantly with the number of attributes. Furthermore, since different vectors to be searched have different attributes, the vectors that meet the conditions are different after each filtering. Graph indexes require pre-construction of the graph; if time is spent rebuilding the graph after each filtering, it will severely reduce search efficiency, making this method unsuitable for graph indexes and detrimental to improving search efficiency. Besides the two main methods mentioned above, some scholars have proposed a different approach... Figure 3 The illustrated graph-indexed attribute query processing framework combines vector distance and attribute distance. Vector distance uses the inner product distance, while attribute distance uses the XOR distance. In this hybrid distance, vector distance dominates, with attribute distance only fine-tuned. This approach fails to prioritize attribute distance in the hybrid distance formula, leading to a sharp decline in recall as the number of attributes increases. This also means that when applied to methods like PQ, the formula cannot guarantee that recalled vectors will necessarily satisfy the attribute conditions.

[0044] According to the exemplary embodiments of the present disclosure, the data query method and apparatus encode the attribute information of a vector into encoded data, making the encoded data capable of participating in the retrieval calculation, thereby optimizing the retrieval results using attributes and improving recall. Furthermore, since all attributes of the vector can be represented by a single encoded data regardless of the number of attributes involved, a multi-attribute constraint problem can be transformed into a single-attribute constraint problem, allowing retrieval efficiency to remain at a high level as the number of attributes increases. Therefore, the present disclosure can balance recall and retrieval efficiency.

[0045] Below, we will refer to Figures 4 to 8 This invention provides a detailed description of a data query method and a data query apparatus according to exemplary embodiments of the present disclosure.

[0046] Figure 4 This is a flowchart illustrating a data query method according to an exemplary embodiment of the present disclosure. It should be understood that the data query method according to an exemplary embodiment of the present disclosure can be implemented in terminal devices such as smartphones, tablets, and personal computers (PCs), or in devices such as servers.

[0047] Reference Figure 4 In step 401, the attribute information of the vector to be retrieved is encoded to obtain corresponding encoded data, which serves as the retrieval encoded data. It should be understood that the vector to be retrieved represents the data to be queried in the data query; that is, unstructured data to be queried can be abstracted into a vector to be retrieved. Of course, structured data to be queried can also be abstracted into a vector to be retrieved. It should be noted that the encoded attribute information is the attribute information that needs to participate in the data query from all the attribute information of the vector to be retrieved. It can be all the attribute information of the vector to be retrieved, or it can be determined according to preset rules; this disclosure does not impose any restrictions on this.

[0048] Optionally, step 401 includes: querying the encoded data set for the attribute combinations formed by the attribute information of the vector to be retrieved, and obtaining the encoded data corresponding to the attribute information of the vector to be retrieved, as the retrieval encoded data, wherein the encoded data set stores a one-to-one correspondence between attribute combinations and encoded data. By constructing the encoded data set in advance according to the attribute combinations and matching the encoded data corresponding to the attribute information of the vector to be retrieved, it is not necessary to directly convert the data from the original attribute information, which can reduce the size of the encoded data, reduce the amount of computation, help to reliably maintain retrieval efficiency when the number of attributes increases, and effectively compress storage space and reduce storage costs.

[0049] Optionally, the encoded dataset is generated through the following steps: obtaining multiple attribute combinations; determining the similarity between the multiple attribute combinations; and assigning corresponding encoded data to each attribute combination, wherein the difference between the encoded data of any two attribute combinations is negatively correlated with the similarity between these two attribute combinations. By assigning encoded data according to the similarity between attribute combinations, the difference in the encoded data of more similar attribute combinations is smaller. This allows the differences between attribute combinations to be reflected by the differences between the encoded data, enabling subsequent retrieval to prioritize not only vectors with the same attribute but also vectors with similar attributes. This reduces the interference of vectors with similar content but large attribute differences on the retrieval results, helping to improve the retrieval recall rate. As an example, the obtained multiple attribute combinations can be sorted according to similarity, with more similar attribute combinations being sorted closer together. That is, the difference between the indices of any two attribute combinations is negatively correlated with the similarity between these two attribute combinations. The indices of the attribute combinations are then used as the encoded data for that attribute combination. This approach not only uses short indices as encoded data, significantly reducing the size of the encoded data, but also utilizes the proximity of the indices to reflect the differences between attributes. Furthermore, the encoded data can be consecutively numbered, for example, n attribute combinations can correspond to numbers from 1 to n. Alternatively, when initially constructing the encoded data set, the encoded data can use non-consecutive numbers, such as, but not limited to, multiples of 2 (i.e., even numbers), multiples of 3, and multiples of 5, thus leaving space to expand the number of attribute combinations in the encoded data set. In addition, it should be understood that the difference between the encoded data of two attribute combinations only needs to reflect the degree of difference between them to reflect the similarity between the corresponding attribute combinations. As an example, it could be the absolute value of the difference between the encoded data of two attribute combinations, or the logarithm of the absolute value of the difference between the encoded data of two attribute combinations; this disclosure does not impose any limitations on this.

[0050] Taking the example of encoded data being sorted sequence numbers, the process of establishing an encoded data set is as follows:

[0051] First, organize all attribute combinations involved in the reference vector set. For ease of comparison, the different attribute types can be arranged in order, and each attribute can be assigned a corresponding value range. For example, 0 represents that the attribute combination does not contain that attribute type, and then a value between 1 and x represents that the attribute combination contains that attribute type and what the specific attribute is. For example, for the merchant attribute in the e-commerce field, a single value can be used to represent a specific merchant. Thus, an attribute combination can be assigned a value sequence based on the attribute types and specific attribute content it contains.

[0052] Then, based on the number of attribute types contained in the attribute combination, all attribute combinations are grouped. For example, if the reference vector set involves a total of 4 attributes, the attribute combinations can be divided into 4 groups. Each attribute combination in the first group contains all 4 attributes, each attribute combination in the second group contains 3 of them, and the attribute combinations in the third and fourth groups contain 2 and 1 of them, respectively.

[0053] Finally, for each group, a single attribute combination is randomly selected as the first attribute combination in that group. The index of this attribute combination is based on the last index of the previous group. Then, the similarity between the other attribute combinations in the group and this attribute combination is calculated. The similarity is, for example, but not limited to, the XOR distance between the value sequences of the two attribute combinations. The other attribute combinations are then sorted in descending order of similarity.

[0054] In step 402, a set of reference vectors is obtained. The set of reference vectors includes multiple reference vectors, each of which is associated with reference encoded data. The reference encoded data is encoded data obtained by encoding the attribute information of the associated reference vector. Slightly different from the encoding process in step 401, the reference encoded data can be obtained simultaneously when the set of encoded data is established.

[0055] It should be understood that when the encoded data set is expandable, the reference vector set is also expandable accordingly. When a new reference vector appears in the reference vector set, and the attribute combination formed by the attribute information of the reference vector is an attribute combination that does not appear in the encoded data set (it could be that a certain attribute has new content or a new attribute type has appeared), the sequence number of the new attribute combination, i.e. the encoded data, is determined according to the sorting rules when the encoded data set was established, and it is added to the encoded data set.

[0056] The following describes a specific embodiment of building an encoded dataset based on a set of reference vectors. In this embodiment, attribute combinations consist of all attribute information of the corresponding vectors, and the attribute combinations are sorted according to similarity. The encoded data is the sequence number of the attribute combination.

[0057] The reference vector set contains a total of 30 reference vectors, involving 4 attribute categories. Each attribute category can take values ​​of 0-3, 0-4, 0-2, and 0-6, respectively. Among these reference vectors, 5 reference vectors have completely identical attribute combinations, while the remaining 25 reference vectors have different attribute combinations. Therefore, 26 attribute combinations can be identified, and each attribute combination's value sequence consists of 4 possible values.

[0058] Then, based on the number of non-zero values ​​in the value sequence, these 26 attribute combinations are divided into 4 groups. In the first group, each value sequence contains no non-zero values, meaning each attribute combination contains 4 attribute categories. In the second group, each value sequence contains 3 non-zero values ​​and 1 zero, meaning each attribute combination contains 3 attribute categories. In the third group, each value sequence contains 2 non-zero values ​​and 2 zeros, meaning each attribute combination contains 2 attribute categories. In the fourth group, each value sequence contains 1 non-zero value and 3 zeros, meaning each attribute combination contains 1 attribute category. For example, among these 26 attribute combinations, the value sequence of attribute combination A is 0, 1, 2, 3; the value sequence of attribute combination B is 0, 1, 2, 4; the value sequence of attribute combination C is 0, 1, 1, 2; and the value sequence of attribute combination D is 3, 1, 2, 5. Then, attribute combination D is assigned to the first group, and attribute combinations A, B, and C are assigned to the second group. The other unlisted attribute combinations are grouped in the same way, resulting in a total of 3 attribute combinations in the first group, 8 attribute combinations in the second group, 10 attribute combinations in the third group, and 5 attribute combinations in the fourth group.

[0059] Finally, the sequence number of each attribute combination is determined as its encoded data. For the first group, an attribute combination (e.g., attribute combination D) is randomly selected as the attribute combination with sequence number 1. For the other attribute combinations in the first group, they are sorted in descending order according to the XOR distance between their respective value sequences and the value sequences of the attribute combination with sequence number 1, thus obtaining the sequence numbers of the attribute combinations in the first group. For the second group, an attribute combination (e.g., attribute combination A) is randomly selected as the attribute combination with sequence number 4, and the other attribute combinations in the group are sorted in the same way as the first group. Among them, for the aforementioned attribute combinations B and C, it is not difficult to find that attribute combination B has 3 attribute values ​​that are the same as attribute combination A, while attribute combination C only has 2 attribute values ​​that are the same as attribute combination A. Therefore, attribute combination B is closer to attribute combination A than attribute combination C, so attribute combination B ranks higher and has a smaller sequence number. For example, the sequence number of attribute combination B is 5, and the sequence number of attribute combination C is 9. The third and fourth groups are also sorted in the same way to obtain sequence numbers as encoded data, which will not be described in detail.

[0060] In step 403, combining the coded data to be retrieved and the reference coded data, a target vector corresponding to the coded data to be retrieved is determined from the reference vector set. The target vector is used for data querying. It should be understood that the data represented by the target vector is the target data to be retrieved, thereby realizing the query of the data to be queried.

[0061] For step 403, two types of retrieval strategies can be adopted, which will be introduced separately below.

[0062] The first type is graph indexing strategies.

[0063] In the graph indexing strategy, optionally, the reference vector set is generated through the following steps: obtaining multiple reference vectors; constructing a composite nearest neighbor graph based on the multiple reference vectors as the reference vector set, wherein: the multiple reference vectors represent multiple image points in the composite nearest neighbor graph, denoted as reference points; each reference point is a neighbor of at least one other reference point, and the other reference points are reference points in the composite nearest neighbor graph other than the current reference point; each reference point is a similar point to at least one neighbor of each reference point, and similar points are points whose encoded data satisfy a preset relationship, which is used to indicate that two encoded data are the same or similar. By combining the preset relationship between similar points to assist in constructing the nearest neighbor graph, a composite nearest neighbor graph with neighborhood information of the same or similar attributes can be obtained, associating reference vectors with the same or similar attributes, making it easy to quickly locate reference vectors with the same or similar attributes as the vector to be retrieved, thus introducing attribute-based queries into the graph index. While using the graph index to improve retrieval efficiency, the attribute query ensures the high quality of the vector neighborhood in the graph index, which helps to improve the recall rate and realizes an efficient and fast retrieval process. It should be understood that, depending on the actual retrieval situation, the preset relationship may only indicate that two encoded data are identical, that is, similar points are reference points with the same attributes, or it may simultaneously include cases where two encoded data are identical and similar. For the latter, each reference point may have both similar points with the same attributes and similar points with similar attributes, or some reference points may have both similar points with the same attributes and similar points with similar attributes, while others may only have similar points with the same attributes, and still others may only have similar points with similar attributes. As an example, similar points with the same attributes can be prioritized for each reference point, and when the number of similar points with the same attributes is small or nonexistent, a suitable number of similar points with similar attributes can be introduced. This disclosure does not impose any limitations on this.

[0064] Optionally, when using a graph indexing strategy, step 403 may include: taking any reference point in the composite nearest neighbor graph as a parent candidate point, determining the vector distance between the reference vectors corresponding to all neighbor points of the parent candidate point and the vector to be retrieved; selecting a preset number of neighbor points from all neighbor points as child candidate points of the parent candidate point based on the vector distance; taking each child candidate point that does not meet the truncation condition as a new parent candidate point, and for the new parent candidate point, repeating the steps of determining the vector distance between the reference vectors corresponding to all neighbor points of the parent candidate point and the vector to be retrieved, and selecting a preset number of neighbor points from all neighbor points as child candidate points of the parent candidate point based on the vector distance, wherein the truncation condition is used to indicate that the difference between the reference encoded data corresponding to the child candidate point and the encoded data to be retrieved is greater than the difference threshold, and the difference between the reference encoded data corresponding to the consecutive N levels of parent candidate points before the child candidate point and the encoded data to be retrieved is greater than the difference threshold, where N is a preset value; until the termination condition is met, selecting the target vector from the reference vectors corresponding to all parent candidate points, where the termination condition indicates that no new child candidate point can be obtained.

[0065] When using a composite nearest neighbor graph according to an exemplary embodiment of this disclosure to implement vector retrieval, the retrieval process, in addition to the conventional step-by-step filtering of reference points by combining vector distance and termination conditions, also includes determining whether to continue exploring a reference point that has passed the filtering based on attribute truncation conditions. This means determining whether to filter its neighboring points (as an example, to reduce computation, the vector distance of each reference point is not repeatedly determined; based on this, determining whether to continue exploring a reference point that has passed the filtering is equivalent to determining whether to filter its neighboring points whose vector distances have not been calculated). This fully utilizes attributes to accelerate the retrieval process, reduce computation, and further improve retrieval efficiency. The truncation condition indicates that the difference between the reference encoded data corresponding to the current sub-candidate point and its preceding N consecutive parent candidate points and the encoded data to be retrieved is greater than a difference threshold. In other words, the attributes of the current sub-candidate point and its preceding N consecutive parent candidate points are different from and not similar to the attributes of the vector to be retrieved. At this point, it can be reliably considered that the neighboring points of this sub-candidate point are also highly likely to be different from and not similar to the attributes of the vector to be retrieved, and no further filtering of its neighboring points is necessary. By configuring this cutoff condition, the computational load can be reduced while ensuring search performance. It should be understood that the value of N can be reasonably configured based on the number of reference points in the composite nearest neighbor graph and the required computational accuracy.

[0066] Specifically, the first parent candidate point serves as the entry point for the search and can be randomly selected. The search process is as follows: after selecting the entry point, all its neighboring points are filtered based on vector distance. The filtered neighboring points become the child candidate points of the entry point. Next, each of these child candidate points is checked to see if it meets the truncation condition. If the truncation condition is met, the child candidate point is discarded, meaning it is not considered a new parent candidate point. If the truncation condition is not met, the current child candidate point is taken as a new parent candidate point, and the search can continue, meaning the search continues to select new child candidate points of the parent candidate point. Here, the search can continue immediately after each new parent candidate point is determined, or it can be done after all new parent candidate points have been determined. This disclosure does not impose any restrictions on this. During the downward exploration, if the termination condition is met, meaning no new child candidate point can be obtained (i.e., the exploration fails), the exploration of that new parent candidate point stops. If the termination condition is not met, it indicates that exploration can continue. In this case, exploration can continue until the path meets the termination condition, or it can pause and explore other new parent candidate points at the same level as the new parent candidate point. That is, all new parent candidate points at the same level are explored at the first level. This disclosure does not impose any restrictions on this. This step-by-step exploration continues until all paths meet the termination condition. Then, the exploration stops, and the reference vectors corresponding to all previously obtained parent candidate points are used as candidate vectors. The top-k vectors with the smallest vector distance are selected as the target vector.

[0067] Optionally, the composite nearest neighbor graph is constructed through the following three steps:

[0068] The steps for constructing the benchmark graph are as follows: Specifically, a nearest neighbor graph is constructed based on the vector distances between multiple reference vectors, serving as the benchmark graph. The construction of the nearest neighbor graph is a mature technology in this field and will not be elaborated upon here.

[0069] The attribute graph construction steps are as follows: First, based on the reference encoding data, the set of reference vectors is divided into multiple subsets. Then, a nearest neighbor graph is constructed for each subset, serving as the attribute graph. By grouping the reference vectors using the reference encoding data and constructing attribute graphs separately, each attribute graph retains only the neighborhood information of reference vectors with the same or similar attributes. This obtains the neighbor relationships between reference vectors with the same or similar attributes, thus compensating for the lack of attribute information in the baseline graph. It should be understood that if reference vectors are grouped into a subset with the same attribute, then the neighbor relationships between reference vectors with the same attribute can be obtained; if reference vectors are grouped into a subset with similar attributes, then the neighbor relationships between reference vectors with similar attributes can be obtained.

[0070] Optionally, the step of dividing the reference vector set into multiple subsets based on the reference coding data includes: grouping reference vectors with the same reference coding data into the same subset to obtain multiple initial subsets; merging initial subsets with fewer than a threshold elements into corresponding approximate subsets to obtain multiple subsets, wherein the attribute similarity between the reference coding data of the initial subsets and the reference coding data of the corresponding approximate subsets is greater than or equal to a similarity threshold. By first dividing the initial subsets into subsets composed of reference vectors with the same attributes, it is possible to prioritize the clustering of reference vectors with the same attributes together, thereby obtaining neighbor relationships between reference vectors with the same attributes and improving the recall rate. Furthermore, by merging the initial subsets with fewer elements into corresponding approximate subsets, it is possible to ensure that each subset has a sufficient number of elements, thus obtaining a sufficient number of neighbor relationships with similar attributes, which helps to improve the retrieval recall rate. As an example, attribute similarity can be the vector distance between attribute combinations, such as, but not limited to, XOR distance; when the encoded data is the sorting number of the aforementioned attribute combinations, attribute similarity can be obtained by negatively correlated processing of the absolute value of the difference between the encoded data, such as, but not limited to, finding the opposite number or the reciprocal number. When finding the opposite number, the similarity threshold should be a negative value.

[0071] The composite nearest neighbor graph integration process involves combining the neighbor relationships from the baseline graph and the attribute graph to obtain the composite nearest neighbor graph. By integrating the neighbor relationships from the baseline graph and each attribute graph, relatively close neighbors with the same attribute can be added to each reference point based on the baseline graph, ensuring the reliable implementation of the graph indexing strategy. As an example, by reasonably dividing the subset during the attribute graph construction step, it can be ensured that the number of neighbors with the same or similar attributes and neighbors with different attributes is roughly equal among the neighbors of each reference point in the composite nearest neighbor graph. This ensures that reference points have neighbors with the same or similar attributes while also guaranteeing a sufficient number of neighbors overall, increasing the connectivity of the composite nearest neighbor graph.

[0072] It should be understood that the baseline graph and attribute graphs can be constructed simultaneously or sequentially. Different attribute graphs can also be constructed simultaneously or sequentially. The composite nearest neighbor graph can be obtained by integrating the baseline graph and all attribute graphs after their construction, or it can be integrated once at least two nearest neighbor graphs are constructed, and then integrated into the existing composite nearest neighbor graph after a new nearest neighbor graph is constructed, until all nearest neighbor graphs are constructed, thus obtaining the complete composite nearest neighbor graph. This disclosure does not limit the execution order of these nearest neighbor graph construction steps or the detailed integration strategy of the composite nearest neighbor graph.

[0073] Next, combine Figures 5 to 7This document describes a specific embodiment of a composite nearest neighbor graph integration method, and the process of using this composite nearest neighbor graph for graph index retrieval. In this specific embodiment, the reference vector set includes 14 reference vectors, which are sequentially numbered. Figure 5 A baseline diagram of these 14 reference vectors is shown, with reference points labeled using the same symbols as the reference vectors.

[0074] These 14 reference vectors involve three attribute combinations, i.e., three reference encoded data, and are divided into three subsets. That is, each subset contains reference vectors with the same attributes.

[0075] The specific divisions are as follows:

[0076] First subset: reference vectors 1, 3, 6, 8;

[0077] Second subset: reference vectors 2, 4, 9, 11, 12, 13;

[0078] Third subset: reference vectors 5, 7, 10, 14.

[0079] Figures 6a to 6c The attribute graphs for these three subsets are shown separately. For ease of distinction, in... Figure 5 In the diagram, a thin solid double-headed line represents the neighbor relationship between reference points. Figures 6a to 6c The neighbor relationships between reference points are represented by thick dashed double-headed arrows. After obtaining the above diagrams, the following can be used: Figures 6a to 6c Add the double-headed thick dashed line to Figure 5 In the middle, we get Figure 7 The composite nearest neighbor graph shown.

[0080] During the retrieval, the preset number of points to be filtered in each step is 2, and the vector distance of each reference point is determined without repetition. The attributes of the vector to be retrieved are the same as those of the third subset, that is, the same as those of reference vectors 5, 7, 10, and 14. The truncation condition is that the reference encoded data corresponding to the current child candidate point and its two consecutive parent candidate points are all different from the encoded data to be retrieved, i.e., N=2. The termination condition is that the vector distance of all neighbors of the current parent candidate point is greater than the minimum vector distance of all existing parent candidate points. The final number of target vectors to be obtained is 3. The retrieval process is as follows:

[0081] 1) Randomly determine reference point 1 as the entry point, which is the first-level parent candidate point.

[0082] 2) Combining vector distance, assuming that child candidate points 2 and 3 are selected from all the neighbor points 2, 3 and 5 of reference point 1, there are only first-level parent candidate points, which will not meet the truncation condition. The second-level parent candidate points 2 and 3 can be determined and the search can continue.

[0083] At this point, it should be noted that since the reference coding data of the first-level parent candidate point and the second-level parent candidate point are different from the coding data to be retrieved, if the sub-candidate point obtained by the downward exploration is not any of the reference points 5, 7, 10, and 14, then the truncation condition is met.

[0084] 3) For the second-level parent candidate point 2, the remaining neighbor points are 6, 7, and 13. Assuming that child candidate points 7 and 13 are selected from them, then the third-level parent candidate point 7 can be obtained.

[0085] For the second-level parent candidate point 3, all remaining neighbor points are 8 and 9. Assuming that the termination condition is not met, they can all be used as child candidate points. However, since the truncation condition is met, none of them can be used as the third-level parent candidate point.

[0086] At this point, the third-level parent candidate point 7 has been determined.

[0087] 4) For the third-level parent candidate point 7, only one neighbor point 12 remains. If the termination condition is not met, it can be used as a child candidate point. At the same time, since its parent candidate point is 7, the truncation condition is not met, so the child candidate point 12 can be used as a fourth-level parent candidate point.

[0088] 5) For the fourth-level parent candidate point 12, since there are no remaining neighbor points, no new child candidate points can be obtained, the termination condition is met, and the downward exploration stops.

[0089] 6) From all the parent candidate points 1, 2, 3, 7, 12, determine the 3 with the smallest vector distance. Let's assume they are parent candidate points 2, 3, and 7. Then, take the reference vectors 2, 3, and 7 as the target vectors.

[0090] This concludes the introduction to graph indexing strategies.

[0091] The second type of search strategy is to filter first and then search.

[0092] Optionally, step 403 includes: determining reference encoded data that matches the coded data to be retrieved from the reference vector set; obtaining a candidate vector set based on the reference vectors corresponding to the matching reference encoded data; and determining the target vector corresponding to the vector to be retrieved from the candidate vector set. By first performing attribute filtering and then retrieving the target vector from the filtered candidate vector set, a high recall rate can be guaranteed. Furthermore, since both the reference encoded data and the coded data to be retrieved in this disclosure are single-coded data, the attribute filtering speed can be greatly improved, and the multi-attribute constraint problem can be transformed into a single-attribute constraint problem, allowing the retrieval efficiency to remain at a high level as the number of attributes increases. Therefore, both recall rate and retrieval efficiency can be balanced.

[0093] It should be understood that, depending on the retrieval requirements, when filtering attributes, the reference coded data that matches the coded data to be retrieved can be the same reference coded data as the coded data to be retrieved, or it can include both the same reference coded data as the coded data to be retrieved and reference coded data similar to the coded data to be retrieved. This disclosure does not impose any limitations in this regard. As an example, when retrieving a candidate vector set, the vector distance can be calculated directly for retrieval, or a retrieval method such as quantization compression can be used, or other feasible retrieval methods can be used. This disclosure also does not impose any limitations in this regard.

[0094] Experiments have shown that the data query method according to the exemplary embodiments of this disclosure improves performance by 10 times compared to existing methods. Furthermore, as the number of attribute combinations increases (either by adding new content to a certain attribute or by adding new attribute types), the recall rate of other methods will decrease drastically, but the method of this disclosure can still maintain a recall rate of over 95%.

[0095] Figure 8 This is a block diagram illustrating a data query apparatus according to exemplary embodiments of the present disclosure. It should be understood that the data query apparatus according to exemplary embodiments of the present disclosure can be implemented in terminal devices such as smartphones, tablets, and personal computers (PCs) in a software, hardware, or software-hardware combination manner, or in devices such as servers.

[0096] Reference Figure 8 The data query device 800 includes an encoding unit 801, an acquisition unit 802, and a retrieval unit 803.

[0097] The encoding unit 801 can encode the attribute information of the vector to be retrieved to obtain a corresponding encoded data, which is used as the encoded data to be retrieved.

[0098] Optionally, the encoding unit 801 can query the encoding data set for the attribute combination formed by the attribute information of the vector to be retrieved, and obtain the encoding data corresponding to the attribute information of the vector to be retrieved, as the encoding data to be retrieved. The encoding data set stores attribute combinations and encoding data that correspond one-to-one.

[0099] Optionally, the encoded dataset is generated through the following steps: obtaining multiple attribute combinations; determining the similarity between the multiple attribute combinations; and assigning corresponding encoded data to each attribute combination, wherein the difference between the encoded data of any two attribute combinations is negatively correlated with the similarity between any two attribute combinations.

[0100] The acquisition unit 802 can acquire a set of reference vectors, which includes multiple reference vectors. Each reference vector is associated with reference encoded data. A set of reference encoded data is a set of encoded data obtained by encoding the attribute information of the associated reference vector.

[0101] Optionally, the reference vector set is generated through the following steps: obtaining multiple reference vectors; constructing a composite nearest neighbor graph based on the multiple reference vectors as the reference vector set, wherein: the multiple reference vectors represent multiple image points in the composite nearest neighbor graph, denoted as reference points; each reference point is a neighbor point with at least one other reference point, and the other reference points are reference points in the composite nearest neighbor graph other than the current reference point; each reference point is a similar point with at least one neighbor point of each reference point, and similar points are points whose encoded data satisfy a preset relationship, which is used to indicate that two encoded data are the same or similar.

[0102] The retrieval unit 803 can combine the coded data to be retrieved and the reference coded data to determine the target vector corresponding to the coded data to be retrieved from the reference vector set. The target vector is used for data querying.

[0103] Optionally, the retrieval unit 803 may also take any reference point in the composite nearest neighbor graph as a parent candidate point, determine the vector distance between the reference vectors corresponding to all neighbor points of the parent candidate point and the vector to be retrieved; based on the vector distance, select a preset number of neighbor points from all neighbor points as child candidate points of the parent candidate point; take each child candidate point that does not meet the truncation condition as a new parent candidate point, and for the new parent candidate point, repeat the steps of determining the vector distance between the reference vectors corresponding to all neighbor points of the parent candidate point and the vector to be retrieved, and selecting a preset number of neighbor points from all neighbor points as child candidate points of the parent candidate point based on the vector distance. The truncation condition is used to indicate that the difference between the reference encoded data corresponding to the child candidate point and the encoded data to be retrieved is greater than the difference threshold, and the difference between the reference encoded data corresponding to the consecutive N levels of parent candidate points before the child candidate point and the encoded data to be retrieved is greater than the difference threshold, where N is a preset value; until the termination condition is met, select the target vector from the reference vectors corresponding to all parent candidate points. The termination condition is used to indicate that no new child candidate point can be obtained.

[0104] Optionally, the composite nearest neighbor graph is constructed through the following steps: constructing a nearest neighbor graph as a baseline graph based on the vector distances between multiple reference vectors; dividing the set of reference vectors into multiple subsets based on the reference encoded data, and constructing a nearest neighbor graph for each subset as an attribute graph; and combining the neighbor relationships in the baseline graph and the attribute graph to obtain the composite nearest neighbor graph.

[0105] Optionally, based on the reference coding data, the reference vector set is divided into multiple subsets, including: assigning reference vectors with the same reference coding data in the reference vector set to the same subset to obtain multiple initial subsets; merging the initial subsets with fewer than a number threshold into the corresponding approximate subsets to obtain multiple subsets, wherein the attribute similarity between the reference coding data of the initial subset and the reference coding data of the corresponding approximate subset is greater than or equal to a similarity threshold.

[0106] Optionally, the retrieval unit 803 may also determine, based on the coded data to be retrieved, reference coded data in the reference vector set that matches the coded data to be retrieved; obtain a candidate vector set based on the reference vectors corresponding to the matching reference coded data; and determine the target vector corresponding to the vector to be retrieved from the candidate vector set.

[0107] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0108] Figure 9 This is a block diagram of an electronic device according to exemplary embodiments of the present disclosure.

[0109] Reference Figure 9 The electronic device 900 includes at least one memory 901 and at least one processor 902. The at least one memory 901 stores a set of computer-executable instructions. When the set of computer-executable instructions is executed by the at least one processor 902, a data query method according to an exemplary embodiment of the present disclosure is performed.

[0110] As an example, electronic device 900 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, electronic device 900 is not necessarily a single electronic device, but may be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 900 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0111] In electronic device 900, processor 902 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0112] The processor 902 can execute instructions or code stored in the memory 901, which can also store data. Instructions and data can also be sent and received via a network through a network interface device, which can employ any known transmission protocol.

[0113] The memory 901 may be integrated with the processor 902, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 901 may include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory 901 and the processor 902 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 902 to read files stored in the memory.

[0114] In addition, the electronic device 900 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 900 can be interconnected via a bus and / or network.

[0115] According to exemplary embodiments of the present disclosure, a computer-readable storage medium may also be provided, which, when instructions in the computer-readable storage medium are executed by at least one processor, causes at least one processor to perform a data query method according to exemplary embodiments of the present disclosure. Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0116] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, the computer program product including computer instructions that, when executed by at least one processor, cause at least one processor to perform a data query method according to exemplary embodiments of the present disclosure.

[0117] According to the exemplary embodiments of the data query method, apparatus, electronic device, and computer-readable storage medium of this disclosure, by encoding all attribute information of a vector into a single encoded data, the encoded data can be made capable of participating in the retrieval calculation, thereby optimizing the retrieval results using attributes and improving recall. Furthermore, since all attributes of the vector can be represented by a single encoded data regardless of the number of attributes involved, a multi-attribute constraint problem can be transformed into a single-attribute constraint problem, allowing retrieval efficiency to remain at a high level as the number of attributes increases. Therefore, this disclosure can balance recall and retrieval efficiency.

[0118] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0119] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data query method, characterized by, The method comprises the following steps: encoding attribute information of a to-be-searched vector to obtain a corresponding encoding data as to-be-searched encoding data; obtaining a reference vector set, the reference vector set comprising a plurality of reference vectors, each reference vector being associated with reference encoding data, the reference encoding data being an encoding data obtained by encoding attribute information of the associated reference vector; determining a target vector corresponding to the to-be-searched vector from the reference vector set in combination with the to-be-searched encoding data and the reference encoding data, the target vector being used for data query; wherein the strategy for determining the target vector comprises at least one of a graph indexing strategy and a strategy of filtering first and searching later, in the graph indexing strategy, the reference vector set is generated by the following steps: obtaining the plurality of reference vectors; constructing a composite proximity graph according to the plurality of reference vectors as the reference vector set, wherein: the plurality of reference vectors are represented as a plurality of image points in the composite proximity graph, denoted as reference points; each reference point is a neighbor point of at least one other reference point, the other reference point being a reference point of the composite proximity graph other than the current reference point; each reference point and at least one neighbor point of each reference point are similar points, the similar points being points whose encoding data satisfy a preset relationship, the preset relationship being used to represent that two encoding data are the same or similar, the flow of the graph indexing strategy comprises step-by-step screening of reference points in the composite proximity graph in combination with vector distance and end condition, and further comprises judging whether to screen neighbor points of the reference points that have passed the screening according to a truncation condition, wherein the truncation condition is used to represent that the current reference point that has passed the screening and the previous N-level continuous reference points are all different from and not similar to the attribute of the to-be-searched vector.

2. The data query method of claim 1, wherein, The encoding processing of the attribute information of the to-be-searched vector to obtain a corresponding encoding data as to-be-searched encoding data comprises: querying an encoding data set for an attribute combination formed by the attribute information of the to-be-searched vector to obtain an encoding data corresponding to the attribute information of the to-be-searched vector as the to-be-searched encoding data, wherein the encoding data set stores a one-to-one corresponding attribute combination and encoding data.

3. The data query method of claim 2, wherein, The encoding data set is generated by the following steps: obtaining a plurality of attribute combinations; determining the similarity between the plurality of attribute combinations; assigning a corresponding encoding data to each attribute combination, wherein the difference between the encoding data of any two attribute combinations is negatively correlated with the similarity between the any two attribute combinations.

4. The data query method of any one of claims 1 to 3, wherein, The flow of the graph indexing strategy comprises: taking any reference point in the composite proximity graph as a parent candidate point, determining the vector distance between the reference vectors corresponding to all neighbor points of the parent candidate point and the to-be-searched vector; screening a preset number of neighbor points from the all neighbor points as child candidate points of the parent candidate point according to the vector distance; The sub-candidate points that do not satisfy the truncation condition are taken as new parent candidate points one by one, and the steps of determining the vector distance between the reference vectors corresponding to all neighbor points of the parent candidate points and the to-be-retrieved vector, and screening a preset number of neighbor points from the all neighbor points as the sub-candidate points of the parent candidate points according to the vector distance are repeatedly performed for the new parent candidate points, wherein the truncation condition is used to represent that the difference value between the reference encoding data and the to-be-retrieved encoding data of the sub-candidate point is greater than a difference threshold, and the difference values between the reference encoding data and the to-be-retrieved encoding data of the continuous N-level parent candidate points before the sub-candidate point are all greater than the difference threshold, and the N is a preset value; Until the end condition is satisfied, the target vector is selected from the reference vectors corresponding to all the parent candidate points, and the end condition is used to represent that no new sub-candidate point can be obtained.

5. The data query method of any one of claims 1 to 3, wherein, The composite neighbor graph is constructed by the following steps: A neighbor graph is constructed as a benchmark graph according to the vector distances between the plurality of reference vectors; The reference vector set is divided into a plurality of sub-sets according to the reference encoding data, and a neighbor graph is constructed as an attribute graph for each sub-set; The composite neighbor graph is obtained by combining the neighbor relationships in the benchmark graph and the attribute graph.

6. The data query method of claim 5, wherein, The reference vector set is divided into a plurality of sub-sets according to the reference encoding data, and a neighbor graph is constructed as an attribute graph for each sub-set; The reference vectors with the same reference encoding data in the reference vector set are classified into the same sub-set to obtain a plurality of initial sub-sets; The initial sub-set with an element number less than a number threshold is merged into a corresponding approximate sub-set to obtain the plurality of sub-sets, wherein the attribute similarity between the reference encoding data of the initial sub-set and the reference encoding data of the corresponding approximate sub-set is greater than or equal to a similarity threshold.

7. The data query method of any one of claims 1 to 3, wherein, The process of the filtering-retrieval strategy includes: The reference encoding data consistent with the to-be-retrieved encoding data in the reference vector set is determined according to the to-be-retrieved encoding data; A candidate vector set is obtained according to the reference vectors corresponding to the consistent reference encoding data; The target vector corresponding to the to-be-retrieved vector is determined from the candidate vector set.

8. A data query apparatus, characterized by comprising: It includes: An encoding unit configured to perform encoding processing on attribute information of a to-be-retrieved vector to obtain corresponding one encoding data as to-be-retrieved encoding data; An acquisition unit configured to perform acquisition of a reference vector set, the reference vector set including a plurality of reference vectors, each reference vector being associated with reference encoding data, one reference encoding data being one encoding data obtained by performing encoding processing on attribute information of the associated reference vector; A retrieval unit configured to perform determination of a target vector corresponding to the to-be-retrieved vector from the reference vector set in combination with the to-be-retrieved encoding data and the reference encoding data, the target vector being used for data query; The strategy for determining the target vector includes at least one of a graph index strategy and a pre-filtering and then retrieving strategy. In the graph index strategy, the reference vector set is generated by the following steps: obtaining the plurality of reference vectors; constructing a composite neighbor graph according to the plurality of reference vectors as the reference vector set, wherein the plurality of reference vectors are represented as a plurality of image points in the composite neighbor graph, denoted as reference points; each reference point is a neighbor point of at least one other reference point, and the other reference point is a reference point in the composite neighbor graph except the current reference point; each reference point and at least one neighbor point of the each reference point are similar points, and the similar points are points whose encoding data satisfy a preset relationship, and the preset relationship is used to represent that two encoding data are the same or similar, The flow of the graph index strategy includes step-by-step screening of the reference points in the composite neighbor graph in combination with vector distance and an end condition, and further includes judging whether to screen the neighbor points of the reference points that have passed the screening according to a truncation condition, wherein the truncation condition is used to represent that the current reference point that has passed the screening and the previous N-level continuous reference points are different from and not similar to the attribute of the vector to be retrieved.

9. An electronic device, comprising: comprise: at least one processor; at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to perform the data query method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, and when the instructions are executed by at least one processor, the at least one processor is caused to perform the data query method according to any one of claims 1 to 7.

Images